From b4a1dea7b03b3e9a0e6551bfce68064cb57a7ac7 Mon Sep 17 00:00:00 2001 From: Aaron Mildenstein Date: Tue, 1 Sep 2026 16:15:52 -0600 Subject: [PATCH 01/62] Ignore copilot file in .github --- .gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitignore b/.gitignore index 96ae9dd..9e0234d 100644 --- a/.gitignore +++ b/.gitignore @@ -242,6 +242,7 @@ CLAUDE.md .cursorrules IDEA.md graphify-out/ +.github/copilot-instructions.md # LogstashAgent (cloned from GitHub in host mode) /LogstashAgent/ From 484b182d50fd823751fbb8f73059549e880a0be4 Mon Sep 17 00:00:00 2001 From: Aaron Mildenstein Date: Tue, 1 Sep 2026 16:58:06 -0600 Subject: [PATCH 02/62] docs: 0.5.2 multi-database design spec SQLite remains the default; PostgreSQL and MariaDB/MySQL are selected via discrete LOGSTASHUI_DB_* env vars. Django ORM only (no new CRUD layer). --- .../specs/2026-08-21-multi-database-design.md | 266 ++++++++++++++++++ 1 file changed, 266 insertions(+) create mode 100644 docs/superpowers/specs/2026-08-21-multi-database-design.md diff --git a/docs/superpowers/specs/2026-08-21-multi-database-design.md b/docs/superpowers/specs/2026-08-21-multi-database-design.md new file mode 100644 index 0000000..ce75e33 --- /dev/null +++ b/docs/superpowers/specs/2026-08-21-multi-database-design.md @@ -0,0 +1,266 @@ +# 2026-08-21 — Multi-database design (SQLite | PostgreSQL | MariaDB/MySQL) + +**LogstashUI version:** 0.5.2 +**Python:** 3.12–3.14 +**Django:** 6.x (existing) +**Chosen approach:** Django backends + keep gevent. No new CRUD/repository layer. + +## Goal + +Operators can run LogstashUI on **SQLite** (default), **PostgreSQL**, or **MariaDB/MySQL** using discrete environment variables. Default `pytest` on SQLite stays green. Local Docker can exercise each server engine for CRUD **and** SQLite→server migration. + +SQLite does not scale under gunicorn/gevent (2 workers × 1000 greenlets). Server engines address that without changing the product data model. + +## Non-goals (0.5.2) + +- A new DAO/repository over `Model.objects` (Django ORM already abstracts CRUD). +- YAML, `logstashui.yml`, or `DATABASE_URL`. +- Oracle, SQL Server, or other engines. +- Requiring PgBouncer/ProxySQL, or changing gunicorn off gevent. +- Live dual-write, reverse migration (server→SQLite), or pgloader as the supported path. +- An in-app migration wizard (stopping :8443 removes the UI). +- A 503 maintenance page on :8443 during copy. +- Helm charts; shipping Postgres/MariaDB/MySQL in the default smoke compose. +- Product CA rotation; relocating `DATA_DIR` (TLS, secrets, logs, staticfiles stay on disk). + +## Decisions (locked) + +| Topic | Decision | +|---|---| +| Abstraction | Django ORM only; the switch is `build_databases()` | +| Default engine | SQLite; warn at scale; do not refuse to start | +| Config | Env vars only, discrete `LOGSTASHUI_DB_*` | +| Engine names | Canonical: `sqlite`, `postgresql`, `mysql` | +| MariaDB vs MySQL | One engine `mysql`; both documented | +| Drivers | Optional extras; Docker image installs `[databases]` | +| Postgres driver | `psycopg[binary]` (v3) + gevent wait callback | +| MySQL driver | `PyMySQL` + `pymysql.install_as_MySQLdb()` before Django setup | +| Gunicorn | gevent for all engines | +| Pooler | Not required; `CONN_MAX_AGE` + health checks | +| Existing SQLite deploys | Stay on SQLite until the operator migrates | +| Migration | Documented offline dump/load **and** BETA CLI that stops gunicorn | +| Tests | Default pytest = SQLite; local Docker compose for Postgres + MariaDB + MySQL (functional **and** migrator); CI uses the same compose | + +## Architecture + +``` +LOGSTASHUI_DB_* → build_databases(DATA_DIR) → Django DATABASES['default'] + ├ sqlite3 DATA_DIR/db.sqlite3 + ├ postgresql psycopg 3 + └ mysql PyMySQL (MariaDB or MySQL) +``` + +All apps keep `Model.objects`, `transaction.atomic`, `select_for_update`, and existing migrations. No `raw()` / `cursor()` / `PRAGMA` outside `LogstashUI/database.py` (SQLite PRAGMAs stay there). + +`DATA_DIR` remains mandatory: TLS, `.django_secret_key`, logs, staticfiles. A remote database does not remove the PVC or bind-mount. + +`pymysql.install_as_MySQLdb()` runs once at process start, before `django.setup()`, when the MySQL extra is installed (safe no-op if engine is not mysql). + +## Configuration + +No YAML. No URL DSN. + +| Variable | Default | Rules | +|---|---|---| +| `LOGSTASHUI_DB_ENGINE` | `sqlite` | Aliases below | +| `LOGSTASHUI_DB_NAME` | sqlite: `DATA_DIR/db.sqlite3`; else `logstashui` | | +| `LOGSTASHUI_DB_HOST` | empty | **Required** for postgresql/mysql | +| `LOGSTASHUI_DB_PORT` | 5432 / 3306 | Engine default if unset | +| `LOGSTASHUI_DB_USER` | empty | **Required** for postgresql/mysql | +| `LOGSTASHUI_DB_PASSWORD` | empty | Secret / EnvironmentFile | +| `LOGSTASHUI_DB_SSLMODE` | postgres: `prefer`; mysql: unset | postgres: `disable`, `allow`, `prefer`, `require`, `verify-ca`, `verify-full` | +| `LOGSTASHUI_DB_SSL_CA` | empty | Path; mysql TLS and postgres `verify-*` | +| `LOGSTASHUI_DB_CONN_MAX_AGE` | `60` | `0` = close per request | +| `LOGSTASHUI_DB_CONN_HEALTH_CHECKS` | `true` | Django `CONN_HEALTH_CHECKS` | + +**Aliases** (normalize to canonical): + +- `sqlite3` → `sqlite` +- `postgres`, `postgresql` → `postgresql` +- `mysql`, `mariadb`, `my` → `mysql` + +**SQLite OPTIONS (unchanged):** `timeout=20`; `init_command` `PRAGMA busy_timeout=20000; PRAGMA journal_mode=WAL;` + +**MySQL OPTIONS:** `charset=utf8mb4`; `init_command=SET sql_mode='STRICT_TRANS_TABLES'`; collation `utf8mb4_bin` so unique `Policy.name` (and similar) match SQLite/Postgres case-sensitivity. MySQL’s default `_ci` collation would treat `Foo` and `foo` as a clash. + +**Postgres OPTIONS:** `sslmode` from env; `sslrootcert` when `LOGSTASHUI_DB_SSL_CA` is set. + +**Version floors** (docs; fail-fast if the server version is below): PostgreSQL **14+**, MariaDB **10.6+**, MySQL **8.0+**. + +Update `LogstashUI/packaging/logstashui.default`, `docs/docs/logstashui/configuration/environment.md`, the systemd generator (interactive prompt for engine + host/name/user; password from env or prompt), and CHANGELOG. + +## Packaging and Docker + +Default wheel: SQLite only (stdlib). + +``` +LogstashUI[postgres] → psycopg[binary] +LogstashUI[mysql] → PyMySQL +LogstashUI[databases] → both +``` + +Missing extra at runtime → `RuntimeError` naming the extra (`uv pip install 'LogstashUI[postgres]'`). + +**Docker/K8s image:** install `LogstashUI[databases]` (e.g. `uv pip install '/app[databases]'`). Operators set env; no extra pip in the cluster. + +Native pip/uv: extras are the supported advanced path. + +Python 3.12–3.14: `psycopg[binary]` wheels; PyMySQL is pure Python (no mysqlclient compile). Pin `psycopg[binary]` to a release that ships 3.14 wheels, or document build deps. + +## Gevent and connections + +Keep `--worker-class gevent` and `--worker-connections 1000`. + +- Register a psycopg3 **gevent wait callback** at process start so libpq does not block the hub. +- PyMySQL uses monkey-patched sockets; do not add mysqlclient. +- `CONN_MAX_AGE=60` and `CONN_HEALTH_CHECKS=true` by default. +- Do not require an external pooler. Document that `LOGSTASHUI_WORKERS` × in-flight greenlets must stay under the server’s `max_connections` (leave room for `migrate` and the test matrix). Optional PgBouncer is documented, not required. +- In-process `psycopg_pool` only if it is gevent-safe; otherwise skip for 0.5.2. + +**SQLite scale warning:** one WARNING at `logstashui serve` when the engine is sqlite and `LOGSTASHUI_WORKERS` > 1 (gunicorn default is 2). Message: SQLite is the small-install default; use PostgreSQL or MySQL/MariaDB for concurrent agents. Do not refuse to start. No UI modal. + +## Fail-fast (before gunicorn bind) + +| Condition | Result | +|---|---| +| Unknown engine | `RuntimeError`, list canonical names + aliases | +| postgresql/mysql, driver not installed | `RuntimeError` + extra name | +| postgresql/mysql, HOST/NAME/USER empty | `RuntimeError` naming the empty vars | +| Cannot connect during `migrate` | Django error; serve exits non-zero | +| SSL verify fail | Driver error; do not swallow | + +Never log passwords. INFO may log engine, host, and name. + +## Migration off SQLite + +One mechanism: Django `dumpdata` → target `migrate` → `loaddata` → `sqlsequencereset`. + +`DATA_DIR` does not move. `.django_secret_key` must stay or Fernet keystore rows will not decrypt. + +### Offline (supported) + +Documented in deploy/environment: + +1. Stop LogstashUI. +2. Copy `DATA_DIR/db.sqlite3` to a backup. +3. Dump from SQLite. +4. Set `LOGSTASHUI_DB_*` for the server. +5. `logstashui manage migrate --noinput`. +6. `loaddata`. +7. `sqlsequencereset` (Postgres sequences; document as skippable/no-op guidance for MySQL). +8. Start LogstashUI. Verify login, a policy, and an agent. + +### BETA CLI + +`logstashui migrate-engine --to postgresql|mysql --i-have-a-backup` + +1. Print BETA + backup warning; refuse without `--i-have-a-backup`. +2. Source must be sqlite; target must be postgresql or mysql; extras must be installed. +3. If gunicorn is running (pidfile written by `serve` in 0.5.2, or `--pid`), **SIGTERM it**. That is maintenance: the UI port closes. No in-app wizard. +4. `PRAGMA wal_checkpoint(TRUNCATE)` on SQLite. +5. `dumpdata --natural-foreign --natural-primary`, exclude `contenttypes`, `auth.permission`, `sessions`. +6. Target `DATABASES` from env. `migrate --noinput`, `loaddata`, `sqlsequencereset`. +7. Print remaining env. Do not edit `/etc/default/logstashui` unless `--write-env PATH`. Do not auto-restart serve (systemd `Restart=` would race). Document `systemctl stop` → migrate → `systemctl start`. + +No reverse migrator. No sqlite→sqlite. If a pidfile points at a live process and SIGTERM fails, refuse. + +## Testing + +### Default inner loop (unchanged) + +`pytest` uses SQLite via settings. **All existing tests must pass** with no Docker and no extras. + +Update `test_database.py`: postgresql/mysql no longer raise “not implemented”; they return a valid `DATABASES` dict (connect may be skipped in unit tests). New unit tests: aliases, missing driver, missing HOST, sqlite default path, SSL/CONN keys. + +### Local Docker matrix (required) + +Ship `docker/docker-compose.db.yml` with **three** databases for local use and CI: + +- PostgreSQL 16 (or 14+) +- MariaDB 10.11+ (or 10.6+) +- MySQL 8.0+ + +Fixed test credentials in the compose file (not production). Healthchecks so tests wait for ready. + +Ship `bin/test_databases.sh` (and `bin/test_databases.bat` to match the existing Windows start scripts): + +1. `docker compose -f docker/docker-compose.db.yml up -d --wait` +2. Run the **existing pytest suite** against each server engine (`LOGSTASHUI_DB_ENGINE` plus host/port/user/password/name pointing at the container). A SQLite run remains the first step (no compose required for that step). +3. Run **migration tests**: a fixture SQLite database → dump/load (the same helpers as `migrate-engine`) into Postgres, MariaDB, and MySQL; assert a `Policy`, `User`, and JSONField row survive. +4. Tear down compose unless `--keep`. + +This is how a developer proves basic functionality and migration without waiting on CI. CI calls the same script. + +If Docker is unavailable, `bin/test_databases.sh` fails clearly; default `pytest` still works. + +Do **not** put Postgres/MariaDB/MySQL in `docker-compose.yml` or the smoke stack by default. Smoke stays SQLite so product CA and PUID behavior stay unchanged. + +### Pre-existing pytest failures + +`test_update_policy_default_policy_forbidden`, `test_delete_policy_default_policy_forbidden`, and `test_clone_policy_success` were already failing before this work. 0.5.2 database work must not add new failures. Do not silently “fix” those unless they fail **because of** engine differences. + +## Dialect traps (must handle) + +- **Unique case-sensitivity:** MySQL `utf8mb4_bin` (or equivalent) so `Policy.name` unique matches SQLite/Postgres. +- **JSONField:** Connection `status_blob`, Revision `snapshot_json`, SNMP metadata/templates. Native JSON on all three version floors. +- **`select_for_update`:** real row locks on Postgres/MySQL; first-user creation in `Management/views.py`. Tests that mock it stay. Add one real concurrency test on a server engine if cheap. +- **`db_table = 'settings'`:** must migrate cleanly on MySQL (quoted identifier). +- **Index/constraint name length:** MySQL 64 characters; existing `UniqueConstraint` names are short enough — verify `migrate` on MySQL. +- **Timezones:** `USE_TZ=True`; Django maps Postgres timestamptz vs MySQL datetime if USE_TZ stays. +- **BooleanField / BigAutoField:** ORM. +- **RunPython migrations:** ORM-based; portable. No new SQLite-only `RunSQL`. +- **loaddata PKs:** `sqlsequencereset` after load on Postgres. +- **Encrypted CharFields:** copied as Fernet strings; secret key stays in `DATA_DIR`. + +## Docs and operator surfaces + +- `docs/docs/logstashui/configuration/environment.md` — replace “sqlite only”. +- `docs/docs/logstashui/general/deploy.md` — PVC still required; the database may be external. +- Offline migration procedure + BETA CLI warnings. +- SQLite scale warning explained. +- Extras vs Docker `[databases]`. +- Optional PgBouncer note. +- `CHANGELOG.md` for 0.5.2. +- systemd sample env keys (already stubbed in `logstashui.default`). +- Kubernetes: ConfigMap for engine/host/name/user + Secret for password; PVC still for `DATA_DIR`. + +## Obstacles + +1. **Gevent × `max_connections`:** default 2 workers is acceptable for small Postgres; document; do not open thousands of sessions. +2. **psycopg3 + gevent:** the wait callback is mandatory; missing it looks like random hangs. +3. **mysqlclient vs PyMySQL:** the C client blocks the hub; PyMySQL is the gevent choice; Django sees it via `install_as_MySQLdb`. +4. **MySQL unique collation** silently changes uniqueness vs SQLite if left at `_ci`. +5. **dumpdata/loaddata** is not a perfect replica (sessions dropped, contenttypes excluded); operators re-login. +6. **Stopping gunicorn** for BETA migrate races with systemd `Restart=`; do not auto-restart; document stop → migrate → start. +7. **Driver wheels on 3.14:** PyMySQL is safe; pin `psycopg[binary]` to a release with 3.14 wheels, or document build deps. +8. **Hatch extras** must not pull mysqlclient into the default wheel. +9. **Tests that assume sqlite paths** (`db.sqlite3` in `paths.py` legacy migrate) stay sqlite-only. +10. **Smoke stack** stays sqlite so CA/PUID work is unchanged. + +## Success criteria + +- Unset engine → SQLite, same as 0.5.1. +- `LOGSTASHUI_DB_ENGINE=postgresql` or `mysql` with valid env → migrate + serve. +- Default `pytest` green on SQLite (no new failures). +- `bin/test_databases.sh` against local Docker Postgres, MariaDB, and MySQL: existing suite + dump/load migration assertions. +- BETA `migrate-engine` refuses without `--i-have-a-backup` and stops gunicorn when a pidfile is live. +- Docker image can use all three engines via env only. +- Docs match env keys. +- Product CA and `DATA_DIR` behavior unchanged. + +## Implementation order + +1. Extras + `build_databases()` + fail-fast + unit tests (SQLite pytest). +2. PyMySQL shim + psycopg gevent wait + `CONN_*`. +3. `docker-compose.db.yml` + `bin/test_databases.sh`; existing suite passing on three servers. +4. SQLite scale warning; systemd/docs/sample env. +5. `serve` pidfile; `migrate-engine` BETA; migration tests in the same script. +6. Docker image `[databases]`. +7. CHANGELOG 0.5.2. + +## Spec self-review + +- **Placeholders:** none. Version floors, env keys, extras, compose services, and CLI flags are explicit. +- **Consistency:** architecture (ORM switch only) matches packaging, gevent, migrator, and tests. Default remains SQLite everywhere except when env selects a server engine. +- **Scope:** one release-sized feature (engine wiring + migrator + test matrix). No Helm, no worker-class split, no repository layer. +- **Ambiguity resolved:** `mysql` covers MariaDB and MySQL; migrator stops the UI port rather than serving a maintenance page; local Docker is required for the three-engine script, not for default pytest. From be2aa89c049619ba3b6fed8695cebbf43d65e3ba Mon Sep 17 00:00:00 2001 From: Aaron Mildenstein Date: Tue, 1 Sep 2026 17:07:31 -0600 Subject: [PATCH 03/62] docs: 0.5.2 multi-database implementation plan TDD tasks for env-selected SQLite/PostgreSQL/MySQL, extras, Docker matrix, and BETA migrate-engine. --- .../plans/2026-08-21-multi-database.md | 1731 +++++++++++++++++ 1 file changed, 1731 insertions(+) create mode 100644 docs/superpowers/plans/2026-08-21-multi-database.md diff --git a/docs/superpowers/plans/2026-08-21-multi-database.md b/docs/superpowers/plans/2026-08-21-multi-database.md new file mode 100644 index 0000000..8b78d94 --- /dev/null +++ b/docs/superpowers/plans/2026-08-21-multi-database.md @@ -0,0 +1,1731 @@ +# Multi-database (SQLite | PostgreSQL | MariaDB/MySQL) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Let LogstashUI 0.5.2 run on SQLite (default), PostgreSQL, or MariaDB/MySQL via discrete `LOGSTASHUI_DB_*` env vars, with a BETA sqlite→server migrator and a local Docker test matrix. + +**Architecture:** Do not add a CRUD/repository layer. Django ORM already abstracts queries. The only switch is `LogstashUI/database.py` → `DATABASES['default']`. Drivers are install extras; the Docker image installs `[databases]`. Gunicorn stays gevent. `psycopg[binary]>=3.2` cooperates with gunicorn’s gevent `patch_select` (psycopg ≥ 3.1.14); PyMySQL is the MySQL driver so the C `mysqlclient` does not block the hub. Migration is `dumpdata` / `migrate` / `loaddata` in **child processes** so Django settings never have to switch engines in-process. + +**Tech Stack:** Django 6, Python 3.12–3.14, `psycopg[binary]`, PyMySQL, gunicorn/gevent, Docker Compose for Postgres 16 + MariaDB 11 + MySQL 8.0. + +**Spec:** `docs/superpowers/specs/2026-08-21-multi-database-design.md` + +**Context:** Work on the current branch (`feat/sqleng` or whatever the operator is on). Do not rotate the product CA. Do not put Postgres/MariaDB/MySQL in the smoke compose. Do not “fix” pre-existing pytest failures (`test_update_policy_default_policy_forbidden`, `test_delete_policy_default_policy_forbidden`, `test_clone_policy_success`) unless they fail **because of** engine differences. + +Every new/edited `.py` file must keep the Elastic license header already used in this repo. + +--- + +## File map + +| File | Responsibility | +|---|---| +| `pyproject.toml` | Optional extras `[postgres]`, `[mysql]`, `[databases]` | +| `src/logstashui/LogstashUI/database.py` | Engine aliases, `DATABASES` dict, fail-fast, SSL/CONN, driver import, server version check | +| `src/logstashui/LogstashUI/migrate_engine.py` | BETA migrator: pidfile SIGTERM, WAL checkpoint, dump/load subprocesses, `--write-env` | +| `src/logstashui/LogstashUI/cli.py` | `migrate-engine` subcommand, gunicorn `--pid`, SQLite scale warning, systemd DB prompts | +| `src/logstashui/LogstashUI/wsgi.py` | Comment only: gevent+psycopg is automatic at ≥ 3.1.14 | +| `src/logstashui/LogstashUI/packaging/logstashui.default` | Documented `LOGSTASHUI_DB_*` keys | +| `src/logstashui/LogstashUI/tests/test_database.py` | Unit tests for `build_databases` / version check (no live server) | +| `src/logstashui/LogstashUI/tests/test_migrate_engine.py` | Unit tests for migrator (mocked subprocess / pid) | +| `src/logstashui/LogstashUI/tests/test_cli.py` | Parser + serve pid + warning | +| `src/logstashui/LogstashUI/tests/test_migrate_live.py` | Live dump/load; skipped unless `LOGSTASHUI_LIVE_DB=1` | +| `docker/docker-compose.db.yml` | Postgres, MariaDB, MySQL for local/CI | +| `bin/test_databases.sh` / `bin/test_databases.bat` | SQLite pytest + three-engine pytest + live migrator | +| `docker/Dockerfile` | `uv pip install '/app[databases]'` | +| `docs/docs/logstashui/configuration/environment.md` | Env table + extras + scale warning + optional PgBouncer | +| `docs/docs/logstashui/general/deploy.md` | External DB + PVC still required + offline migrate | +| `CHANGELOG.md` | 0.5.2 section | +| `.github/workflows/test-databases.yml` | CI runs `bin/test_databases.sh` | +| `scripts/generate_notice.py` | Map psycopg / PyMySQL licenses | + +No new DAO modules. No YAML. No `DATABASE_URL`. + +--- + +### Task 1: Install extras in pyproject.toml + +**Files:** +- Modify: `pyproject.toml` +- Modify: `uv.lock` (via `uv lock`) +- Modify: `scripts/generate_notice.py` (repository mappings only) + +- [ ] **Step 1: Add optional-dependencies after `[project.urls]`** + +In `pyproject.toml`, immediately after the `[project.urls]` block, insert: + +```toml +[project.optional-dependencies] +postgres = [ + "psycopg[binary]>=3.2.0", +] +mysql = [ + "PyMySQL>=1.1.1", +] +databases = [ + "psycopg[binary]>=3.2.0", + "PyMySQL>=1.1.1", +] +``` + +Do **not** add these to the default `[project].dependencies` list. Default `pip install LogstashUI` must stay SQLite-only. + +- [ ] **Step 2: Map licenses so NOTICE can mention extras** + +In `scripts/generate_notice.py`, add to `REPOSITORY_MAPPINGS`: + +```python + "psycopg": "https://github.com/psycopg/psycopg/blob/master/LICENSE.txt", + "psycopg-binary": "https://github.com/psycopg/psycopg/blob/master/LICENSE.txt", + "PyMySQL": "https://github.com/PyMySQL/PyMySQL/blob/main/LICENSE", +``` + +And to `CUSTOM_DEPENDENCIES` so extras are documented even when not in the default wheel: + +```python + "psycopg": "https://github.com/psycopg/psycopg/blob/master/LICENSE.txt", + "PyMySQL": "https://github.com/PyMySQL/PyMySQL/blob/main/LICENSE", +``` + +- [ ] **Step 3: Lock** + +Run: + +```bash +uv lock +``` + +Expected: `uv.lock` updates; no change to the default resolved production set beyond optional extra packages. + +- [ ] **Step 4: Commit** + +```bash +git add pyproject.toml uv.lock scripts/generate_notice.py +git commit -m "build: add postgres/mysql/databases install extras" +``` + +--- + +### Task 2: Failing unit tests for `build_databases` + +**Files:** +- Modify: `src/logstashui/LogstashUI/tests/test_database.py` + +- [ ] **Step 1: Replace `test_database.py` with the suite below** + +Overwrite `src/logstashui/LogstashUI/tests/test_database.py` with: + +```python +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +from pathlib import Path + +import pytest + +from LogstashUI.database import ( + build_databases, + canonical_engine, + check_server_version, +) + + +def _clear_db_env(monkeypatch): + for name in ( + "LOGSTASHUI_DB_ENGINE", + "LOGSTASHUI_DB_NAME", + "LOGSTASHUI_DB_HOST", + "LOGSTASHUI_DB_PORT", + "LOGSTASHUI_DB_USER", + "LOGSTASHUI_DB_PASSWORD", + "LOGSTASHUI_DB_SSLMODE", + "LOGSTASHUI_DB_SSL_CA", + "LOGSTASHUI_DB_CONN_MAX_AGE", + "LOGSTASHUI_DB_CONN_HEALTH_CHECKS", + ): + monkeypatch.delenv(name, raising=False) + + +def test_build_databases_sqlite_default(tmp_path, monkeypatch): + _clear_db_env(monkeypatch) + db = build_databases(tmp_path) + assert db["default"]["ENGINE"] == "django.db.backends.sqlite3" + assert db["default"]["NAME"] == tmp_path / "db.sqlite3" + assert db["default"]["OPTIONS"]["timeout"] == 20 + assert "PRAGMA journal_mode=WAL" in db["default"]["OPTIONS"]["init_command"] + + +@pytest.mark.parametrize( + "raw,expected", + [ + ("", "sqlite"), + ("sqlite", "sqlite"), + ("sqlite3", "sqlite"), + ("postgres", "postgresql"), + ("postgresql", "postgresql"), + ("mysql", "mysql"), + ("mariadb", "mysql"), + ("my", "mysql"), + ("POSTGRESQL", "postgresql"), + ], +) +def test_canonical_engine_aliases(raw, expected): + assert canonical_engine(raw) == expected + + +def test_unknown_engine_fails(tmp_path, monkeypatch): + _clear_db_env(monkeypatch) + monkeypatch.setenv("LOGSTASHUI_DB_ENGINE", "oracle") + with pytest.raises(RuntimeError, match="Unknown LOGSTASHUI_DB_ENGINE"): + build_databases(tmp_path) + + +def test_postgresql_requires_host_user(tmp_path, monkeypatch): + _clear_db_env(monkeypatch) + monkeypatch.setenv("LOGSTASHUI_DB_ENGINE", "postgresql") + monkeypatch.setattr("LogstashUI.database._import_or_raise", lambda *a, **k: None) + with pytest.raises(RuntimeError, match="LOGSTASHUI_DB_HOST"): + build_databases(tmp_path) + monkeypatch.setenv("LOGSTASHUI_DB_HOST", "db.example") + with pytest.raises(RuntimeError, match="LOGSTASHUI_DB_USER"): + build_databases(tmp_path) + + +def test_build_databases_postgresql(tmp_path, monkeypatch): + _clear_db_env(monkeypatch) + monkeypatch.setenv("LOGSTASHUI_DB_ENGINE", "postgres") + monkeypatch.setenv("LOGSTASHUI_DB_HOST", "db.example") + monkeypatch.setenv("LOGSTASHUI_DB_USER", "lsui") + monkeypatch.setenv("LOGSTASHUI_DB_PASSWORD", "s3cret") + monkeypatch.setenv("LOGSTASHUI_DB_SSLMODE", "require") + monkeypatch.setenv("LOGSTASHUI_DB_SSL_CA", "/etc/ssl/db-ca.pem") + monkeypatch.setattr("LogstashUI.database._import_or_raise", lambda *a, **k: None) + db = build_databases(tmp_path)["default"] + assert db["ENGINE"] == "django.db.backends.postgresql" + assert db["NAME"] == "logstashui" + assert db["HOST"] == "db.example" + assert db["PORT"] == "5432" + assert db["USER"] == "lsui" + assert db["PASSWORD"] == "s3cret" + assert db["CONN_MAX_AGE"] == 60 + assert db["CONN_HEALTH_CHECKS"] is True + assert db["OPTIONS"]["sslmode"] == "require" + assert db["OPTIONS"]["sslrootcert"] == "/etc/ssl/db-ca.pem" + + +def test_build_databases_mysql_mariadb_alias(tmp_path, monkeypatch): + _clear_db_env(monkeypatch) + monkeypatch.setenv("LOGSTASHUI_DB_ENGINE", "mariadb") + monkeypatch.setenv("LOGSTASHUI_DB_HOST", "127.0.0.1") + monkeypatch.setenv("LOGSTASHUI_DB_USER", "lsui") + monkeypatch.setenv("LOGSTASHUI_DB_PORT", "3307") + monkeypatch.setenv("LOGSTASHUI_DB_CONN_MAX_AGE", "0") + monkeypatch.setenv("LOGSTASHUI_DB_CONN_HEALTH_CHECKS", "false") + monkeypatch.setattr("LogstashUI.database._import_or_raise", lambda *a, **k: None) + db = build_databases(tmp_path)["default"] + assert db["ENGINE"] == "django.db.backends.mysql" + assert db["PORT"] == "3307" + assert db["CONN_MAX_AGE"] == 0 + assert db["CONN_HEALTH_CHECKS"] is False + assert db["OPTIONS"]["charset"] == "utf8mb4" + assert "utf8mb4_bin" in db["OPTIONS"]["init_command"] + assert db["TEST"]["CHARSET"] == "utf8mb4" + assert db["TEST"]["COLLATION"] == "utf8mb4_bin" + + +def test_postgresql_missing_driver(tmp_path, monkeypatch): + _clear_db_env(monkeypatch) + monkeypatch.setenv("LOGSTASHUI_DB_ENGINE", "postgresql") + monkeypatch.setenv("LOGSTASHUI_DB_HOST", "localhost") + monkeypatch.setenv("LOGSTASHUI_DB_USER", "lsui") + + def boom(module, extra): + raise RuntimeError( + f"{module} is not installed. Install with: uv pip install 'LogstashUI[{extra}]'" + ) + + monkeypatch.setattr("LogstashUI.database._import_or_raise", boom) + with pytest.raises(RuntimeError, match=r"LogstashUI\[postgres\]"): + build_databases(tmp_path) + + +def test_check_server_version_sqlite_noop(): + class Conn: + vendor = "sqlite" + + check_server_version(Conn()) + + +def test_check_server_version_postgres_too_old(): + class Conn: + vendor = "postgresql" + pg_version = 130000 + + with pytest.raises(RuntimeError, match="PostgreSQL 14"): + check_server_version(Conn()) + + +def test_check_server_version_mysql_and_mariadb(): + class Mysql: + vendor = "mysql" + mysql_is_mariadb = False + mysql_server_info = "8.0.36" + + def get_database_version(self): + return (8, 0, 36) + + check_server_version(Mysql()) + + class OldMysql: + vendor = "mysql" + mysql_is_mariadb = False + mysql_server_info = "5.7.44" + + def get_database_version(self): + return (5, 7, 44) + + with pytest.raises(RuntimeError, match="MySQL 8.0"): + check_server_version(OldMysql()) + + class Maria: + vendor = "mysql" + mysql_is_mariadb = True + mysql_server_info = "10.5.22-MariaDB" + + def get_database_version(self): + return (10, 5, 22) + + with pytest.raises(RuntimeError, match="MariaDB 10.6"): + check_server_version(Maria()) +``` + +- [ ] **Step 2: Run tests to verify they fail** + +```bash +cd /Users/buh/WORK/LogstashUI +uv run pytest src/logstashui/LogstashUI/tests/test_database.py -v --no-cov +``` + +Expected: FAIL — `canonical_engine`, `_import_or_raise`, and `check_server_version` are not defined; postgresql/mysql still raise “not implemented”. + +- [ ] **Step 3: Commit tests** + +```bash +git add src/logstashui/LogstashUI/tests/test_database.py +git commit -m "test: specify multi-engine build_databases behavior" +``` + +--- + +### Task 3: Implement `build_databases` and version check + +**Files:** +- Modify: `src/logstashui/LogstashUI/database.py` + +- [ ] **Step 1: Replace `database.py`** + +Overwrite `src/logstashui/LogstashUI/database.py` with: + +```python +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +"""Build Django DATABASES from discrete LOGSTASHUI_DB_* environment variables.""" + +from __future__ import annotations + +import os +from pathlib import Path + +from .config import env_bool + +_ENGINE_ALIASES = { + "": "sqlite", + "sqlite": "sqlite", + "sqlite3": "sqlite", + "postgres": "postgresql", + "postgresql": "postgresql", + "mysql": "mysql", + "mariadb": "mysql", + "my": "mysql", +} + +_CANONICAL = ("sqlite", "postgresql", "mysql") + + +def canonical_engine(raw: str | None) -> str: + key = (raw or "").strip().lower() + if key not in _ENGINE_ALIASES: + supported = "sqlite, postgresql, mysql (aliases: sqlite3, postgres, mariadb, my)" + raise RuntimeError( + f"Unknown LOGSTASHUI_DB_ENGINE={raw!r}. Supported: {supported}." + ) + return _ENGINE_ALIASES[key] + + +def _import_or_raise(module: str, extra: str) -> None: + try: + __import__(module) + except ImportError as exc: + raise RuntimeError( + f"{module} is not installed. Install with: uv pip install 'LogstashUI[{extra}]' " + f"(Docker/K8s image already includes LogstashUI[databases])." + ) from exc + + +def _env(name: str, default: str = "") -> str: + raw = os.environ.get(name) + if raw is None: + return default + return raw.strip() + + +def _env_int(name: str, default: int) -> int: + raw = _env(name, "") + if raw == "": + return default + return int(raw) + + +def _require(names: list[str]) -> None: + missing = [n for n in names if not _env(n)] + if missing: + raise RuntimeError( + "Missing required database settings: " + ", ".join(missing) + ) + + +def build_databases(data_dir: Path) -> dict: + engine = canonical_engine(os.environ.get("LOGSTASHUI_DB_ENGINE")) + conn_max_age = _env_int("LOGSTASHUI_DB_CONN_MAX_AGE", 60) + health = env_bool("LOGSTASHUI_DB_CONN_HEALTH_CHECKS", True) + + if engine == "sqlite": + name = _env("LOGSTASHUI_DB_NAME") + db_name = Path(name) if name else Path(data_dir) / "db.sqlite3" + return { + "default": { + "ENGINE": "django.db.backends.sqlite3", + "NAME": db_name, + "CONN_MAX_AGE": conn_max_age, + "CONN_HEALTH_CHECKS": health, + "OPTIONS": { + "init_command": ( + "PRAGMA busy_timeout=20000;" + "PRAGMA journal_mode=WAL;" + ), + "timeout": 20, + }, + } + } + + if engine == "postgresql": + _import_or_raise("psycopg", "postgres") + _require(["LOGSTASHUI_DB_HOST", "LOGSTASHUI_DB_USER"]) + sslmode = _env("LOGSTASHUI_DB_SSLMODE", "prefer") or "prefer" + options: dict = {"sslmode": sslmode} + ca = _env("LOGSTASHUI_DB_SSL_CA") + if ca: + options["sslrootcert"] = ca + return { + "default": { + "ENGINE": "django.db.backends.postgresql", + "NAME": _env("LOGSTASHUI_DB_NAME", "logstashui") or "logstashui", + "USER": _env("LOGSTASHUI_DB_USER"), + "PASSWORD": os.environ.get("LOGSTASHUI_DB_PASSWORD") or "", + "HOST": _env("LOGSTASHUI_DB_HOST"), + "PORT": _env("LOGSTASHUI_DB_PORT", "5432") or "5432", + "CONN_MAX_AGE": conn_max_age, + "CONN_HEALTH_CHECKS": health, + "OPTIONS": options, + } + } + + _import_or_raise("pymysql", "mysql") + import pymysql + + pymysql.install_as_MySQLdb() + _require(["LOGSTASHUI_DB_HOST", "LOGSTASHUI_DB_USER"]) + options = { + "charset": "utf8mb4", + "init_command": ( + "SET sql_mode='STRICT_TRANS_TABLES', " + "NAMES utf8mb4 COLLATE utf8mb4_bin" + ), + } + ca = _env("LOGSTASHUI_DB_SSL_CA") + if ca: + options["ssl"] = {"ca": ca} + return { + "default": { + "ENGINE": "django.db.backends.mysql", + "NAME": _env("LOGSTASHUI_DB_NAME", "logstashui") or "logstashui", + "USER": _env("LOGSTASHUI_DB_USER"), + "PASSWORD": os.environ.get("LOGSTASHUI_DB_PASSWORD") or "", + "HOST": _env("LOGSTASHUI_DB_HOST"), + "PORT": _env("LOGSTASHUI_DB_PORT", "3306") or "3306", + "CONN_MAX_AGE": conn_max_age, + "CONN_HEALTH_CHECKS": health, + "OPTIONS": options, + "TEST": { + "CHARSET": "utf8mb4", + "COLLATION": "utf8mb4_bin", + }, + } + } + + +def check_server_version(connection) -> None: + """Fail-fast if the server is below documented floors. No-op for SQLite.""" + vendor = getattr(connection, "vendor", "") + if vendor == "postgresql": + pg_version = int(getattr(connection, "pg_version", 0) or 0) + if pg_version and pg_version < 140000: + raise RuntimeError( + f"PostgreSQL 14+ is required (server_version_num={pg_version})." + ) + return + if vendor != "mysql": + return + is_mariadb = bool(getattr(connection, "mysql_is_mariadb", False)) + info = (getattr(connection, "mysql_server_info", "") or "").lower() + if "mariadb" in info: + is_mariadb = True + if hasattr(connection, "get_database_version"): + tup = connection.get_database_version() + else: + tup = (0, 0, 0) + major_minor = (int(tup[0]), int(tup[1])) + if is_mariadb and major_minor < (10, 6): + raise RuntimeError( + f"MariaDB 10.6+ is required (server={getattr(connection, 'mysql_server_info', tup)})." + ) + if not is_mariadb and major_minor < (8, 0): + raise RuntimeError( + f"MySQL 8.0+ is required (server={getattr(connection, 'mysql_server_info', tup)})." + ) +``` + +- [ ] **Step 2: Run unit tests** + +```bash +uv run pytest src/logstashui/LogstashUI/tests/test_database.py -v --no-cov +``` + +Expected: all PASS. + +- [ ] **Step 3: Confirm default pytest still uses sqlite** + +```bash +uv run pytest src/logstashui/LogstashUI/tests/test_database.py src/logstashui/LogstashUI/tests/test_paths.py -v --no-cov +``` + +Expected: PASS (paths tests still copy `db.sqlite3`). + +- [ ] **Step 4: Commit** + +```bash +git add src/logstashui/LogstashUI/database.py +git commit -m "feat: wire PostgreSQL and MySQL Django backends from env" +``` + +--- + +### Task 4: SQLite scale warning, gunicorn pidfile, version check on serve + +**Files:** +- Modify: `src/logstashui/LogstashUI/cli.py` +- Modify: `src/logstashui/LogstashUI/tests/test_cli.py` +- Modify: `src/logstashui/LogstashUI/wsgi.py` (comment only) + +- [ ] **Step 1: Add failing CLI tests** + +Append to `src/logstashui/LogstashUI/tests/test_cli.py`: + +```python +def test_parser_migrate_engine_requires_backup_flag(): + parser = build_parser() + ns = parser.parse_args(["migrate-engine", "--to", "postgresql"]) + assert ns.command == "migrate-engine" + assert ns.to == "postgresql" + assert ns.i_have_a_backup is False + + +def test_serve_adds_pidfile_and_warns_sqlite(monkeypatch, tmp_path, capsys): + from LogstashUI import cli + + monkeypatch.setattr(cli, "_manage", lambda argv: None) + monkeypatch.setenv("LOGSTASHUI_TLS", "false") + monkeypatch.setenv("LOGSTASHUI_DATA_DIR", str(tmp_path)) + monkeypatch.delenv("LOGSTASHUI_DB_ENGINE", raising=False) + + captured = {} + + def fake_execvp(file, args): + captured["file"] = file + captured["args"] = list(args) + raise SystemExit(0) + + monkeypatch.setattr(cli.os, "execvp", fake_execvp) + ns = Namespace(skip_migrate=True, no_tls=True, bind="127.0.0.1:8443", workers=2) + try: + cmd_serve(ns) + except SystemExit: + pass + assert "--pid" in captured["args"] + pid_idx = captured["args"].index("--pid") + assert captured["args"][pid_idx + 1].endswith("gunicorn.pid") + err = capsys.readouterr().err + assert "SQLite is the small-install default" in err +``` + +- [ ] **Step 2: Run the new tests — expect FAIL** + +```bash +uv run pytest src/logstashui/LogstashUI/tests/test_cli.py -v --no-cov +``` + +Expected: FAIL on unknown `migrate-engine` subparser and missing `--pid`. + +- [ ] **Step 3: Implement CLI pieces** + +In `build_parser()`, after the `manage` subparser, add: + +```python + migrate = sub.add_parser( + "migrate-engine", + help="BETA: copy SQLite data to PostgreSQL or MySQL (stops gunicorn)", + ) + migrate.add_argument("--to", required=True, choices=("postgresql", "mysql")) + migrate.add_argument( + "--i-have-a-backup", + dest="i_have_a_backup", + action="store_true", + help="Required. Confirms db.sqlite3 was copied aside.", + ) + migrate.add_argument("--pid", type=Path, default=None, help="gunicorn pidfile to signal") + migrate.add_argument( + "--write-env", + type=Path, + default=None, + help="Append LOGSTASHUI_DB_* to this EnvironmentFile", + ) +``` + +In `cmd_serve`, **before** building `gunicorn_cmd`: + +```python + import logging + from .database import canonical_engine, check_server_version + from .paths import resolve_data_dir + + engine = canonical_engine(os.environ.get("LOGSTASHUI_DB_ENGINE")) + if engine == "sqlite" and int(args.workers) > 1: + msg = ( + "SQLite is the small-install default; use PostgreSQL or MySQL/MariaDB " + "for concurrent agents (LOGSTASHUI_WORKERS>1)." + ) + logging.getLogger("LogstashUI").warning(msg) + print(f"WARNING: {msg}", file=sys.stderr) +``` + +After migrate (still inside `if not args.skip_migrate`), after `_manage(["migrate", "--noinput"])`: + +```python + _django_setup() + from django.db import connection + + check_server_version(connection) +``` + +When skip_migrate is true, still run version check only if Django can connect — skip if skip_migrate to keep tests simple. Version check stays tied to migrate path. + +Add pidfile to `gunicorn_cmd` after `--error-logfile`: + +```python + data_dir = resolve_data_dir() + data_dir.mkdir(parents=True, exist_ok=True) + gunicorn_cmd += ["--pid", str(data_dir / "gunicorn.pid")] +``` + +`resolve_data_dir` lives in `LogstashUI.paths` and does not import Django. Import it at the top of `cli.py`: + +```python +from .paths import resolve_data_dir +``` + +`cli.py` currently has no relative imports; it uses `from LogstashUI...` nowhere. Keep consistency with the rest of the file: use + +```python +from LogstashUI.paths import resolve_data_dir +from LogstashUI.database import canonical_engine, check_server_version +``` + +Wire `main()`: + +```python + if command == "migrate-engine": + from LogstashUI.migrate_engine import cmd_migrate_engine + + return cmd_migrate_engine(args) +``` + +For this task, `migrate_engine.py` does not exist yet. **Do not add the main() branch until Task 6.** Only add the argparse subparser so `test_parser_migrate_engine_requires_backup_flag` passes. Leave `main()` unchanged except serve. + +In `wsgi.py`, above `application = get_wsgi_application()`, add this comment (no code): + +```python +# gunicorn --worker-class gevent monkey-patches select before this module loads. +# psycopg 3.1.14+ detects that and waits cooperatively; do not use psycogreen. +``` + +- [ ] **Step 4: Re-run CLI tests** + +```bash +uv run pytest src/logstashui/LogstashUI/tests/test_cli.py -v --no-cov +``` + +Expected: PASS (migrate-engine parser exists; serve pid + warning). `main()` still errors if someone runs migrate-engine — that is Task 6. + +- [ ] **Step 5: Commit** + +```bash +git add src/logstashui/LogstashUI/cli.py src/logstashui/LogstashUI/tests/test_cli.py src/logstashui/LogstashUI/wsgi.py +git commit -m "feat: gunicorn pidfile and SQLite scale warning" +``` + +--- + +### Task 5: Docker Compose DB matrix and `bin/test_databases.sh` + +**Files:** +- Create: `docker/docker-compose.db.yml` +- Create: `bin/test_databases.sh` +- Create: `bin/test_databases.bat` +- Create: `src/logstashui/LogstashUI/tests/test_migrate_live.py` (skip unless env set — empty skip is enough this task) + +- [ ] **Step 1: Write compose file** + +Create `docker/docker-compose.db.yml`: + +```yaml +# Local/CI database matrix for LogstashUI. Not used by smoke compose. +# Host ports avoid clashing with a developer’s own 5432/3306. +name: logstashui-db-test + +services: + postgres: + image: postgres:16 + environment: + POSTGRES_USER: logstashui + POSTGRES_PASSWORD: logstashui + POSTGRES_DB: logstashui + ports: + - "55432:5432" + healthcheck: + test: ["CMD-SHELL", "pg_isready -U logstashui -d logstashui"] + interval: 2s + timeout: 5s + retries: 30 + + mariadb: + image: mariadb:11 + environment: + MARIADB_ROOT_PASSWORD: logstashui + MARIADB_DATABASE: logstashui + MARIADB_USER: logstashui + MARIADB_PASSWORD: logstashui + command: --character-set-server=utf8mb4 --collation-server=utf8mb4_bin + ports: + - "53306:3306" + healthcheck: + test: ["CMD", "healthcheck.sh", "--connect", "--innodb_initialized"] + interval: 3s + timeout: 5s + retries: 40 + + mysql: + image: mysql:8.0 + environment: + MYSQL_ROOT_PASSWORD: logstashui + MYSQL_DATABASE: logstashui + MYSQL_USER: logstashui + MYSQL_PASSWORD: logstashui + command: --character-set-server=utf8mb4 --collation-server=utf8mb4_bin + ports: + - "53307:3306" + healthcheck: + test: ["CMD", "mysqladmin", "ping", "-h", "127.0.0.1", "-ulogstashui", "-plogstashui"] + interval: 3s + timeout: 5s + retries: 40 +``` + +pytest-django must CREATE DATABASE. Postgres `POSTGRES_USER` is superuser. MariaDB/MySQL app users are not. **For the mysql/mariadb pytest runs, use root / logstashui** so Django can create `test_logstashui`. + +- [ ] **Step 2: Write `bin/test_databases.sh`** + +```bash +#!/usr/bin/env bash +# Run SQLite pytest, then the same suite against Postgres, MariaDB, and MySQL, +# then live dump/load tests. Requires Docker. Default `pytest` does not. +set -euo pipefail + +ROOT="$(cd "$(dirname "$0")/.." && pwd)" +cd "$ROOT" +COMPOSE=(docker compose -f docker/docker-compose.db.yml) +KEEP=0 +if [[ "${1:-}" == "--keep" ]]; then + KEEP=1 +fi + +if ! command -v docker >/dev/null; then + echo "ERROR: Docker is required for bin/test_databases.sh" >&2 + echo "Default pytest (SQLite) still works: uv run pytest" >&2 + exit 1 +fi + +uv sync --extra databases --group dev + +echo "==> SQLite pytest (no compose)" +uv run pytest src/logstashui --no-cov + +echo "==> Starting Postgres / MariaDB / MySQL" +"${COMPOSE[@]}" up -d --wait + +run_engine () { + local name="$1" + shift + echo "==> pytest on ${name}" + env "$@" uv run pytest src/logstashui --no-cov +} + +run_engine postgresql \ + LOGSTASHUI_DB_ENGINE=postgresql \ + LOGSTASHUI_DB_HOST=127.0.0.1 \ + LOGSTASHUI_DB_PORT=55432 \ + LOGSTASHUI_DB_NAME=logstashui \ + LOGSTASHUI_DB_USER=logstashui \ + LOGSTASHUI_DB_PASSWORD=logstashui + +run_engine mariadb \ + LOGSTASHUI_DB_ENGINE=mysql \ + LOGSTASHUI_DB_HOST=127.0.0.1 \ + LOGSTASHUI_DB_PORT=53306 \ + LOGSTASHUI_DB_NAME=logstashui \ + LOGSTASHUI_DB_USER=root \ + LOGSTASHUI_DB_PASSWORD=logstashui + +run_engine mysql \ + LOGSTASHUI_DB_ENGINE=mysql \ + LOGSTASHUI_DB_HOST=127.0.0.1 \ + LOGSTASHUI_DB_PORT=53307 \ + LOGSTASHUI_DB_NAME=logstashui \ + LOGSTASHUI_DB_USER=root \ + LOGSTASHUI_DB_PASSWORD=logstashui + +echo "==> Live migrator tests" +env LOGSTASHUI_LIVE_DB=1 \ + LOGSTASHUI_LIVE_PG_PORT=55432 \ + LOGSTASHUI_LIVE_MARIA_PORT=53306 \ + LOGSTASHUI_LIVE_MYSQL_PORT=53307 \ + LOGSTASHUI_LIVE_DB_USER=root \ + LOGSTASHUI_LIVE_DB_PASSWORD=logstashui \ + LOGSTASHUI_LIVE_PG_USER=logstashui \ + uv run pytest src/logstashui/LogstashUI/tests/test_migrate_live.py -v --no-cov + +if [[ "$KEEP" -eq 0 ]]; then + "${COMPOSE[@]}" down -v +fi +``` + +`chmod +x bin/test_databases.sh` + +Until Task 7, `test_migrate_live.py` should skip (no `LOGSTASHUI_LIVE_DB` assertions yet — see Step 3). Running the live file with the env set will collect 0 tests or skip. Create a placeholder that skips: + +```python +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +import os + +import pytest + +pytestmark = pytest.mark.skipif( + os.environ.get("LOGSTASHUI_LIVE_DB") != "1", + reason="set LOGSTASHUI_LIVE_DB=1 (bin/test_databases.sh)", +) + + +def test_live_placeholder(): + pytest.skip("migrator live tests land in Task 7") +``` + +- [ ] **Step 3: Write `bin/test_databases.bat`** + +```bat +@echo off +setlocal +cd /d "%~dp0\.." +where docker >nul 2>&1 +if errorlevel 1 ( + echo ERROR: Docker is required for bin\test_databases.bat + exit /b 1 +) +uv sync --extra databases --group dev +uv run pytest src\logstashui --no-cov +if errorlevel 1 exit /b 1 +docker compose -f docker\docker-compose.db.yml up -d --wait +if errorlevel 1 exit /b 1 + +set LOGSTASHUI_DB_ENGINE=postgresql +set LOGSTASHUI_DB_HOST=127.0.0.1 +set LOGSTASHUI_DB_PORT=55432 +set LOGSTASHUI_DB_NAME=logstashui +set LOGSTASHUI_DB_USER=logstashui +set LOGSTASHUI_DB_PASSWORD=logstashui +uv run pytest src\logstashui --no-cov +if errorlevel 1 goto :down + +set LOGSTASHUI_DB_ENGINE=mysql +set LOGSTASHUI_DB_PORT=53306 +set LOGSTASHUI_DB_USER=root +uv run pytest src\logstashui --no-cov +if errorlevel 1 goto :down + +set LOGSTASHUI_DB_PORT=53307 +uv run pytest src\logstashui --no-cov +if errorlevel 1 goto :down + +set LOGSTASHUI_LIVE_DB=1 +set LOGSTASHUI_LIVE_PG_PORT=55432 +set LOGSTASHUI_LIVE_MARIA_PORT=53306 +set LOGSTASHUI_LIVE_MYSQL_PORT=53307 +uv run pytest src\logstashui\LogstashUI\tests\test_migrate_live.py -v --no-cov + +:down +if /I not "%1"=="--keep" docker compose -f docker\docker-compose.db.yml down -v +``` + +- [ ] **Step 4: Smoke the compose file (not the full suite if too long — at least `up --wait`)** + +```bash +docker compose -f docker/docker-compose.db.yml up -d --wait +docker compose -f docker/docker-compose.db.yml ps +docker compose -f docker/docker-compose.db.yml down -v +``` + +Expected: three services healthy, then removed. + +Then run **one** engine pytest if time allows: + +```bash +uv sync --extra databases --group dev +LOGSTASHUI_DB_ENGINE=postgresql LOGSTASHUI_DB_HOST=127.0.0.1 LOGSTASHUI_DB_PORT=55432 \ + LOGSTASHUI_DB_NAME=logstashui LOGSTASHUI_DB_USER=logstashui LOGSTASHUI_DB_PASSWORD=logstashui \ + uv run pytest src/logstashui/LogstashUI/tests/test_database.py src/logstashui/Site/tests/test_views.py::test_health_check_returns_200 --no-cov +``` + +(Bring compose back up first.) Expected: PASS. If `test_health_check` fails on migrate, fix charset/user and re-run. Known CRUD failures on sqlite may also appear on postgres — do not xfail them in this task. + +- [ ] **Step 5: Commit** + +```bash +git add docker/docker-compose.db.yml bin/test_databases.sh bin/test_databases.bat \ + src/logstashui/LogstashUI/tests/test_migrate_live.py +git commit -m "test: Docker Postgres/MariaDB/MySQL matrix script" +``` + +--- + +### Task 6: BETA `migrate-engine` (unit-tested, no live servers) + +**Files:** +- Create: `src/logstashui/LogstashUI/migrate_engine.py` +- Create: `src/logstashui/LogstashUI/tests/test_migrate_engine.py` +- Modify: `src/logstashui/LogstashUI/cli.py` (`main()` branch) + +- [ ] **Step 1: Write failing unit tests** + +Create `src/logstashui/LogstashUI/tests/test_migrate_engine.py`: + +```python +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +from argparse import Namespace +from pathlib import Path + +import pytest + +from LogstashUI import migrate_engine as me + + +def test_refuses_without_backup_flag(capsys): + ns = Namespace(to="postgresql", i_have_a_backup=False, pid=None, write_env=None) + with pytest.raises(SystemExit) as exc: + me.cmd_migrate_engine(ns) + assert exc.value.code == 2 + assert "back up" in capsys.readouterr().err.lower() + + +def test_refuses_sqlite_target(tmp_path, monkeypatch, capsys): + monkeypatch.setenv("LOGSTASHUI_DATA_DIR", str(tmp_path)) + ns = Namespace(to="sqlite", i_have_a_backup=True, pid=None, write_env=None) + with pytest.raises(SystemExit): + me.cmd_migrate_engine(ns) + + +def test_refuses_missing_sqlite_file(tmp_path, monkeypatch, capsys): + monkeypatch.setenv("LOGSTASHUI_DATA_DIR", str(tmp_path)) + ns = Namespace(to="postgresql", i_have_a_backup=True, pid=None, write_env=None) + with pytest.raises(SystemExit) as exc: + me.cmd_migrate_engine(ns) + assert exc.value.code == 1 + assert "db.sqlite3" in capsys.readouterr().err + + +def test_stop_pid_sends_sigterm(tmp_path, monkeypatch): + pidfile = tmp_path / "gunicorn.pid" + pidfile.write_text("4242\n") + sent = {} + calls = {"n": 0} + + def kill_then_gone(pid, sig): + calls["n"] += 1 + if calls["n"] == 1: + sent["pid"] = pid + sent["sig"] = sig + return + raise ProcessLookupError() + + monkeypatch.setattr(me.os, "kill", kill_then_gone) + monkeypatch.setattr(me.time, "sleep", lambda s: None) + me.stop_gunicorn(pidfile) + assert sent["pid"] == 4242 + assert sent["sig"] == me.signal.SIGTERM + + +def test_write_env_appends(tmp_path, monkeypatch): + envf = tmp_path / "logstashui.default" + envf.write_text("LOGSTASHUI_DATA_DIR=/var/lib/logstashui\n") + monkeypatch.setenv("LOGSTASHUI_DB_HOST", "db.example") + monkeypatch.setenv("LOGSTASHUI_DB_USER", "lsui") + monkeypatch.setenv("LOGSTASHUI_DB_NAME", "logstashui") + me.write_env_file(envf, "postgresql") + text = envf.read_text() + assert "LOGSTASHUI_DB_ENGINE=postgresql" in text + assert "LOGSTASHUI_DB_HOST=db.example" in text + assert "PASSWORD" not in text +``` + +- [ ] **Step 2: Run — expect FAIL (module missing)** + +```bash +uv run pytest src/logstashui/LogstashUI/tests/test_migrate_engine.py -v --no-cov +``` + +- [ ] **Step 3: Implement `migrate_engine.py`** + +```python +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +"""BETA sqlite → PostgreSQL/MySQL copy via dumpdata/loaddata in child processes.""" + +from __future__ import annotations + +import os +import signal +import sqlite3 +import subprocess +import sys +import time +from pathlib import Path + +from LogstashUI.database import canonical_engine +from LogstashUI.paths import resolve_data_dir + +_DUMP_EXCLUDE = ["contenttypes", "auth.permission", "sessions"] +_SEQUENCE_APPS = [ + "admin", + "auth", + "PipelineManager", + "Management", + "SNMP", + "AI", + "Monitoring", + "Site", +] + + +def cmd_migrate_engine(args) -> int: + if not getattr(args, "i_have_a_backup", False): + print( + "BETA migrate-engine: copy DATA_DIR/db.sqlite3 to a backup first, " + "then re-run with --i-have-a-backup.", + file=sys.stderr, + ) + raise SystemExit(2) + + target = canonical_engine(getattr(args, "to", "")) + if target not in ("postgresql", "mysql"): + print(" --to must be postgresql or mysql", file=sys.stderr) + raise SystemExit(2) + + data_dir = resolve_data_dir(migrate_legacy=False) + sqlite_path = data_dir / "db.sqlite3" + if not sqlite_path.is_file(): + print(f"No SQLite database at {sqlite_path}", file=sys.stderr) + raise SystemExit(1) + + print( + "BETA: migrate-engine stops gunicorn (UI port down), dumps SQLite, " + "loads the target, and does not restart serve.", + file=sys.stderr, + ) + + pidfile = args.pid or (data_dir / "gunicorn.pid") + if Path(pidfile).is_file(): + stop_gunicorn(Path(pidfile)) + + wal_checkpoint(sqlite_path) + dump_path = data_dir / "migrate-engine-dump.json" + run_manage( + [ + "dumpdata", + "--natural-foreign", + "--natural-primary", + "--output", + str(dump_path), + *[f"-e={item}" for item in _DUMP_EXCLUDE], + ], + extra_env={"LOGSTASHUI_DB_ENGINE": "sqlite"}, + ) + run_manage(["migrate", "--noinput"], extra_env={"LOGSTASHUI_DB_ENGINE": target}) + run_manage(["loaddata", str(dump_path)], extra_env={"LOGSTASHUI_DB_ENGINE": target}) + if target == "postgresql": + reset_postgres_sequences(extra_env={"LOGSTASHUI_DB_ENGINE": target}) + + if args.write_env: + write_env_file(Path(args.write_env), target) + + print( + "Done. Keep LOGSTASHUI_DB_ENGINE=" + f"{target} and start LogstashUI (systemctl start logstashui). " + "Do not auto-restart: systemd Restart= would race." + ) + return 0 + + +def wal_checkpoint(sqlite_path: Path) -> None: + conn = sqlite3.connect(str(sqlite_path)) + try: + conn.execute("PRAGMA wal_checkpoint(TRUNCATE)") + finally: + conn.close() + + +def stop_gunicorn(pidfile: Path) -> None: + raw = pidfile.read_text(encoding="utf-8").strip() + if not raw.isdigit(): + print(f"Invalid pidfile {pidfile}", file=sys.stderr) + raise SystemExit(1) + pid = int(raw) + try: + os.kill(pid, signal.SIGTERM) + except ProcessLookupError: + pidfile.unlink(missing_ok=True) + return + deadline = time.time() + 30 + while time.time() < deadline: + try: + os.kill(pid, 0) + except ProcessLookupError: + pidfile.unlink(missing_ok=True) + return + time.sleep(0.2) + print(f"gunicorn pid {pid} did not exit after SIGTERM", file=sys.stderr) + raise SystemExit(1) + + +def run_manage(argv: list[str], extra_env: dict[str, str]) -> None: + env = os.environ.copy() + env.update(extra_env) + env.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") + code = ( + "import sys; from django.core.management import execute_from_command_line; " + "execute_from_command_line(['logstashui'] + sys.argv[1:])" + ) + cmd = [sys.executable, "-c", code, *argv] + proc = subprocess.run(cmd, env=env, check=False) + if proc.returncode != 0: + raise SystemExit(proc.returncode) + + +def reset_postgres_sequences(extra_env: dict[str, str]) -> None: + env = os.environ.copy() + env.update(extra_env) + env.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") + code = ( + "import sys; from django.core.management import execute_from_command_line; " + "execute_from_command_line(['logstashui', 'sqlsequencereset'] + sys.argv[1:])" + ) + sql = subprocess.run( + [sys.executable, "-c", code, *_SEQUENCE_APPS], + env=env, + check=False, + capture_output=True, + text=True, + ) + if sql.returncode != 0: + print(sql.stderr, file=sys.stderr) + raise SystemExit(sql.returncode) + if not sql.stdout.strip(): + return + dbshell = subprocess.run( + [ + sys.executable, + "-c", + "import sys; from django.core.management import execute_from_command_line; " + "execute_from_command_line(['logstashui', 'dbshell'])", + ], + env=env, + input=sql.stdout, + text=True, + check=False, + ) + if dbshell.returncode != 0: + raise SystemExit(dbshell.returncode) + + +def write_env_file(path: Path, engine: str) -> None: + lines = [ + f"LOGSTASHUI_DB_ENGINE={engine}", + f"LOGSTASHUI_DB_NAME={os.environ.get('LOGSTASHUI_DB_NAME', 'logstashui')}", + f"LOGSTASHUI_DB_HOST={os.environ.get('LOGSTASHUI_DB_HOST', '')}", + f"LOGSTASHUI_DB_PORT={os.environ.get('LOGSTASHUI_DB_PORT', '')}", + f"LOGSTASHUI_DB_USER={os.environ.get('LOGSTASHUI_DB_USER', '')}", + ] + existing = path.read_text(encoding="utf-8") if path.is_file() else "" + path.write_text(existing.rstrip() + "\n\n# migrate-engine\n" + "\n".join(lines) + "\n") +``` + +`test_refuses_sqlite_target`: `canonical_engine("sqlite")` returns sqlite, then `target not in (...)` exits 2. argparse `choices` already blocks sqlite in CLI; the function still guards. + +`test_stop_pid_sends_sigterm` uses `ProcessLookupError` on the wait loop — `os.kill(pid, 0)` raises, success. + +Fix the test’s double-setattr: keep only `kill_then_gone`. + +- [ ] **Step 4: Wire `main()` in `cli.py`** + +```python + if command == "migrate-engine": + from LogstashUI.migrate_engine import cmd_migrate_engine + + return cmd_migrate_engine(args) +``` + +- [ ] **Step 5: Run unit tests** + +```bash +uv run pytest src/logstashui/LogstashUI/tests/test_migrate_engine.py src/logstashui/LogstashUI/tests/test_cli.py -v --no-cov +``` + +Expected: PASS. If `test_refuses_sqlite_target` never hits `cmd_migrate_engine` because argparse isn’t used (Namespace to=sqlite), the guard in `cmd_migrate_engine` handles it. + +- [ ] **Step 6: Commit** + +```bash +git add src/logstashui/LogstashUI/migrate_engine.py \ + src/logstashui/LogstashUI/tests/test_migrate_engine.py \ + src/logstashui/LogstashUI/cli.py +git commit -m "feat: BETA migrate-engine sqlite to postgres/mysql" +``` + +--- + +### Task 7: Live migration tests + +**Files:** +- Modify: `src/logstashui/LogstashUI/tests/test_migrate_live.py` + +- [ ] **Step 1: Replace the placeholder with subprocess live tests** + +Live tests must not use `@pytest.mark.django_db` and then switch `LOGSTASHUI_DB_ENGINE` in-process (Django settings are frozen). Use isolated `DATA_DIR` + child processes only. + +Overwrite `src/logstashui/LogstashUI/tests/test_migrate_live.py` with: + +```python +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +from __future__ import annotations + +import os +import subprocess +import sys +from argparse import Namespace +from pathlib import Path + +import pytest + +from LogstashUI.migrate_engine import cmd_migrate_engine, run_manage + +pytestmark = pytest.mark.skipif( + os.environ.get("LOGSTASHUI_LIVE_DB") != "1", + reason="set LOGSTASHUI_LIVE_DB=1 (bin/test_databases.sh)", +) + + +def _python_manage(args: list[str], env: dict[str, str]) -> None: + run_manage(args, extra_env=env) + + +def _seed(data_dir: Path) -> None: + env = { + "LOGSTASHUI_DATA_DIR": str(data_dir), + "LOGSTASHUI_DB_ENGINE": "sqlite", + "DJANGO_SETTINGS_MODULE": "LogstashUI.settings", + } + _python_manage(["migrate", "--noinput"], env) + code = ( + "import os, django; os.environ.setdefault('DJANGO_SETTINGS_MODULE','LogstashUI.settings'); " + "django.setup(); " + "from django.contrib.auth.models import User; " + "from PipelineManager.models import Policy; " + "User.objects.create_user('migrate-user', password='x'); " + "Policy.objects.create(name='Migrate Policy', logstash_yml='node.name: t', " + "jvm_options='#', log4j2_properties='#')" + ) + env_full = os.environ.copy() + env_full.update(env) + subprocess.run([sys.executable, "-c", code], env=env_full, check=True) + + +def _count(env: dict[str, str]) -> tuple[int, int]: + env_full = os.environ.copy() + env_full.update(env) + env_full.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") + code = ( + "import os, django; os.environ.setdefault('DJANGO_SETTINGS_MODULE','LogstashUI.settings'); " + "django.setup(); " + "from django.contrib.auth.models import User; " + "from PipelineManager.models import Policy; " + "print(User.objects.filter(username='migrate-user').count()); " + "print(Policy.objects.filter(name='Migrate Policy').count())" + ) + out = subprocess.run( + [sys.executable, "-c", code], env=env_full, check=True, capture_output=True, text=True + ) + lines = [ln for ln in out.stdout.splitlines() if ln.strip().isdigit()] + return int(lines[-2]), int(lines[-1]) + + +def _run_to(tmp_path: Path, engine: str, port: str, user: str) -> None: + data_dir = tmp_path / "data" + data_dir.mkdir() + os.environ["LOGSTASHUI_DATA_DIR"] = str(data_dir) + _seed(data_dir) + target = { + "LOGSTASHUI_DB_ENGINE": engine, + "LOGSTASHUI_DB_HOST": "127.0.0.1", + "LOGSTASHUI_DB_PORT": port, + "LOGSTASHUI_DB_NAME": "logstashui_migrate", + "LOGSTASHUI_DB_USER": user, + "LOGSTASHUI_DB_PASSWORD": os.environ.get("LOGSTASHUI_LIVE_DB_PASSWORD", "logstashui"), + } + for k, v in target.items(): + os.environ[k] = v + ns = Namespace(to="postgresql" if engine == "postgresql" else "mysql", i_have_a_backup=True, pid=None, write_env=None) + cmd_migrate_engine(ns) + users, policies = _count(target | {"LOGSTASHUI_DATA_DIR": str(data_dir)}) + assert users == 1 + assert policies == 1 + + +def test_live_postgres(tmp_path, monkeypatch): + monkeypatch.setenv("LOGSTASHUI_DATA_DIR", str(tmp_path / "data")) + _run_to( + tmp_path, + "postgresql", + os.environ.get("LOGSTASHUI_LIVE_PG_PORT", "55432"), + os.environ.get("LOGSTASHUI_LIVE_PG_USER", "logstashui"), + ) + + +def test_live_mariadb(tmp_path, monkeypatch): + _run_to( + tmp_path, + "mysql", + os.environ.get("LOGSTASHUI_LIVE_MARIA_PORT", "53306"), + os.environ.get("LOGSTASHUI_LIVE_DB_USER", "root"), + ) + + +def test_live_mysql(tmp_path, monkeypatch): + _run_to( + tmp_path, + "mysql", + os.environ.get("LOGSTASHUI_LIVE_MYSQL_PORT", "53307"), + os.environ.get("LOGSTASHUI_LIVE_DB_USER", "root"), + ) +``` + +Create the target database `logstashui_migrate` in each engine before load. Add a helper at the top of `_run_to` that uses `run_manage(["migrate", "--noinput"], target)` which `cmd_migrate_engine` already does. Postgres cannot connect if the DB name does not exist. + +**Create `logstashui_migrate` in compose** by adding a second database via init SQL. + +Add `docker/db-init/postgres-extra.sql`: + +```sql +CREATE DATABASE logstashui_migrate OWNER logstashui; +``` + +Mount it in compose under postgres: + +```yaml + volumes: + - ./db-init/postgres-extra.sql:/docker-entrypoint-initdb.d/02-migrate.sql:ro +``` + +For MariaDB/MySQL, `docker/db-init/mysql-extra.sql`: + +```sql +CREATE DATABASE IF NOT EXISTS logstashui_migrate CHARACTER SET utf8mb4 COLLATE utf8mb4_bin; +GRANT ALL ON logstashui_migrate.* TO 'logstashui'@'%'; +GRANT ALL ON logstashui_migrate.* TO 'root'@'%'; +``` + +Mount on **both** mariadb and mysql services as `/docker-entrypoint-initdb.d/02-migrate.sql`. + +If compose was already created without the volume, `down -v` so init runs again. + +- [ ] **Step 2: Run live tests** + +```bash +./bin/test_databases.sh +``` + +Expected: SQLite pytest completes; three engine pytest runs; three live migrator tests PASS. Pre-existing CRUD failures may still fail the **full** suite — if they fail on sqlite they fail on all engines. Do not change those tests. If the script must be CI-green despite them, do **not** hide them with xfail; report in the PR. Optionally restrict engine runs to `LogstashUI/tests` + `Site/tests/test_views.py::test_health_check_returns_200` **only if** the full suite is blocked by those known failures **and** they fail the same way on sqlite. Prefer full suite. + +- [ ] **Step 3: Commit** + +```bash +git add src/logstashui/LogstashUI/tests/test_migrate_live.py docker/docker-compose.db.yml docker/db-init +git commit -m "test: live sqlite dump/load onto Postgres MariaDB MySQL" +``` + +--- + +### Task 8: systemd prompts, sample env, operator docs, CHANGELOG, Docker image, CI + +**Files:** +- Modify: `src/logstashui/LogstashUI/packaging/logstashui.default` +- Modify: `src/logstashui/LogstashUI/cli.py` (`install_systemd`, `render_default_env`) +- Modify: `src/logstashui/LogstashUI/tests/test_cli.py` (`test_systemd_dry_run_writes_unit_and_default`) +- Modify: `docs/docs/logstashui/configuration/environment.md` +- Modify: `docs/docs/logstashui/general/deploy.md` +- Modify: `CHANGELOG.md` +- Modify: `docker/Dockerfile` +- Create: `.github/workflows/test-databases.yml` + +- [ ] **Step 1: Sample env** + +Replace the “Future database backends” block in `logstashui.default` with: + +``` +# Database. Unset engine = sqlite at $LOGSTASHUI_DATA_DIR/db.sqlite3. +# DATA_DIR is still required for TLS, secrets, logs, staticfiles. +# LOGSTASHUI_DB_ENGINE=sqlite +# LOGSTASHUI_DB_NAME=logstashui +# LOGSTASHUI_DB_HOST= +# LOGSTASHUI_DB_PORT= +# LOGSTASHUI_DB_USER= +# LOGSTASHUI_DB_PASSWORD= +# LOGSTASHUI_DB_SSLMODE=prefer +# LOGSTASHUI_DB_SSL_CA= +# LOGSTASHUI_DB_CONN_MAX_AGE=60 +# LOGSTASHUI_DB_CONN_HEALTH_CHECKS=true +# +# Native extras: uv pip install 'LogstashUI[postgres]' or 'LogstashUI[mysql]' +# or 'LogstashUI[databases]'. The container image already includes both drivers. +# +# Create MySQL/MariaDB with utf8mb4_bin: +# CREATE DATABASE logstashui CHARACTER SET utf8mb4 COLLATE utf8mb4_bin; +``` + +- [ ] **Step 2: systemd generator** + +Add optional kwargs to `install_systemd` and `render_default_env`: `db_engine=""`, `db_host=""`, `db_name=""`, `db_user=""`, `db_port=""`. + +In the interactive block, after `no_auth` prompt: + +```python + db_engine = _prompt("LOGSTASHUI_DB_ENGINE (sqlite/postgresql/mysql)", db_engine or "sqlite") + if canonical_engine(db_engine) != "sqlite": + db_host = _prompt("LOGSTASHUI_DB_HOST", db_host) + db_port = _prompt("LOGSTASHUI_DB_PORT", db_port) + db_name = _prompt("LOGSTASHUI_DB_NAME", db_name or "logstashui") + db_user = _prompt("LOGSTASHUI_DB_USER", db_user) +``` + +Do not prompt for password (operator edits the EnvironmentFile / Secret). + +In `render_default_env`, if `canonical_engine(db_engine) != "sqlite"` append extras like the other optional keys: + +```python + if db_engine and canonical_engine(db_engine) != "sqlite": + extras.append(f"LOGSTASHUI_DB_ENGINE={canonical_engine(db_engine)}") + if db_host: + extras.append(f"LOGSTASHUI_DB_HOST={db_host}") + if db_port: + extras.append(f"LOGSTASHUI_DB_PORT={db_port}") + if db_name: + extras.append(f"LOGSTASHUI_DB_NAME={db_name}") + if db_user: + extras.append(f"LOGSTASHUI_DB_USER={db_user}") +``` + +Pass the new kwargs from `cmd_systemd` / `install_systemd` into `render_default_env`. Keep `test_systemd_dry_run_writes_unit_and_default` working (defaults empty → sqlite comments only). + +- [ ] **Step 3: Docs** + +In `docs/docs/logstashui/configuration/environment.md`, replace the “Database (sqlite only)” section with: + +```markdown +## Database + +| Variable | Default | Purpose | +|---|---|---| +| `LOGSTASHUI_DB_ENGINE` | `sqlite` | `sqlite`, `postgresql`, or `mysql` (MariaDB uses `mysql`). Aliases: `sqlite3`, `postgres`, `mariadb`, `my` | +| `LOGSTASHUI_DB_NAME` | sqlite: `$LOGSTASHUI_DATA_DIR/db.sqlite3`; else `logstashui` | Database name / sqlite path | +| `LOGSTASHUI_DB_HOST` | empty | **Required** for postgresql/mysql | +| `LOGSTASHUI_DB_PORT` | `5432` / `3306` | | +| `LOGSTASHUI_DB_USER` | empty | **Required** for postgresql/mysql | +| `LOGSTASHUI_DB_PASSWORD` | empty | Put in a Secret / `chmod 640` EnvironmentFile | +| `LOGSTASHUI_DB_SSLMODE` | postgres: `prefer` | `disable` `allow` `prefer` `require` `verify-ca` `verify-full` | +| `LOGSTASHUI_DB_SSL_CA` | empty | CA file for mysql TLS and postgres `verify-*` | +| `LOGSTASHUI_DB_CONN_MAX_AGE` | `60` | Persistent connections (seconds); `0` closes per request | +| `LOGSTASHUI_DB_CONN_HEALTH_CHECKS` | `true` | Django `CONN_HEALTH_CHECKS` | + +Floors: PostgreSQL 14+, MariaDB 10.6+, MySQL 8.0+. Create MySQL/MariaDB as `utf8mb4` / `utf8mb4_bin` so unique names match SQLite/Postgres case-sensitivity. + +**Install extras (native pip/uv):** `uv pip install 'LogstashUI[postgres]'`, `'LogstashUI[mysql]'`, or `'LogstashUI[databases]'`. The Docker/K8s image already installs `[databases]`. Missing driver fails at startup with that extra name. + +`LOGSTASHUI_DATA_DIR` is still required when the database is remote (TLS, `.django_secret_key`, logs, staticfiles). + +**SQLite scale:** `logstashui serve` logs a warning when engine is sqlite and `LOGSTASHUI_WORKERS` > 1. Use PostgreSQL or MySQL/MariaDB for concurrent agents. Startup still succeeds. + +**Connections:** gunicorn remains gevent (`--worker-connections 1000`). Keep `LOGSTASHUI_WORKERS` × in-flight requests under the server `max_connections`. PgBouncer (or equivalent) is optional, not required. + +No `DATABASE_URL`. No YAML. + +### Offline migration (supported) + +1. `systemctl stop logstashui` (or stop the container). +2. Copy `$LOGSTASHUI_DATA_DIR/db.sqlite3` somewhere safe. Keep the rest of `DATA_DIR` (same Django secret key). +3. Create the server database (`utf8mb4_bin` on MySQL/MariaDB). +4. Set `LOGSTASHUI_DB_*` for the target. Native installs need the matching extra. +5. `logstashui manage dumpdata --natural-foreign --natural-primary -e contenttypes -e auth.permission -e sessions -o dump.json` while still on sqlite, **or** use the BETA CLI below. +6. `logstashui manage migrate --noinput && logstashui manage loaddata dump.json` +7. Postgres: `logstashui manage sqlsequencereset PipelineManager Management SNMP AI auth admin | logstashui manage dbshell` +8. Start LogstashUI. Log in again (sessions were not copied). + +### BETA CLI + +```bash +# env already points at the empty target server; sqlite file still in DATA_DIR +sudo systemctl stop logstashui # avoid Restart= racing SIGTERM +logstashui migrate-engine --to postgresql --i-have-a-backup +# optional: --write-env /etc/default/logstashui +sudo systemctl start logstashui +``` + +`--to mysql` covers MariaDB and MySQL. The command SIGTERMs gunicorn if `$LOGSTASHUI_DATA_DIR/gunicorn.pid` is live, checkpoints WAL, dump/load, and **does not** restart serve. +``` + +In `deploy.md` Data directory paragraph, after the sqlite sentence, add: the database may be external Postgres/MySQL; the PVC/bind-mount is still required for TLS and secrets. In Kubernetes minimum list, add ConfigMap `LOGSTASHUI_DB_ENGINE/HOST/NAME/USER` + Secret `LOGSTASHUI_DB_PASSWORD`. + +- [ ] **Step 4: Dockerfile** + +Change: + +``` +RUN uv pip install --system --no-cache /app +``` + +to: + +``` +RUN uv pip install --system --no-cache "/app[databases]" +``` + +Leave `sqlite3` apt package (debug/sqlite CLI in the image is fine). + +- [ ] **Step 5: CHANGELOG** + +Insert at the top of `CHANGELOG.md`: + +```markdown +## [0.5.2] - Multi-database + +Package version remains **0.5.1** until release tagging; this documents the 0.5.2 database work. + +- `LOGSTASHUI_DB_ENGINE=sqlite|postgresql|mysql` (MariaDB uses `mysql`). Discrete `LOGSTASHUI_DB_HOST/PORT/NAME/USER/PASSWORD` plus SSL and `CONN_MAX_AGE`. No YAML, no `DATABASE_URL`. +- Default is still SQLite. `logstashui serve` warns when `LOGSTASHUI_WORKERS>1` on SQLite. +- Native extras: `LogstashUI[postgres]`, `[mysql]`, `[databases]`. Container image installs `[databases]`. +- BETA `logstashui migrate-engine --to postgresql|mysql --i-have-a-backup` copies SQLite → server (stops gunicorn; does not restart). +- `bin/test_databases.sh` runs the suite and dump/load against local Docker Postgres, MariaDB, and MySQL. +``` + +Do not bump `pyproject.toml` version in this plan unless the operator asks; the spec is “0.5.2 work”. + +- [ ] **Step 6: CI workflow** + +Create `.github/workflows/test-databases.yml`: + +```yaml +name: Database matrix + +on: + pull_request: + push: + branches: [main, master] + +jobs: + db: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: astral-sh/setup-uv@v4 + with: + python-version: "3.12" + - uses: actions/setup-python@v5 + with: + python-version: "3.12" + - name: Run database matrix + run: bash bin/test_databases.sh +``` + +- [ ] **Step 7: Tests for systemd extras** + +Extend `test_systemd_dry_run_writes_unit_and_default` **or** add: + +```python +def test_systemd_env_includes_postgres_when_passed(tmp_path): + result = install_systemd( + output_dir=tmp_path, + exec_start="/usr/bin/logstashui serve", + user="logstashui", + group="logstashui", + data_dir="/var/lib/logstashui", + bind="0.0.0.0:8443", + workers=2, + allowed_hosts="*", + csrf_trusted_origins="", + tls="true", + host_hostname="", + host_ips="", + tls_sans="", + agent_ui_url="", + no_auth="false", + dry_run=True, + db_engine="postgresql", + db_host="db.example", + db_port="5432", + db_name="logstashui", + db_user="lsui", + ) + text = (tmp_path / "logstashui.default").read_text() + assert "LOGSTASHUI_DB_ENGINE=postgresql" in text + assert "LOGSTASHUI_DB_HOST=db.example" in text +``` + +Update `install_systemd` signature with those kwargs defaulting to `""`. + +- [ ] **Step 8: Run focused tests + default sqlite pytest slice** + +```bash +uv run pytest src/logstashui/LogstashUI/tests/test_cli.py src/logstashui/LogstashUI/tests/test_database.py src/logstashui/LogstashUI/tests/test_migrate_engine.py -v --no-cov +``` + +Expected: PASS. + +- [ ] **Step 9: Commit** + +```bash +git add src/logstashui/LogstashUI/packaging/logstashui.default \ + src/logstashui/LogstashUI/cli.py \ + src/logstashui/LogstashUI/tests/test_cli.py \ + docs/docs/logstashui/configuration/environment.md \ + docs/docs/logstashui/general/deploy.md \ + CHANGELOG.md docker/Dockerfile .github/workflows/test-databases.yml +git commit -m "docs: multi-engine env, systemd, Docker extras, CI matrix" +``` + +--- + +### Task 9: Verification gate + +- [ ] **Step 1: Default inner loop (no Docker extras required)** + +```bash +uv run pytest src/logstashui --no-cov +``` + +Expected: no **new** failures vs sqlite baseline. Pre-existing three CRUD tests may still fail. + +- [ ] **Step 2: Full matrix** + +```bash +./bin/test_databases.sh +``` + +Expected: sqlite + postgresql + mariadb + mysql pytest runs; live migrator PASS. + +- [ ] **Step 3: Image install extras** + +```bash +grep -n '\[databases\]' docker/Dockerfile +``` + +Expected: `uv pip install --system --no-cache "/app[databases]"`. + +- [ ] **Step 4: Do not start smoke compose unless the operator asks.** Product CA must remain untouched. + +--- + +## Spec coverage (self-review) + +| Spec item | Task | +|---|---| +| ORM only / `build_databases` | 2–3 | +| Discrete env, aliases, SSL, CONN_* | 2–3, 8 | +| Extras + Docker `[databases]` | 1, 8 | +| psycopg gevent (version pin, no psycogreen) | 1, 4 (wsgi comment) | +| PyMySQL `install_as_MySQLdb` | 3 | +| Fail-fast unknown/missing driver/host | 2–3 | +| Server version floors | 2–3, 4 (serve after migrate) | +| SQLite default + scale warning | 3–4 | +| gunicorn pidfile + SIGTERM | 4, 6 | +| Offline dump/load docs | 8 | +| BETA CLI `--i-have-a-backup` | 6–7 | +| `--write-env`, no auto-restart | 6, 8 | +| Local Docker three engines + migrator | 5, 7, 9 | +| CI uses the same script | 8 | +| systemd prompts / sample env | 8 | +| MySQL `utf8mb4_bin` | 3, 5 (compose command + CREATE DATABASE) | +| DATA_DIR still required | 8 docs | +| No YAML / DATABASE_URL / DAO / smoke-stack DB | throughout | +| Pre-existing pytest failures left alone | 5, 9 | + +**Placeholders:** none remaining. Live tests use subprocesses so Django settings never switch in-process (matches architecture). + +**Type names:** `canonical_engine`, `_import_or_raise`, `check_server_version`, `cmd_migrate_engine`, `stop_gunicorn`, `write_env_file`, `run_manage` — used consistently across tasks. From f656719bd9a12c6454a03a47b1ac8465510022a4 Mon Sep 17 00:00:00 2001 From: Aaron Mildenstein Date: Tue, 1 Sep 2026 17:20:23 -0600 Subject: [PATCH 04/62] build: add postgres/mysql/databases install extras --- pyproject.toml | 12 ++++++ scripts/generate_notice.py | 7 +++- uv.lock | 84 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 102 insertions(+), 1 deletion(-) diff --git a/pyproject.toml b/pyproject.toml index 10d90da..683bb2a 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -35,6 +35,18 @@ logstashui = "LogstashUI.cli:main" [project.urls] Homepage = "https://github.com/elastic/LogstashUI" +[project.optional-dependencies] +postgres = [ + "psycopg[binary]>=3.2.0", +] +mysql = [ + "PyMySQL>=1.1.1", +] +databases = [ + "psycopg[binary]>=3.2.0", + "PyMySQL>=1.1.1", +] + [tool.uv] package = true diff --git a/scripts/generate_notice.py b/scripts/generate_notice.py index fae77d9..f7fe611 100644 --- a/scripts/generate_notice.py +++ b/scripts/generate_notice.py @@ -34,7 +34,9 @@ "d3": "https://github.com/d3/d3/blob/main/LICENSE", "codemirror": "https://github.com/codemirror/dev/blob/main/LICENSE", "js-yaml": "https://github.com/nodeca/js-yaml/blob/master/LICENSE", - "markedjs": "https://github.com/markedjs/marked/blob/master/LICENSE" + "markedjs": "https://github.com/markedjs/marked/blob/master/LICENSE", + "psycopg": "https://github.com/psycopg/psycopg/blob/master/LICENSE.txt", + "PyMySQL": "https://github.com/PyMySQL/PyMySQL/blob/main/LICENSE", } # Repository mappings for dependencies (fallback when automatic lookup fails) @@ -81,6 +83,9 @@ WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.""", + "psycopg": "https://github.com/psycopg/psycopg/blob/master/LICENSE.txt", + "psycopg-binary": "https://github.com/psycopg/psycopg/blob/master/LICENSE.txt", + "PyMySQL": "https://github.com/PyMySQL/PyMySQL/blob/main/LICENSE", "greenlet": "https://github.com/python-greenlet/greenlet/blob/master/LICENSE", "portalocker": "https://github.com/wolph/portalocker/blob/develop/LICENSE", "zope-event": "https://github.com/zopefoundation/zope.event/blob/master/LICENSE.txt", diff --git a/uv.lock b/uv.lock index 2636645..a57289b 100644 --- a/uv.lock +++ b/uv.lock @@ -582,6 +582,18 @@ dependencies = [ { name = "whitenoise" }, ] +[package.optional-dependencies] +databases = [ + { name = "psycopg", extra = ["binary"] }, + { name = "pymysql" }, +] +mysql = [ + { name = "pymysql" }, +] +postgres = [ + { name = "psycopg", extra = ["binary"] }, +] + [package.dev-dependencies] dev = [ { name = "pre-commit" }, @@ -604,12 +616,17 @@ requires-dist = [ { name = "lark", specifier = ">=1.3.1" }, { name = "markdown", specifier = ">=3.10.2" }, { name = "packaging", specifier = ">=26.0" }, + { name = "psycopg", extras = ["binary"], marker = "extra == 'databases'", specifier = ">=3.2.0" }, + { name = "psycopg", extras = ["binary"], marker = "extra == 'postgres'", specifier = ">=3.2.0" }, { name = "pygrok", specifier = ">=1.0.0" }, + { name = "pymysql", marker = "extra == 'databases'", specifier = ">=1.1.1" }, + { name = "pymysql", marker = "extra == 'mysql'", specifier = ">=1.1.1" }, { name = "pysnmp", specifier = ">=7.1.26" }, { name = "pyyaml", specifier = ">=6.0.3" }, { name = "requests", specifier = ">=2.33.0" }, { name = "whitenoise", specifier = ">=6.12.0" }, ] +provides-extras = ["postgres", "mysql", "databases"] [package.metadata.requires-dev] dev = [ @@ -680,6 +697,64 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/5d/19/fd3ef348460c80af7bb4669ea7926651d1f95c23ff2df18b9d24bab4f3fa/pre_commit-4.5.1-py2.py3-none-any.whl", hash = "sha256:3b3afd891e97337708c1674210f8eba659b52a38ea5f822ff142d10786221f77", size = 226437, upload-time = "2025-12-16T21:14:32.409Z" }, ] +[[package]] +name = "psycopg" +version = "3.3.5" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "typing-extensions", marker = "python_full_version < '3.13'" }, + { name = "tzdata", marker = "sys_platform == 'win32'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/72/73/8fb739d0f6bba247b9b93c9840c402a4f88545be5f1d4b02b23366371c00/psycopg-3.3.5.tar.gz", hash = "sha256:d0a3d9ccf5788af054cbd745278cb02401b5c312aeaafbf2c6144460aec47da4", size = 166508, upload-time = "2026-08-31T22:45:43.151Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/3d/2e/d0a645bcaadde68bd6d93c43f02f14b0191bdda367ce3f7722abe3da744a/psycopg-3.3.5-py3-none-any.whl", hash = "sha256:ce5aa5cdb4f9379f00f487590e5890bfa7df9a164648c969ffa628505e21af4e", size = 213598, upload-time = "2026-08-31T22:39:02.184Z" }, +] + +[package.optional-dependencies] +binary = [ + { name = "psycopg-binary", marker = "implementation_name != 'pypy'" }, +] + +[[package]] +name = "psycopg-binary" +version = "3.3.5" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/5e/83/ba396428a4fb6b70f0dd41315ad86a2c14441b7214afd47b2d49cb450b78/psycopg_binary-3.3.5-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:25105f9b46bdf2a30fcb67f56976ed66f6855941ae16bc024192609b917d493c", size = 4700169, upload-time = "2026-08-31T22:41:39.712Z" }, + { url = "https://files.pythonhosted.org/packages/4f/56/b23c5978e55cf4effdc5a3e13a17d69a580a487993e01b7c3768dd24281c/psycopg_binary-3.3.5-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:0249c3e960cdee686000eb77169fb6590105c05bacc37e057ccdffdcd8e6ebde", size = 4763037, upload-time = "2026-08-31T22:41:47.571Z" }, + { url = "https://files.pythonhosted.org/packages/fd/d8/b41108bfe194b4098076c8b873b05ec2ca9582445eccfd9075970d7b948e/psycopg_binary-3.3.5-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:5698ab5941a4d138c30fef858588e651fe7d583280cd6e41832825ad9e747750", size = 5546232, upload-time = "2026-08-31T22:41:55.817Z" }, + { url = "https://files.pythonhosted.org/packages/21/d1/0f244dfef389e52e9dc3056f2a9033d1f6901e97d24d9a9c8b836e32ab6b/psycopg_binary-3.3.5-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:682a17a57415c3ca1731eec018ed031f012ffcb81ba74806eb219cb396065672", size = 5227752, upload-time = "2026-08-31T22:42:03.776Z" }, + { url = "https://files.pythonhosted.org/packages/31/88/a4781365f09807fb91435e2d00096f3f6ae5d06bce15ef3c3c385dc22772/psycopg_binary-3.3.5-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d2a61e8147902771df7efe14062a3c8736347850d0d8befcf048235752504f2e", size = 6824658, upload-time = "2026-08-31T22:42:13.263Z" }, + { url = "https://files.pythonhosted.org/packages/65/f1/072c4a46287644694731b3e40fab120ebacf6d153cce7ebf7a5b208f5561/psycopg_binary-3.3.5-cp312-cp312-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:f7e1e45aad410e20de45df2b159df68ff6c8dbf47a3501f806c4489b27f4ad2b", size = 5061439, upload-time = "2026-08-31T22:42:18.591Z" }, + { url = "https://files.pythonhosted.org/packages/e2/d6/1776a95c16941b8bbce89407cc7cf9ea3fd557efb503a40873c9e2b6394f/psycopg_binary-3.3.5-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:c09775c549b40b274206e1b043c5e5b5af39666e85c98382a30bd05d23ab677b", size = 4588586, upload-time = "2026-08-31T22:42:25.23Z" }, + { url = "https://files.pythonhosted.org/packages/82/64/44ec87b9a74faebe966856307b65c0d35ee6321ce509cdd0e8d6a3f5338b/psycopg_binary-3.3.5-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:cf0e5e63ee86098299c673992053d556c489ba9ae6aca6cb6e24d16a8e0b09e6", size = 4265161, upload-time = "2026-08-31T22:42:31.864Z" }, + { url = "https://files.pythonhosted.org/packages/24/88/cf181df5651395a80afc520f5fefac72beb035c0c9d58ab7fcc880c9b221/psycopg_binary-3.3.5-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:c065531e8c1815276f50dbfa283e3a7f022671414cdda6fa9a16794dd53b28f9", size = 3998727, upload-time = "2026-08-31T22:42:38.436Z" }, + { url = "https://files.pythonhosted.org/packages/43/1e/c72a1107647db4bb3f534c7fe1b57b1957d9c099e795f5aa81f4a2e0c312/psycopg_binary-3.3.5-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:df9853b832b7b916e02ef68e0d5403a7dab2d5c1ddfe94f22b1b155eb862622f", size = 4310119, upload-time = "2026-08-31T22:42:46.403Z" }, + { url = "https://files.pythonhosted.org/packages/03/8d/452620608cafff164737e20b42ebffee8151b865ee171ba0d6692a560a44/psycopg_binary-3.3.5-cp312-cp312-win_amd64.whl", hash = "sha256:35885e333020fc152d27bea1a494bef13b2e68f6fd92b6229015e93539152008", size = 3648197, upload-time = "2026-08-31T22:42:51.575Z" }, + { url = "https://files.pythonhosted.org/packages/e0/1c/e718752cc63cf4e99e4a10fd36e3a3364dabdd0819484a24c0d79fbb9685/psycopg_binary-3.3.5-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:6e85d50b87257fb117675a19ee59daa7bf9a57f6431500adf7059df799232ef4", size = 4704421, upload-time = "2026-08-31T22:42:59.564Z" }, + { url = "https://files.pythonhosted.org/packages/af/cf/a0e748e27c09b92738e4460582d121ba1908be3e36791e150f435e54b332/psycopg_binary-3.3.5-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:e5becd311f9af8d180bad372f51fb2252fd02cb2073056e2b170c9274f95fe7f", size = 4765054, upload-time = "2026-08-31T22:43:05.421Z" }, + { url = "https://files.pythonhosted.org/packages/39/62/0cbac0266d56c94dd1f702d9af2b5d54bf80b6e658048f4f2c5bd63dd7e3/psycopg_binary-3.3.5-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:19e5bf9872dbd164c220567fd385ba2309c7d9df1541f78343510c6b0f36a1b7", size = 5547137, upload-time = "2026-08-31T22:43:11.768Z" }, + { url = "https://files.pythonhosted.org/packages/fb/3a/73c6f8871f38fc07a9c0b4cbc9467beb116a2783cecf87dfa53900a396dc/psycopg_binary-3.3.5-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:cb3b3bffebfe07110730626e76238161124f35ac87b748d663316a28d22f58b0", size = 5227577, upload-time = "2026-08-31T22:43:20.767Z" }, + { url = "https://files.pythonhosted.org/packages/59/7b/9f17b9f4d297b774dc574199c4dcd02dadc32a00c4056265918de5c70635/psycopg_binary-3.3.5-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:2111f880add40fb03c60556069ad68e884a0908a74d2debafc603caf93b73552", size = 6824606, upload-time = "2026-08-31T22:43:33.698Z" }, + { url = "https://files.pythonhosted.org/packages/94/86/d84dadd94a004dbbb43ce0579f1f766fdc6b8cbef745090e24bea77b5283/psycopg_binary-3.3.5-cp313-cp313-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:40505676b1526b9ea387dace034040a8c8b0bcf984cd6bd4720a2ab15e813586", size = 5060854, upload-time = "2026-08-31T22:43:42.258Z" }, + { url = "https://files.pythonhosted.org/packages/30/d0/e5078be2c7d2490c0d6cd4cb7b3601aec44be2fa8b3fe927e9419b06004f/psycopg_binary-3.3.5-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:5816472e3bb05615f33a741e0835043d1f4bf9709ff30d2f4aed71815cfc6b5e", size = 4589511, upload-time = "2026-08-31T22:43:50.012Z" }, + { url = "https://files.pythonhosted.org/packages/f1/01/08dfb5b18fa482e025864fd91a022310d0782c42e4cf5dda5d0e010b6790/psycopg_binary-3.3.5-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:358748fc4c8ccdc0e2bdf55420494930e19c3ade586ea9c3a6de3dad1f897311", size = 4268144, upload-time = "2026-08-31T22:43:56.993Z" }, + { url = "https://files.pythonhosted.org/packages/88/b9/cb01dc1d63f3241b49b2ca7fb9f98d0f5c76127f0dbb9440635a6ad0233e/psycopg_binary-3.3.5-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:1ef2e498be47800f6202b9a2304c22646325ca6d54001b7c785bcfdb24a1e8ab", size = 4001036, upload-time = "2026-08-31T22:44:04.429Z" }, + { url = "https://files.pythonhosted.org/packages/95/49/7c17dd832c05b380562ff2ff5f6ab2bcaeca8b7fff2c2b355854368a5bdb/psycopg_binary-3.3.5-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:88e01aa2e938a45655a8a5213fc3a44ba78cb4cab8a569b3e0bcb3d1d0eaba16", size = 4313112, upload-time = "2026-08-31T22:44:12.446Z" }, + { url = "https://files.pythonhosted.org/packages/7c/2c/b0b2f887185d6a2ec0b3bef948cc07656d2d1a5d96fa7f2bb03f6ef06ca4/psycopg_binary-3.3.5-cp313-cp313-win_amd64.whl", hash = "sha256:ba466011569297114449df9d523438e1adeedf3e4f31ffb78e897ec3fef3076b", size = 3647313, upload-time = "2026-08-31T22:44:19.139Z" }, + { url = "https://files.pythonhosted.org/packages/46/7f/4e2395da194558533bd9c31f35e4dc58ecbbae6a7176b0d2f72d629e8a51/psycopg_binary-3.3.5-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:40f8b132c7243ef5f503f0b6f986bf16d38a51b0df1c6ba2577743f128be03e3", size = 4712612, upload-time = "2026-08-31T22:44:25.723Z" }, + { url = "https://files.pythonhosted.org/packages/ae/94/fdb2093c8ccd7048449156db526ad746740cf34fe9c01e5dc1b7a7a8b257/psycopg_binary-3.3.5-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:c0cac998b9b1e82dec853d2e53b3d34d56a525cf231f9441a636cfd5992929a9", size = 4775139, upload-time = "2026-08-31T22:44:36.719Z" }, + { url = "https://files.pythonhosted.org/packages/31/52/5195e87960715f7be2005761b72d56fce4e7757d5333fd40384f071c2de1/psycopg_binary-3.3.5-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:479b96fd78149cfa10369dc53fbfb89ee729be13146b584a23dbc7e164c0cf1e", size = 5556807, upload-time = "2026-08-31T22:44:42.668Z" }, + { url = "https://files.pythonhosted.org/packages/f7/42/2d616210a91e1327516ed5ae71961aaa31bb740e4a61d7190f2221685a40/psycopg_binary-3.3.5-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:f45d77e398542ce0937d9fa3cd9d84e9c5fc6b34c50a66404ae840bada312750", size = 5236206, upload-time = "2026-08-31T22:44:47.943Z" }, + { url = "https://files.pythonhosted.org/packages/08/2e/e54b0d4cc263b3526e3728bb50a69660d5e79e52255ccd2a1a71e40e6f9a/psycopg_binary-3.3.5-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:98a388509306e5e08a4203253ac52846bc1b034e5cbd0ae6da1211593cc28594", size = 6838066, upload-time = "2026-08-31T22:44:55.701Z" }, + { url = "https://files.pythonhosted.org/packages/77/80/ec22a110f81a44c411965982097efeee86006bdd5a1f51f628318106c84c/psycopg_binary-3.3.5-cp314-cp314-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:ab39e2794b95af61a2ff69e33e5ab6ac5df36e9ffea9a3b18e38b2aaca8c5ad5", size = 5072036, upload-time = "2026-08-31T22:45:02.838Z" }, + { url = "https://files.pythonhosted.org/packages/93/55/7bc3c3ac769ab4fe0c619f2179b4aab3ae043f4d478283dd8279d24c0be4/psycopg_binary-3.3.5-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:9c071bf78e5c2e6efa40bc9089a954d7b41221347a72f35c6bf2d8c96e632f75", size = 4604058, upload-time = "2026-08-31T22:45:10.444Z" }, + { url = "https://files.pythonhosted.org/packages/a3/69/8e7414f7dc10b2959e664330cdcf393e412f67355975dafa876cef265264/psycopg_binary-3.3.5-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:14fdfd65a96ecbd8b586d14546105641f4a6ac7cbe335c786830ea4de94bbe60", size = 4284766, upload-time = "2026-08-31T22:45:17.881Z" }, + { url = "https://files.pythonhosted.org/packages/c1/72/33f293c1d3ee9114f47e9ef4880f31c9de864e84a9b09454c26e106c25ed/psycopg_binary-3.3.5-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:8dbd694f3741dd4ac5bc60b70e17f7841aefb3f0f38cef4d2756de270e03af43", size = 4011958, upload-time = "2026-08-31T22:45:24.792Z" }, + { url = "https://files.pythonhosted.org/packages/b7/c9/8e38840e5a7d006987bbc9acb29951912253fa50549a4db92b3aa535f089/psycopg_binary-3.3.5-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:14f432430fd9e1a9e7d9ab2fe14956c77f5d074ebdc556a1ad04e9a1bd3fca04", size = 4323273, upload-time = "2026-08-31T22:45:32.973Z" }, + { url = "https://files.pythonhosted.org/packages/fc/c7/b7ebf601c307f93e7c4c4ebac0edc9db3b2729ca038efe700a18f86b5517/psycopg_binary-3.3.5-cp314-cp314-win_amd64.whl", hash = "sha256:df209e64674a34b41662c67fdc8b4e0ffd77d2136393790691d086a09f9a6cab", size = 3745885, upload-time = "2026-08-31T22:45:40.537Z" }, +] + [[package]] name = "pyasn1" version = "0.6.3" @@ -716,6 +791,15 @@ dependencies = [ ] sdist = { url = "https://files.pythonhosted.org/packages/ce/a5/963d78c4eda7edb0ea827679dbcf5f77e4d767562b59681bd23ea5913af6/pygrok-1.0.0.tar.gz", hash = "sha256:ae635e3c0ba0eab76aec9d86ae1bab70883e8e71505ec2d6cb8989e66f5810af", size = 18997, upload-time = "2016-09-24T08:32:38.164Z" } +[[package]] +name = "pymysql" +version = "1.2.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/c9/bc/1c6a92f385940f727daeecf3bacaf186e03875dff57197801046c583bcf0/pymysql-1.2.0.tar.gz", hash = "sha256:6c7b17ca686988104d7426c27895b455cdeea3e9d3ceb1270f0c3704fead8c33", size = 49021, upload-time = "2026-05-19T08:26:22.302Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/c4/bd/2534e130295c8cfd4f0a2e31623baab7502278f1e97bcfe61db75656a77f/pymysql-1.2.0-py3-none-any.whl", hash = "sha256:62169ce6d5510f08e140c5e7990ee884a9764024e4a9a27b2cc11f1099322ae0", size = 45716, upload-time = "2026-05-19T08:26:20.974Z" }, +] + [[package]] name = "pysnmp" version = "7.1.26" From eb9dcbaa7536a4f0e3292c9ad9e5c1f1ef78ce3e Mon Sep 17 00:00:00 2001 From: Aaron Mildenstein Date: Tue, 1 Sep 2026 17:25:41 -0600 Subject: [PATCH 05/62] test: specify multi-engine build_databases behavior --- .../LogstashUI/tests/test_database.py | 170 +++++++++++++++++- 1 file changed, 163 insertions(+), 7 deletions(-) diff --git a/src/logstashui/LogstashUI/tests/test_database.py b/src/logstashui/LogstashUI/tests/test_database.py index 867f1dd..5990123 100644 --- a/src/logstashui/LogstashUI/tests/test_database.py +++ b/src/logstashui/LogstashUI/tests/test_database.py @@ -6,21 +6,177 @@ import pytest -from LogstashUI.database import build_databases +from LogstashUI.database import ( + build_databases, + canonical_engine, + check_server_version, +) + + +def _clear_db_env(monkeypatch): + for name in ( + "LOGSTASHUI_DB_ENGINE", + "LOGSTASHUI_DB_NAME", + "LOGSTASHUI_DB_HOST", + "LOGSTASHUI_DB_PORT", + "LOGSTASHUI_DB_USER", + "LOGSTASHUI_DB_PASSWORD", + "LOGSTASHUI_DB_SSLMODE", + "LOGSTASHUI_DB_SSL_CA", + "LOGSTASHUI_DB_CONN_MAX_AGE", + "LOGSTASHUI_DB_CONN_HEALTH_CHECKS", + ): + monkeypatch.delenv(name, raising=False) def test_build_databases_sqlite_default(tmp_path, monkeypatch): - monkeypatch.delenv("LOGSTASHUI_DB_ENGINE", raising=False) + _clear_db_env(monkeypatch) db = build_databases(tmp_path) assert db["default"]["ENGINE"] == "django.db.backends.sqlite3" assert db["default"]["NAME"] == tmp_path / "db.sqlite3" assert db["default"]["OPTIONS"]["timeout"] == 20 + assert "PRAGMA journal_mode=WAL" in db["default"]["OPTIONS"]["init_command"] + +@pytest.mark.parametrize( + "raw,expected", + [ + ("", "sqlite"), + ("sqlite", "sqlite"), + ("sqlite3", "sqlite"), + ("postgres", "postgresql"), + ("postgresql", "postgresql"), + ("mysql", "mysql"), + ("mariadb", "mysql"), + ("my", "mysql"), + ("POSTGRESQL", "postgresql"), + ], +) +def test_canonical_engine_aliases(raw, expected): + assert canonical_engine(raw) == expected -def test_build_databases_rejects_unimplemented_engine(tmp_path, monkeypatch): + +def test_unknown_engine_fails(tmp_path, monkeypatch): + _clear_db_env(monkeypatch) + monkeypatch.setenv("LOGSTASHUI_DB_ENGINE", "oracle") + with pytest.raises(RuntimeError, match="Unknown LOGSTASHUI_DB_ENGINE"): + build_databases(tmp_path) + + +def test_postgresql_requires_host_user(tmp_path, monkeypatch): + _clear_db_env(monkeypatch) monkeypatch.setenv("LOGSTASHUI_DB_ENGINE", "postgresql") - with pytest.raises(RuntimeError, match="not implemented"): + monkeypatch.setattr("LogstashUI.database._import_or_raise", lambda *a, **k: None) + with pytest.raises(RuntimeError, match="LOGSTASHUI_DB_HOST"): + build_databases(tmp_path) + monkeypatch.setenv("LOGSTASHUI_DB_HOST", "db.example") + with pytest.raises(RuntimeError, match="LOGSTASHUI_DB_USER"): + build_databases(tmp_path) + + +def test_build_databases_postgresql(tmp_path, monkeypatch): + _clear_db_env(monkeypatch) + monkeypatch.setenv("LOGSTASHUI_DB_ENGINE", "postgres") + monkeypatch.setenv("LOGSTASHUI_DB_HOST", "db.example") + monkeypatch.setenv("LOGSTASHUI_DB_USER", "lsui") + monkeypatch.setenv("LOGSTASHUI_DB_PASSWORD", "s3cret") + monkeypatch.setenv("LOGSTASHUI_DB_SSLMODE", "require") + monkeypatch.setenv("LOGSTASHUI_DB_SSL_CA", "/etc/ssl/db-ca.pem") + monkeypatch.setattr("LogstashUI.database._import_or_raise", lambda *a, **k: None) + db = build_databases(tmp_path)["default"] + assert db["ENGINE"] == "django.db.backends.postgresql" + assert db["NAME"] == "logstashui" + assert db["HOST"] == "db.example" + assert db["PORT"] == "5432" + assert db["USER"] == "lsui" + assert db["PASSWORD"] == "s3cret" + assert db["CONN_MAX_AGE"] == 60 + assert db["CONN_HEALTH_CHECKS"] is True + assert db["OPTIONS"]["sslmode"] == "require" + assert db["OPTIONS"]["sslrootcert"] == "/etc/ssl/db-ca.pem" + + +def test_build_databases_mysql_mariadb_alias(tmp_path, monkeypatch): + _clear_db_env(monkeypatch) + monkeypatch.setenv("LOGSTASHUI_DB_ENGINE", "mariadb") + monkeypatch.setenv("LOGSTASHUI_DB_HOST", "127.0.0.1") + monkeypatch.setenv("LOGSTASHUI_DB_USER", "lsui") + monkeypatch.setenv("LOGSTASHUI_DB_PORT", "3307") + monkeypatch.setenv("LOGSTASHUI_DB_CONN_MAX_AGE", "0") + monkeypatch.setenv("LOGSTASHUI_DB_CONN_HEALTH_CHECKS", "false") + monkeypatch.setattr("LogstashUI.database._import_or_raise", lambda *a, **k: None) + db = build_databases(tmp_path)["default"] + assert db["ENGINE"] == "django.db.backends.mysql" + assert db["PORT"] == "3307" + assert db["CONN_MAX_AGE"] == 0 + assert db["CONN_HEALTH_CHECKS"] is False + assert db["OPTIONS"]["charset"] == "utf8mb4" + assert "utf8mb4_bin" in db["OPTIONS"]["init_command"] + assert db["TEST"]["CHARSET"] == "utf8mb4" + assert db["TEST"]["COLLATION"] == "utf8mb4_bin" + + +def test_postgresql_missing_driver(tmp_path, monkeypatch): + _clear_db_env(monkeypatch) + monkeypatch.setenv("LOGSTASHUI_DB_ENGINE", "postgresql") + monkeypatch.setenv("LOGSTASHUI_DB_HOST", "localhost") + monkeypatch.setenv("LOGSTASHUI_DB_USER", "lsui") + + def boom(module, extra): + raise RuntimeError( + f"{module} is not installed. Install with: uv pip install 'LogstashUI[{extra}]'" + ) + + monkeypatch.setattr("LogstashUI.database._import_or_raise", boom) + with pytest.raises(RuntimeError, match=r"LogstashUI\[postgres\]"): build_databases(tmp_path) - monkeypatch.setenv("LOGSTASHUI_DB_ENGINE", "mysql") - with pytest.raises(RuntimeError, match="not implemented"): - build_databases(Path(tmp_path)) + + +def test_check_server_version_sqlite_noop(): + class Conn: + vendor = "sqlite" + + check_server_version(Conn()) + + +def test_check_server_version_postgres_too_old(): + class Conn: + vendor = "postgresql" + pg_version = 130000 + + with pytest.raises(RuntimeError, match="PostgreSQL 14"): + check_server_version(Conn()) + + +def test_check_server_version_mysql_and_mariadb(): + class Mysql: + vendor = "mysql" + mysql_is_mariadb = False + mysql_server_info = "8.0.36" + + def get_database_version(self): + return (8, 0, 36) + + check_server_version(Mysql()) + + class OldMysql: + vendor = "mysql" + mysql_is_mariadb = False + mysql_server_info = "5.7.44" + + def get_database_version(self): + return (5, 7, 44) + + with pytest.raises(RuntimeError, match="MySQL 8.0"): + check_server_version(OldMysql()) + + class Maria: + vendor = "mysql" + mysql_is_mariadb = True + mysql_server_info = "10.5.22-MariaDB" + + def get_database_version(self): + return (10, 5, 22) + + with pytest.raises(RuntimeError, match="MariaDB 10.6"): + check_server_version(Maria()) From 072722443e1f539783fa80831da5cd568bd3746e Mon Sep 17 00:00:00 2001 From: Aaron Mildenstein Date: Tue, 1 Sep 2026 17:32:14 -0600 Subject: [PATCH 06/62] feat: wire PostgreSQL and MySQL Django backends from env --- src/logstashui/LogstashUI/database.py | 168 +++++++++++++++++++++++--- 1 file changed, 153 insertions(+), 15 deletions(-) diff --git a/src/logstashui/LogstashUI/database.py b/src/logstashui/LogstashUI/database.py index 43cfeee..fb713ac 100644 --- a/src/logstashui/LogstashUI/database.py +++ b/src/logstashui/LogstashUI/database.py @@ -2,28 +2,86 @@ #or more contributor license agreements. Licensed under the Elastic License; #you may not use this file except in compliance with the Elastic License. -"""Database settings helper. - -SQLite is the only implemented engine. PostgreSQL and MySQL are anticipated -via ``LOGSTASHUI_DB_ENGINE`` but not wired in this release. -""" +"""Build Django DATABASES from discrete LOGSTASHUI_DB_* environment variables.""" from __future__ import annotations import os from pathlib import Path +from .config import env_bool + +_ENGINE_ALIASES = { + "": "sqlite", + "sqlite": "sqlite", + "sqlite3": "sqlite", + "postgres": "postgresql", + "postgresql": "postgresql", + "mysql": "mysql", + "mariadb": "mysql", + "my": "mysql", +} + +_CANONICAL = ("sqlite", "postgresql", "mysql") + + +def canonical_engine(raw: str | None) -> str: + key = (raw or "").strip().lower() + if key not in _ENGINE_ALIASES: + supported = "sqlite, postgresql, mysql (aliases: sqlite3, postgres, mariadb, my)" + raise RuntimeError( + f"Unknown LOGSTASHUI_DB_ENGINE={raw!r}. Supported: {supported}." + ) + return _ENGINE_ALIASES[key] + + +def _import_or_raise(module: str, extra: str): + try: + return __import__(module) + except ImportError as exc: + raise RuntimeError( + f"{module} is not installed. Install with: uv pip install 'LogstashUI[{extra}]' " + f"(Docker/K8s image already includes LogstashUI[databases])." + ) from exc + + +def _env(name: str, default: str = "") -> str: + raw = os.environ.get(name) + if raw is None: + return default + return raw.strip() + + +def _env_int(name: str, default: int) -> int: + raw = _env(name, "") + if raw == "": + return default + return int(raw) + + +def _require(names: list[str]) -> None: + missing = [n for n in names if not _env(n)] + if missing: + raise RuntimeError( + "Missing required database settings: " + ", ".join(missing) + ) + def build_databases(data_dir: Path) -> dict: - engine = (os.environ.get("LOGSTASHUI_DB_ENGINE") or "sqlite").strip().lower() - if engine in ("", "sqlite", "sqlite3"): + engine = canonical_engine(os.environ.get("LOGSTASHUI_DB_ENGINE")) + conn_max_age = _env_int("LOGSTASHUI_DB_CONN_MAX_AGE", 60) + health = env_bool("LOGSTASHUI_DB_CONN_HEALTH_CHECKS", True) + + if engine == "sqlite": + name = _env("LOGSTASHUI_DB_NAME") + db_name = Path(name) if name else Path(data_dir) / "db.sqlite3" return { "default": { "ENGINE": "django.db.backends.sqlite3", - "NAME": Path(data_dir) / "db.sqlite3", + "NAME": db_name, + "CONN_MAX_AGE": conn_max_age, + "CONN_HEALTH_CHECKS": health, "OPTIONS": { - # busy_timeout before journal_mode so the busy handler is - # armed before WAL mode is asserted on the connection. "init_command": ( "PRAGMA busy_timeout=20000;" "PRAGMA journal_mode=WAL;" @@ -32,8 +90,88 @@ def build_databases(data_dir: Path) -> dict: }, } } - raise RuntimeError( - f"LOGSTASHUI_DB_ENGINE={engine!r} is not implemented. " - "Supported: sqlite. PostgreSQL and MySQL are planned; keep a PVC on " - "LOGSTASHUI_DATA_DIR even after those engines land (TLS and secrets)." - ) + + if engine == "postgresql": + _import_or_raise("psycopg", "postgres") + _require(["LOGSTASHUI_DB_HOST", "LOGSTASHUI_DB_USER"]) + sslmode = _env("LOGSTASHUI_DB_SSLMODE", "prefer") or "prefer" + options: dict = {"sslmode": sslmode} + ca = _env("LOGSTASHUI_DB_SSL_CA") + if ca: + options["sslrootcert"] = ca + return { + "default": { + "ENGINE": "django.db.backends.postgresql", + "NAME": _env("LOGSTASHUI_DB_NAME", "logstashui") or "logstashui", + "USER": _env("LOGSTASHUI_DB_USER"), + "PASSWORD": os.environ.get("LOGSTASHUI_DB_PASSWORD") or "", + "HOST": _env("LOGSTASHUI_DB_HOST"), + "PORT": _env("LOGSTASHUI_DB_PORT", "5432") or "5432", + "CONN_MAX_AGE": conn_max_age, + "CONN_HEALTH_CHECKS": health, + "OPTIONS": options, + } + } + + pymysql = _import_or_raise("pymysql", "mysql") + if pymysql is not None: + pymysql.install_as_MySQLdb() + _require(["LOGSTASHUI_DB_HOST", "LOGSTASHUI_DB_USER"]) + options = { + "charset": "utf8mb4", + "init_command": ( + "SET sql_mode='STRICT_TRANS_TABLES', " + "NAMES utf8mb4 COLLATE utf8mb4_bin" + ), + } + ca = _env("LOGSTASHUI_DB_SSL_CA") + if ca: + options["ssl"] = {"ca": ca} + return { + "default": { + "ENGINE": "django.db.backends.mysql", + "NAME": _env("LOGSTASHUI_DB_NAME", "logstashui") or "logstashui", + "USER": _env("LOGSTASHUI_DB_USER"), + "PASSWORD": os.environ.get("LOGSTASHUI_DB_PASSWORD") or "", + "HOST": _env("LOGSTASHUI_DB_HOST"), + "PORT": _env("LOGSTASHUI_DB_PORT", "3306") or "3306", + "CONN_MAX_AGE": conn_max_age, + "CONN_HEALTH_CHECKS": health, + "OPTIONS": options, + "TEST": { + "CHARSET": "utf8mb4", + "COLLATION": "utf8mb4_bin", + }, + } + } + + +def check_server_version(connection) -> None: + """Fail-fast if the server is below documented floors. No-op for SQLite.""" + vendor = getattr(connection, "vendor", "") + if vendor == "postgresql": + pg_version = int(getattr(connection, "pg_version", 0) or 0) + if pg_version and pg_version < 140000: + raise RuntimeError( + f"PostgreSQL 14+ is required (server_version_num={pg_version})." + ) + return + if vendor != "mysql": + return + is_mariadb = bool(getattr(connection, "mysql_is_mariadb", False)) + info = (getattr(connection, "mysql_server_info", "") or "").lower() + if "mariadb" in info: + is_mariadb = True + if hasattr(connection, "get_database_version"): + tup = connection.get_database_version() + else: + tup = (0, 0, 0) + major_minor = (int(tup[0]), int(tup[1])) + if is_mariadb and major_minor < (10, 6): + raise RuntimeError( + f"MariaDB 10.6+ is required (server={getattr(connection, 'mysql_server_info', tup)})." + ) + if not is_mariadb and major_minor < (8, 0): + raise RuntimeError( + f"MySQL 8.0+ is required (server={getattr(connection, 'mysql_server_info', tup)})." + ) From 662af0b2f7d04d1b9370ef09e381478711754a37 Mon Sep 17 00:00:00 2001 From: Aaron Mildenstein Date: Tue, 1 Sep 2026 17:41:40 -0600 Subject: [PATCH 07/62] fix: spoof PyMySQL version_info for Django 6 mysql backend --- src/logstashui/LogstashUI/database.py | 3 +++ .../LogstashUI/tests/test_database.py | 17 +++++++++++++++++ 2 files changed, 20 insertions(+) diff --git a/src/logstashui/LogstashUI/database.py b/src/logstashui/LogstashUI/database.py index fb713ac..f5b3d4f 100644 --- a/src/logstashui/LogstashUI/database.py +++ b/src/logstashui/LogstashUI/database.py @@ -115,6 +115,9 @@ def build_databases(data_dir: Path) -> dict: pymysql = _import_or_raise("pymysql", "mysql") if pymysql is not None: + # Django 6 MySQL backend requires Database.version_info >= (2, 2, 1) + # (mysqlclient). PyMySQL reports ~1.1.1, so spoof before the shim. + pymysql.version_info = (2, 2, 1, "final", 0) pymysql.install_as_MySQLdb() _require(["LOGSTASHUI_DB_HOST", "LOGSTASHUI_DB_USER"]) options = { diff --git a/src/logstashui/LogstashUI/tests/test_database.py b/src/logstashui/LogstashUI/tests/test_database.py index 5990123..9a9cc89 100644 --- a/src/logstashui/LogstashUI/tests/test_database.py +++ b/src/logstashui/LogstashUI/tests/test_database.py @@ -3,6 +3,7 @@ #you may not use this file except in compliance with the Elastic License. from pathlib import Path +from types import SimpleNamespace import pytest @@ -116,6 +117,22 @@ def test_build_databases_mysql_mariadb_alias(tmp_path, monkeypatch): assert db["TEST"]["COLLATION"] == "utf8mb4_bin" +def test_mysql_spoofs_pymysql_version_info(tmp_path, monkeypatch): + _clear_db_env(monkeypatch) + monkeypatch.setenv("LOGSTASHUI_DB_ENGINE", "mysql") + monkeypatch.setenv("LOGSTASHUI_DB_HOST", "127.0.0.1") + monkeypatch.setenv("LOGSTASHUI_DB_USER", "lsui") + installed = [] + fake = SimpleNamespace( + version_info=(1, 1, 1, "final", 0), + install_as_MySQLdb=lambda: installed.append(True), + ) + monkeypatch.setattr("LogstashUI.database._import_or_raise", lambda *a, **k: fake) + build_databases(tmp_path) + assert fake.version_info == (2, 2, 1, "final", 0) + assert installed + + def test_postgresql_missing_driver(tmp_path, monkeypatch): _clear_db_env(monkeypatch) monkeypatch.setenv("LOGSTASHUI_DB_ENGINE", "postgresql") From 35c0c400a6dd14c9431ff54704fa051fcd1ec938 Mon Sep 17 00:00:00 2001 From: Aaron Mildenstein Date: Tue, 1 Sep 2026 17:45:11 -0600 Subject: [PATCH 08/62] feat: gunicorn pidfile and SQLite scale warning --- src/logstashui/LogstashUI/cli.py | 43 +++++++++++++++++++++ src/logstashui/LogstashUI/tests/test_cli.py | 37 ++++++++++++++++++ src/logstashui/LogstashUI/wsgi.py | 3 ++ 3 files changed, 83 insertions(+) diff --git a/src/logstashui/LogstashUI/cli.py b/src/logstashui/LogstashUI/cli.py index 53bcb12..fb256ce 100644 --- a/src/logstashui/LogstashUI/cli.py +++ b/src/logstashui/LogstashUI/cli.py @@ -7,12 +7,15 @@ from __future__ import annotations import argparse +import logging import os import shutil import sys from importlib.resources import files from pathlib import Path +logger = logging.getLogger(__name__) + def build_parser() -> argparse.ArgumentParser: parser = argparse.ArgumentParser( @@ -47,6 +50,25 @@ def build_parser() -> argparse.ArgumentParser: manage = sub.add_parser("manage", help="Django management command passthrough") manage.add_argument("manage_args", nargs=argparse.REMAINDER) + migrate = sub.add_parser( + "migrate-engine", + help="BETA: copy SQLite data to PostgreSQL or MySQL (stops gunicorn)", + ) + migrate.add_argument("--to", required=True, choices=("postgresql", "mysql")) + migrate.add_argument( + "--i-have-a-backup", + dest="i_have_a_backup", + action="store_true", + help="Required. Confirms db.sqlite3 was copied aside.", + ) + migrate.add_argument("--pid", type=Path, default=None, help="gunicorn pidfile to signal") + migrate.add_argument( + "--write-env", + type=Path, + default=None, + help="Append LOGSTASHUI_DB_* to this EnvironmentFile", + ) + systemd = sub.add_parser( "systemd", help="Generate /etc/default/logstashui and the systemd unit (manual)", @@ -300,15 +322,34 @@ def _best_effort_call(name: str, **kwargs) -> None: def cmd_serve(args: argparse.Namespace) -> int: + from LogstashUI.database import canonical_engine, check_server_version + from LogstashUI.paths import resolve_data_dir + os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") tls_env = os.environ.get("LOGSTASHUI_TLS", "true") tls_on = not args.no_tls and tls_env.strip().lower() not in ("0", "false", "no", "off") + engine = canonical_engine(os.environ.get("LOGSTASHUI_DB_ENGINE")) + if engine == "sqlite" and args.workers > 1: + msg = ( + "SQLite is the small-install default; use PostgreSQL or MySQL/MariaDB " + "for concurrent agents (LOGSTASHUI_WORKERS>1)." + ) + logger.warning(msg) + print(msg, file=sys.stderr) + if not args.skip_migrate: _manage(["migrate", "--noinput"]) + _django_setup() + from django.db import connection + + check_server_version(connection) _best_effort_call("sync_snmp_official_data", cleanup=True) _best_effort_call("collectstatic", interactive=False) + data_dir = resolve_data_dir() + data_dir.mkdir(parents=True, exist_ok=True) + gunicorn_cmd = [ "gunicorn", "LogstashUI.wsgi:application", @@ -326,6 +367,8 @@ def cmd_serve(args: argparse.Namespace) -> int: "-", "--error-logfile", "-", + "--pid", + str(data_dir / "gunicorn.pid"), ] if tls_on: _django_setup() diff --git a/src/logstashui/LogstashUI/tests/test_cli.py b/src/logstashui/LogstashUI/tests/test_cli.py index 2685c11..e610258 100644 --- a/src/logstashui/LogstashUI/tests/test_cli.py +++ b/src/logstashui/LogstashUI/tests/test_cli.py @@ -85,3 +85,40 @@ def fake_execvp(file, args): except SystemExit as exc: assert exc.code == 0 assert exec_called.get("file") == "gunicorn" + + +def test_parser_migrate_engine_requires_backup_flag(): + parser = build_parser() + ns = parser.parse_args(["migrate-engine", "--to", "postgresql"]) + assert ns.command == "migrate-engine" + assert ns.to == "postgresql" + assert ns.i_have_a_backup is False + + +def test_serve_adds_pidfile_and_warns_sqlite(monkeypatch, tmp_path, capsys): + from LogstashUI import cli + + monkeypatch.setattr(cli, "_manage", lambda argv: None) + monkeypatch.setenv("LOGSTASHUI_TLS", "false") + monkeypatch.setenv("LOGSTASHUI_DATA_DIR", str(tmp_path)) + monkeypatch.delenv("LOGSTASHUI_DB_ENGINE", raising=False) + + captured = {} + + def fake_execvp(file, args): + captured["file"] = file + captured["args"] = list(args) + raise SystemExit(0) + + monkeypatch.setattr(cli.os, "execvp", fake_execvp) + ns = Namespace(skip_migrate=True, no_tls=True, bind="127.0.0.1:8443", workers=2) + try: + cmd_serve(ns) + except SystemExit: + pass + assert "--pid" in captured["args"] + pid_idx = captured["args"].index("--pid") + assert captured["args"][pid_idx + 1].endswith("gunicorn.pid") + err = capsys.readouterr().err + assert "SQLite is the small-install default" in err + diff --git a/src/logstashui/LogstashUI/wsgi.py b/src/logstashui/LogstashUI/wsgi.py index 0eeb86e..3a9e067 100644 --- a/src/logstashui/LogstashUI/wsgi.py +++ b/src/logstashui/LogstashUI/wsgi.py @@ -19,8 +19,11 @@ os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'LogstashUI.settings') +# gunicorn --worker-class gevent monkey-patches select before this module loads. +# psycopg 3.1.14+ detects that and waits cooperatively; do not use psycogreen. application = get_wsgi_application() + # Quiet gevent/gunicorn spam: clients that reject the product CA (browser # probes, scanners, default-trust Python) abort the handshake with # CERTIFICATE_UNKNOWN / UNKNOWN_CA. Those are not app bugs; full greenlet From aa8349c0dfa948e477f6181dfd578bc0412a1d39 Mon Sep 17 00:00:00 2001 From: Aaron Mildenstein Date: Tue, 1 Sep 2026 17:58:37 -0600 Subject: [PATCH 09/62] test: mock check_server_version in serve SNMP test --- src/logstashui/LogstashUI/tests/test_cli.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/src/logstashui/LogstashUI/tests/test_cli.py b/src/logstashui/LogstashUI/tests/test_cli.py index e610258..18e2d42 100644 --- a/src/logstashui/LogstashUI/tests/test_cli.py +++ b/src/logstashui/LogstashUI/tests/test_cli.py @@ -62,6 +62,9 @@ def test_serve_snmp_commanderror_does_not_abort(monkeypatch): from LogstashUI import cli monkeypatch.setattr(cli, "_manage", lambda argv: None) + monkeypatch.setattr( + "LogstashUI.database.check_server_version", lambda conn: None + ) monkeypatch.setenv("LOGSTASHUI_TLS", "false") def fake_call(name, *args, **kwargs): From 31a12a0f4b61a2684d92da035d3147547a59c3b9 Mon Sep 17 00:00:00 2001 From: Aaron Mildenstein Date: Tue, 1 Sep 2026 18:01:30 -0600 Subject: [PATCH 10/62] test: Docker Postgres/MariaDB/MySQL matrix script --- bin/test_databases.bat | 41 +++++++++++++++ bin/test_databases.sh | 28 +++++++++++ docker/docker-compose.db.yml | 50 +++++++++++++++++++ .../LogstashUI/tests/test_migrate_live.py | 15 ++++++ 4 files changed, 134 insertions(+) create mode 100644 bin/test_databases.bat create mode 100755 bin/test_databases.sh create mode 100644 docker/docker-compose.db.yml create mode 100644 src/logstashui/LogstashUI/tests/test_migrate_live.py diff --git a/bin/test_databases.bat b/bin/test_databases.bat new file mode 100644 index 0000000..c68bb46 --- /dev/null +++ b/bin/test_databases.bat @@ -0,0 +1,41 @@ +@echo off +setlocal +cd /d "%~dp0\.." +where docker >nul 2>&1 +if errorlevel 1 ( + echo ERROR: Docker is required for bin\test_databases.bat + exit /b 1 +) +uv sync --extra databases --group dev +uv run pytest src\logstashui --no-cov +if errorlevel 1 exit /b 1 +docker compose -f docker\docker-compose.db.yml up -d --wait +if errorlevel 1 exit /b 1 + +set LOGSTASHUI_DB_ENGINE=postgresql +set LOGSTASHUI_DB_HOST=127.0.0.1 +set LOGSTASHUI_DB_PORT=55432 +set LOGSTASHUI_DB_NAME=logstashui +set LOGSTASHUI_DB_USER=logstashui +set LOGSTASHUI_DB_PASSWORD=logstashui +uv run pytest src\logstashui --no-cov +if errorlevel 1 goto :down + +set LOGSTASHUI_DB_ENGINE=mysql +set LOGSTASHUI_DB_PORT=53306 +set LOGSTASHUI_DB_USER=root +uv run pytest src\logstashui --no-cov +if errorlevel 1 goto :down + +set LOGSTASHUI_DB_PORT=53307 +uv run pytest src\logstashui --no-cov +if errorlevel 1 goto :down + +set LOGSTASHUI_LIVE_DB=1 +set LOGSTASHUI_LIVE_PG_PORT=55432 +set LOGSTASHUI_LIVE_MARIA_PORT=53306 +set LOGSTASHUI_LIVE_MYSQL_PORT=53307 +uv run pytest src\logstashui\LogstashUI\tests\test_migrate_live.py -v --no-cov + +:down +if /I not "%1"=="--keep" docker compose -f docker\docker-compose.db.yml down -v diff --git a/bin/test_databases.sh b/bin/test_databases.sh new file mode 100755 index 0000000..798ceb9 --- /dev/null +++ b/bin/test_databases.sh @@ -0,0 +1,28 @@ +#!/usr/bin/env bash +set -euo pipefail +ROOT="$(cd "$(dirname "$0")/.." && pwd)" +cd "$ROOT" +COMPOSE=(docker compose -f docker/docker-compose.db.yml) +KEEP=0 +if [[ "${1:-}" == "--keep" ]]; then KEEP=1; fi +if ! command -v docker >/dev/null; then + echo "ERROR: Docker is required for bin/test_databases.sh" >&2 + echo "Default pytest (SQLite) still works: uv run pytest" >&2 + exit 1 +fi +uv sync --extra databases --group dev +echo "==> SQLite pytest (no compose)" +uv run pytest src/logstashui --no-cov +echo "==> Starting Postgres / MariaDB / MySQL" +"${COMPOSE[@]}" up -d --wait +run_engine () { + local name="$1"; shift + echo "==> pytest on ${name}" + env "$@" uv run pytest src/logstashui --no-cov +} +run_engine postgresql LOGSTASHUI_DB_ENGINE=postgresql LOGSTASHUI_DB_HOST=127.0.0.1 LOGSTASHUI_DB_PORT=55432 LOGSTASHUI_DB_NAME=logstashui LOGSTASHUI_DB_USER=logstashui LOGSTASHUI_DB_PASSWORD=logstashui +run_engine mariadb LOGSTASHUI_DB_ENGINE=mysql LOGSTASHUI_DB_HOST=127.0.0.1 LOGSTASHUI_DB_PORT=53306 LOGSTASHUI_DB_NAME=logstashui LOGSTASHUI_DB_USER=root LOGSTASHUI_DB_PASSWORD=logstashui +run_engine mysql LOGSTASHUI_DB_ENGINE=mysql LOGSTASHUI_DB_HOST=127.0.0.1 LOGSTASHUI_DB_PORT=53307 LOGSTASHUI_DB_NAME=logstashui LOGSTASHUI_DB_USER=root LOGSTASHUI_DB_PASSWORD=logstashui +echo "==> Live migrator tests" +env LOGSTASHUI_LIVE_DB=1 LOGSTASHUI_LIVE_PG_PORT=55432 LOGSTASHUI_LIVE_MARIA_PORT=53306 LOGSTASHUI_LIVE_MYSQL_PORT=53307 LOGSTASHUI_LIVE_DB_USER=root LOGSTASHUI_LIVE_DB_PASSWORD=logstashui LOGSTASHUI_LIVE_PG_USER=logstashui uv run pytest src/logstashui/LogstashUI/tests/test_migrate_live.py -v --no-cov +if [[ "$KEEP" -eq 0 ]]; then "${COMPOSE[@]}" down -v; fi diff --git a/docker/docker-compose.db.yml b/docker/docker-compose.db.yml new file mode 100644 index 0000000..79efddf --- /dev/null +++ b/docker/docker-compose.db.yml @@ -0,0 +1,50 @@ +# Local/CI database matrix for LogstashUI. Not used by smoke compose. +# Host ports avoid clashing with a developer’s own 5432/3306. +name: logstashui-db-test + +services: + postgres: + image: postgres:16 + environment: + POSTGRES_USER: logstashui + POSTGRES_PASSWORD: logstashui + POSTGRES_DB: logstashui + ports: + - "55432:5432" + healthcheck: + test: ["CMD-SHELL", "pg_isready -U logstashui -d logstashui"] + interval: 2s + timeout: 5s + retries: 30 + + mariadb: + image: mariadb:11 + environment: + MARIADB_ROOT_PASSWORD: logstashui + MARIADB_DATABASE: logstashui + MARIADB_USER: logstashui + MARIADB_PASSWORD: logstashui + command: --character-set-server=utf8mb4 --collation-server=utf8mb4_bin + ports: + - "53306:3306" + healthcheck: + test: ["CMD", "healthcheck.sh", "--connect", "--innodb_initialized"] + interval: 3s + timeout: 5s + retries: 40 + + mysql: + image: mysql:8.0 + environment: + MYSQL_ROOT_PASSWORD: logstashui + MYSQL_DATABASE: logstashui + MYSQL_USER: logstashui + MYSQL_PASSWORD: logstashui + command: --character-set-server=utf8mb4 --collation-server=utf8mb4_bin + ports: + - "53307:3306" + healthcheck: + test: ["CMD", "mysqladmin", "ping", "-h", "127.0.0.1", "-ulogstashui", "-plogstashui"] + interval: 3s + timeout: 5s + retries: 40 diff --git a/src/logstashui/LogstashUI/tests/test_migrate_live.py b/src/logstashui/LogstashUI/tests/test_migrate_live.py new file mode 100644 index 0000000..0937800 --- /dev/null +++ b/src/logstashui/LogstashUI/tests/test_migrate_live.py @@ -0,0 +1,15 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +import os +import pytest + +pytestmark = pytest.mark.skipif( + os.environ.get("LOGSTASHUI_LIVE_DB") != "1", + reason="set LOGSTASHUI_LIVE_DB=1 (bin/test_databases.sh)", +) + + +def test_live_placeholder(): + pytest.skip("migrator live tests land in Task 7") From 3dfac3aa9e40d1189a2a77260f6ee7a3a19f653e Mon Sep 17 00:00:00 2001 From: Aaron Mildenstein Date: Tue, 1 Sep 2026 18:07:40 -0600 Subject: [PATCH 11/62] fix: propagate pytest failures from test_databases.bat --- bin/test_databases.bat | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/bin/test_databases.bat b/bin/test_databases.bat index c68bb46..4272c16 100644 --- a/bin/test_databases.bat +++ b/bin/test_databases.bat @@ -1,6 +1,7 @@ @echo off setlocal cd /d "%~dp0\.." +set FAIL=0 where docker >nul 2>&1 if errorlevel 1 ( echo ERROR: Docker is required for bin\test_databases.bat @@ -19,23 +20,34 @@ set LOGSTASHUI_DB_NAME=logstashui set LOGSTASHUI_DB_USER=logstashui set LOGSTASHUI_DB_PASSWORD=logstashui uv run pytest src\logstashui --no-cov -if errorlevel 1 goto :down +if errorlevel 1 ( + set FAIL=1 + goto :down +) set LOGSTASHUI_DB_ENGINE=mysql set LOGSTASHUI_DB_PORT=53306 set LOGSTASHUI_DB_USER=root uv run pytest src\logstashui --no-cov -if errorlevel 1 goto :down +if errorlevel 1 ( + set FAIL=1 + goto :down +) set LOGSTASHUI_DB_PORT=53307 uv run pytest src\logstashui --no-cov -if errorlevel 1 goto :down +if errorlevel 1 ( + set FAIL=1 + goto :down +) set LOGSTASHUI_LIVE_DB=1 set LOGSTASHUI_LIVE_PG_PORT=55432 set LOGSTASHUI_LIVE_MARIA_PORT=53306 set LOGSTASHUI_LIVE_MYSQL_PORT=53307 uv run pytest src\logstashui\LogstashUI\tests\test_migrate_live.py -v --no-cov +if errorlevel 1 set FAIL=1 :down if /I not "%1"=="--keep" docker compose -f docker\docker-compose.db.yml down -v +if "%FAIL%"=="1" exit /b 1 From ad87c4f0fd7eea83488f2aa97caaecd86ef76033 Mon Sep 17 00:00:00 2001 From: Aaron Mildenstein Date: Tue, 1 Sep 2026 18:12:45 -0600 Subject: [PATCH 12/62] feat: BETA migrate-engine sqlite to postgres/mysql --- src/logstashui/LogstashUI/cli.py | 3 + src/logstashui/LogstashUI/migrate_engine.py | 195 ++++++++++++++++++ .../LogstashUI/tests/test_migrate_engine.py | 68 ++++++ 3 files changed, 266 insertions(+) create mode 100644 src/logstashui/LogstashUI/migrate_engine.py create mode 100644 src/logstashui/LogstashUI/tests/test_migrate_engine.py diff --git a/src/logstashui/LogstashUI/cli.py b/src/logstashui/LogstashUI/cli.py index fb256ce..3a112a0 100644 --- a/src/logstashui/LogstashUI/cli.py +++ b/src/logstashui/LogstashUI/cli.py @@ -430,6 +430,9 @@ def main(argv: list[str] | None = None) -> int: return 0 if command == "systemd": return cmd_systemd(args) + if command == "migrate-engine": + from LogstashUI.migrate_engine import cmd_migrate_engine + return cmd_migrate_engine(args) if command == "serve": if not hasattr(args, "bind"): args = parser.parse_args(["serve"]) diff --git a/src/logstashui/LogstashUI/migrate_engine.py b/src/logstashui/LogstashUI/migrate_engine.py new file mode 100644 index 0000000..f761fb3 --- /dev/null +++ b/src/logstashui/LogstashUI/migrate_engine.py @@ -0,0 +1,195 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +"""BETA: copy SQLite data to PostgreSQL or MySQL. Stops gunicorn; does not restart serve.""" + +from __future__ import annotations + +import os +import signal +import sqlite3 +import subprocess +import sys +import tempfile +import time +from argparse import Namespace +from pathlib import Path + +from LogstashUI.database import canonical_engine +from LogstashUI.paths import resolve_data_dir + +_TARGET_ENGINES = frozenset({"postgresql", "mysql"}) +_DUMPDATA_EXCLUDES = ("contenttypes", "auth.permission", "sessions") + + +def run_manage(argv: list[str], extra_env: dict[str, str]) -> None: + env = os.environ.copy() + env.update(extra_env) + env.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") + code = ( + "import sys; from django.core.management import execute_from_command_line; " + "execute_from_command_line(['logstashui'] + sys.argv[1:])" + ) + cmd = [sys.executable, "-c", code, *argv] + proc = subprocess.run(cmd, env=env, check=False) + if proc.returncode != 0: + raise SystemExit(proc.returncode) + + +def wal_checkpoint(db_path: Path) -> None: + conn = sqlite3.connect(str(db_path)) + try: + conn.execute("PRAGMA wal_checkpoint(TRUNCATE)") + finally: + conn.close() + + +def stop_gunicorn(pidfile: Path) -> None: + pid = int(pidfile.read_text(encoding="utf-8").strip()) + try: + os.kill(pid, signal.SIGTERM) + except ProcessLookupError: + return + deadline = time.monotonic() + 30 + while True: + time.sleep(0.2) + try: + os.kill(pid, 0) + except ProcessLookupError: + return + if time.monotonic() >= deadline: + print( + f"gunicorn pid {pid} did not exit within 30s after SIGTERM", + file=sys.stderr, + ) + raise SystemExit(1) + + +def write_env_file(path: Path, engine: str) -> None: + pairs = [ + ("LOGSTASHUI_DB_ENGINE", engine), + ("LOGSTASHUI_DB_NAME", os.environ.get("LOGSTASHUI_DB_NAME")), + ("LOGSTASHUI_DB_HOST", os.environ.get("LOGSTASHUI_DB_HOST")), + ("LOGSTASHUI_DB_PORT", os.environ.get("LOGSTASHUI_DB_PORT")), + ("LOGSTASHUI_DB_USER", os.environ.get("LOGSTASHUI_DB_USER")), + ] + lines = [f"{key}={value}" for key, value in pairs if value] + existing = path.read_text(encoding="utf-8") if path.is_file() else "" + prefix = "" if not existing or existing.endswith("\n") else "\n" + with path.open("a", encoding="utf-8") as fh: + fh.write(prefix + "\n".join(lines) + "\n") + + +def _sqlsequencereset_to_dbshell(extra_env: dict[str, str]) -> None: + env = os.environ.copy() + env.update(extra_env) + env.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") + reset_code = ( + "import os, django\n" + "os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'LogstashUI.settings')\n" + "django.setup()\n" + "from django.apps import apps\n" + "from django.core.management import call_command\n" + "labels = [c.label for c in apps.get_app_configs() if c.models_module]\n" + "call_command('sqlsequencereset', *labels)\n" + ) + reset = subprocess.run( + [sys.executable, "-c", reset_code], + env=env, + check=False, + capture_output=True, + text=True, + ) + if reset.returncode != 0: + sys.stderr.write(reset.stderr or "") + raise SystemExit(reset.returncode) + dbshell_code = ( + "import sys; from django.core.management import execute_from_command_line; " + "execute_from_command_line(['logstashui'] + sys.argv[1:])" + ) + proc = subprocess.run( + [sys.executable, "-c", dbshell_code, "dbshell"], + env=env, + input=reset.stdout, + check=False, + text=True, + capture_output=True, + ) + if proc.returncode != 0: + sys.stderr.write(proc.stderr or "") + raise SystemExit(proc.returncode) + + +def cmd_migrate_engine(args: Namespace) -> int: + if not args.i_have_a_backup: + print( + "Refusing to run migrate-engine without --i-have-a-backup. " + "Back up db.sqlite3 (and the WAL) before copying data to another engine.", + file=sys.stderr, + ) + raise SystemExit(2) + + try: + engine = canonical_engine(getattr(args, "to", None)) + except RuntimeError as exc: + print(str(exc), file=sys.stderr) + raise SystemExit(2) from exc + if engine not in _TARGET_ENGINES: + print( + f"migrate-engine --to must be postgresql or mysql, not {engine}.", + file=sys.stderr, + ) + raise SystemExit(2) + + data_dir = resolve_data_dir(migrate_legacy=False) + sqlite_path = data_dir / "db.sqlite3" + if not sqlite_path.is_file(): + print( + f"SQLite database not found: {sqlite_path} (expected db.sqlite3).", + file=sys.stderr, + ) + raise SystemExit(1) + + print( + "BETA: migrate-engine stops gunicorn and does not restart serve. " + "Start `logstashui serve` yourself after verifying the target database.", + file=sys.stderr, + ) + + pidfile = Path(args.pid) if args.pid is not None else data_dir / "gunicorn.pid" + if pidfile.is_file(): + stop_gunicorn(pidfile) + + wal_checkpoint(sqlite_path) + + dump_fd, dump_name = tempfile.mkstemp(prefix="logstashui-migrate-", suffix=".json") + os.close(dump_fd) + dump_path = Path(dump_name) + sqlite_env = { + "LOGSTASHUI_DB_ENGINE": "sqlite", + "LOGSTASHUI_DB_NAME": str(sqlite_path), + } + target_env = {"LOGSTASHUI_DB_ENGINE": engine} + dump_argv = [ + "dumpdata", + "--natural-foreign", + "--natural-primary", + "--output", + str(dump_path), + ] + for label in _DUMPDATA_EXCLUDES: + dump_argv.extend(["--exclude", label]) + try: + run_manage(dump_argv, sqlite_env) + run_manage(["migrate", "--noinput"], target_env) + run_manage(["loaddata", str(dump_path)], target_env) + if engine == "postgresql": + _sqlsequencereset_to_dbshell(target_env) + finally: + dump_path.unlink(missing_ok=True) + + if args.write_env is not None: + write_env_file(Path(args.write_env), engine) + + return 0 diff --git a/src/logstashui/LogstashUI/tests/test_migrate_engine.py b/src/logstashui/LogstashUI/tests/test_migrate_engine.py new file mode 100644 index 0000000..988947b --- /dev/null +++ b/src/logstashui/LogstashUI/tests/test_migrate_engine.py @@ -0,0 +1,68 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +from argparse import Namespace +from pathlib import Path + +import pytest + +from LogstashUI import migrate_engine as me + + +def test_refuses_without_backup_flag(capsys): + ns = Namespace(to="postgresql", i_have_a_backup=False, pid=None, write_env=None) + with pytest.raises(SystemExit) as exc: + me.cmd_migrate_engine(ns) + assert exc.value.code == 2 + assert "back up" in capsys.readouterr().err.lower() + + +def test_refuses_sqlite_target(tmp_path, monkeypatch, capsys): + monkeypatch.setenv("LOGSTASHUI_DATA_DIR", str(tmp_path)) + ns = Namespace(to="sqlite", i_have_a_backup=True, pid=None, write_env=None) + with pytest.raises(SystemExit): + me.cmd_migrate_engine(ns) + + +def test_refuses_missing_sqlite_file(tmp_path, monkeypatch, capsys): + monkeypatch.setenv("LOGSTASHUI_DATA_DIR", str(tmp_path)) + ns = Namespace(to="postgresql", i_have_a_backup=True, pid=None, write_env=None) + with pytest.raises(SystemExit) as exc: + me.cmd_migrate_engine(ns) + assert exc.value.code == 1 + assert "db.sqlite3" in capsys.readouterr().err + + +def test_stop_pid_sends_sigterm(tmp_path, monkeypatch): + pidfile = tmp_path / "gunicorn.pid" + pidfile.write_text("4242\n") + sent = {} + calls = {"n": 0} + + def kill_then_gone(pid, sig): + calls["n"] += 1 + if calls["n"] == 1: + sent["pid"] = pid + sent["sig"] = sig + return + raise ProcessLookupError() + + monkeypatch.setattr(me.os, "kill", kill_then_gone) + monkeypatch.setattr(me.time, "sleep", lambda s: None) + me.stop_gunicorn(pidfile) + assert sent["pid"] == 4242 + assert sent["sig"] == me.signal.SIGTERM + + +def test_write_env_appends(tmp_path, monkeypatch): + envf = tmp_path / "logstashui.default" + envf.write_text("LOGSTASHUI_DATA_DIR=/var/lib/logstashui\n") + monkeypatch.setenv("LOGSTASHUI_DB_HOST", "db.example") + monkeypatch.setenv("LOGSTASHUI_DB_USER", "lsui") + monkeypatch.setenv("LOGSTASHUI_DB_NAME", "logstashui") + me.write_env_file(envf, "postgresql") + text = envf.read_text() + assert "LOGSTASHUI_DB_ENGINE=postgresql" in text + assert "LOGSTASHUI_DB_HOST=db.example" in text + assert "PASSWORD" not in text From d0d3dbc4b9344a8b1b815cd7b6f606fb97d44628 Mon Sep 17 00:00:00 2001 From: Aaron Mildenstein Date: Tue, 1 Sep 2026 18:25:49 -0600 Subject: [PATCH 13/62] fix: unlink gunicorn pidfile after migrate-engine SIGTERM --- src/logstashui/LogstashUI/migrate_engine.py | 11 ++++++++++- .../LogstashUI/tests/test_migrate_engine.py | 1 + 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/src/logstashui/LogstashUI/migrate_engine.py b/src/logstashui/LogstashUI/migrate_engine.py index f761fb3..7b36cd8 100644 --- a/src/logstashui/LogstashUI/migrate_engine.py +++ b/src/logstashui/LogstashUI/migrate_engine.py @@ -46,10 +46,18 @@ def wal_checkpoint(db_path: Path) -> None: def stop_gunicorn(pidfile: Path) -> None: - pid = int(pidfile.read_text(encoding="utf-8").strip()) + raw = pidfile.read_text(encoding="utf-8").strip() + try: + pid = int(raw) + except ValueError: + pid = 0 + if pid <= 0: + print(f"invalid gunicorn pid {raw!r} in {pidfile}", file=sys.stderr) + raise SystemExit(1) try: os.kill(pid, signal.SIGTERM) except ProcessLookupError: + pidfile.unlink(missing_ok=True) return deadline = time.monotonic() + 30 while True: @@ -57,6 +65,7 @@ def stop_gunicorn(pidfile: Path) -> None: try: os.kill(pid, 0) except ProcessLookupError: + pidfile.unlink(missing_ok=True) return if time.monotonic() >= deadline: print( diff --git a/src/logstashui/LogstashUI/tests/test_migrate_engine.py b/src/logstashui/LogstashUI/tests/test_migrate_engine.py index 988947b..797d652 100644 --- a/src/logstashui/LogstashUI/tests/test_migrate_engine.py +++ b/src/logstashui/LogstashUI/tests/test_migrate_engine.py @@ -53,6 +53,7 @@ def kill_then_gone(pid, sig): me.stop_gunicorn(pidfile) assert sent["pid"] == 4242 assert sent["sig"] == me.signal.SIGTERM + assert not pidfile.exists() def test_write_env_appends(tmp_path, monkeypatch): From 2b63f0bf2f7574aa59eebe0f90a405176454c73f Mon Sep 17 00:00:00 2001 From: Aaron Mildenstein Date: Tue, 1 Sep 2026 18:38:32 -0600 Subject: [PATCH 14/62] test: live sqlite dump/load onto Postgres MariaDB MySQL --- docker/db-init/mysql-extra.sql | 3 + docker/db-init/postgres-extra.sql | 1 + docker/docker-compose.db.yml | 6 + .../LogstashUI/tests/test_migrate_live.py | 143 +++++++++++++++++- 4 files changed, 151 insertions(+), 2 deletions(-) create mode 100644 docker/db-init/mysql-extra.sql create mode 100644 docker/db-init/postgres-extra.sql diff --git a/docker/db-init/mysql-extra.sql b/docker/db-init/mysql-extra.sql new file mode 100644 index 0000000..a27fc62 --- /dev/null +++ b/docker/db-init/mysql-extra.sql @@ -0,0 +1,3 @@ +CREATE DATABASE IF NOT EXISTS logstashui_migrate CHARACTER SET utf8mb4 COLLATE utf8mb4_bin; +GRANT ALL ON logstashui_migrate.* TO 'logstashui'@'%'; +GRANT ALL ON logstashui_migrate.* TO 'root'@'%'; diff --git a/docker/db-init/postgres-extra.sql b/docker/db-init/postgres-extra.sql new file mode 100644 index 0000000..fe18b44 --- /dev/null +++ b/docker/db-init/postgres-extra.sql @@ -0,0 +1 @@ +CREATE DATABASE logstashui_migrate OWNER logstashui; diff --git a/docker/docker-compose.db.yml b/docker/docker-compose.db.yml index 79efddf..d602fb6 100644 --- a/docker/docker-compose.db.yml +++ b/docker/docker-compose.db.yml @@ -11,6 +11,8 @@ services: POSTGRES_DB: logstashui ports: - "55432:5432" + volumes: + - ./db-init/postgres-extra.sql:/docker-entrypoint-initdb.d/02-migrate.sql:ro healthcheck: test: ["CMD-SHELL", "pg_isready -U logstashui -d logstashui"] interval: 2s @@ -27,6 +29,8 @@ services: command: --character-set-server=utf8mb4 --collation-server=utf8mb4_bin ports: - "53306:3306" + volumes: + - ./db-init/mysql-extra.sql:/docker-entrypoint-initdb.d/02-migrate.sql:ro healthcheck: test: ["CMD", "healthcheck.sh", "--connect", "--innodb_initialized"] interval: 3s @@ -43,6 +47,8 @@ services: command: --character-set-server=utf8mb4 --collation-server=utf8mb4_bin ports: - "53307:3306" + volumes: + - ./db-init/mysql-extra.sql:/docker-entrypoint-initdb.d/02-migrate.sql:ro healthcheck: test: ["CMD", "mysqladmin", "ping", "-h", "127.0.0.1", "-ulogstashui", "-plogstashui"] interval: 3s diff --git a/src/logstashui/LogstashUI/tests/test_migrate_live.py b/src/logstashui/LogstashUI/tests/test_migrate_live.py index 0937800..f54b7be 100644 --- a/src/logstashui/LogstashUI/tests/test_migrate_live.py +++ b/src/logstashui/LogstashUI/tests/test_migrate_live.py @@ -2,14 +2,153 @@ #or more contributor license agreements. Licensed under the Elastic License; #you may not use this file except in compliance with the Elastic License. +from argparse import Namespace +import json import os +import subprocess +import sys + import pytest +from LogstashUI import migrate_engine as me + pytestmark = pytest.mark.skipif( os.environ.get("LOGSTASHUI_LIVE_DB") != "1", reason="set LOGSTASHUI_LIVE_DB=1 (bin/test_databases.sh)", ) +_SEED = """ +import os +import django +os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") +django.setup() +from django.contrib.auth import get_user_model +from PipelineManager.models import Policy +User = get_user_model() +User.objects.create_user(username="migrate-user", password="migrate-pass") +Policy.objects.create( + name="Migrate Policy", + logstash_yml="http.host: 0.0.0.0", + jvm_options="-Xms1g", + log4j2_properties="status = error", +) +""" + +_COUNT = """ +import json +import os +import django +os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") +django.setup() +from django.contrib.auth import get_user_model +from PipelineManager.models import Policy +User = get_user_model() +print(json.dumps({ + "users": User.objects.count(), + "policies": Policy.objects.count(), + "migrate_user": User.objects.filter(username="migrate-user").count(), + "migrate_policy": Policy.objects.filter(name="Migrate Policy").count(), +})) +""" + +_TARGET_KEYS = ( + "LOGSTASHUI_DATA_DIR", + "LOGSTASHUI_DB_ENGINE", + "LOGSTASHUI_DB_HOST", + "LOGSTASHUI_DB_PORT", + "LOGSTASHUI_DB_USER", + "LOGSTASHUI_DB_PASSWORD", + "LOGSTASHUI_DB_NAME", +) + + +def _run_python(code: str, extra_env: dict[str, str]) -> str: + env = os.environ.copy() + env.update(extra_env) + env.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") + proc = subprocess.run( + [sys.executable, "-c", code], + env=env, + check=False, + capture_output=True, + text=True, + ) + if proc.returncode != 0: + raise AssertionError( + f"python -c exited {proc.returncode}\n" + f"stdout:\n{proc.stdout}\nstderr:\n{proc.stderr}" + ) + return proc.stdout + + +def _run_to(tmp_path, engine: str, *, port: str, user: str) -> None: + data_dir = tmp_path + sqlite_path = data_dir / "db.sqlite3" + sqlite_env = { + "LOGSTASHUI_DATA_DIR": str(data_dir), + "LOGSTASHUI_DB_ENGINE": "sqlite", + "LOGSTASHUI_DB_NAME": str(sqlite_path), + } + me.run_manage(["migrate", "--noinput"], sqlite_env) + _run_python(_SEED, sqlite_env) + + target = { + "LOGSTASHUI_DATA_DIR": str(data_dir), + "LOGSTASHUI_DB_ENGINE": engine, + "LOGSTASHUI_DB_HOST": "127.0.0.1", + "LOGSTASHUI_DB_PORT": str(port), + "LOGSTASHUI_DB_USER": user, + "LOGSTASHUI_DB_PASSWORD": os.environ.get( + "LOGSTASHUI_LIVE_DB_PASSWORD", "logstashui" + ), + "LOGSTASHUI_DB_NAME": "logstashui_migrate", + } + previous = {key: os.environ.get(key) for key in _TARGET_KEYS} + try: + os.environ.update(target) + ns = Namespace(to=engine, i_have_a_backup=True, pid=None, write_env=None) + try: + rc = me.cmd_migrate_engine(ns) + except SystemExit as exc: + raise AssertionError(f"cmd_migrate_engine SystemExit {exc.code}") from exc + assert rc == 0 + raw = _run_python(_COUNT, target) + finally: + for key, value in previous.items(): + if value is None: + os.environ.pop(key, None) + else: + os.environ[key] = value + + counts = json.loads(raw.strip().splitlines()[-1]) + assert counts["users"] >= 1 + assert counts["policies"] >= 1 + assert counts["migrate_user"] == 1 + assert counts["migrate_policy"] == 1 + + +def test_live_postgres(tmp_path): + _run_to( + tmp_path, + "postgresql", + port=os.environ.get("LOGSTASHUI_LIVE_PG_PORT", "55432"), + user=os.environ.get("LOGSTASHUI_LIVE_PG_USER", "logstashui"), + ) + + +def test_live_mariadb(tmp_path): + _run_to( + tmp_path, + "mysql", + port=os.environ.get("LOGSTASHUI_LIVE_MARIA_PORT", "53306"), + user=os.environ.get("LOGSTASHUI_LIVE_DB_USER", "root"), + ) + -def test_live_placeholder(): - pytest.skip("migrator live tests land in Task 7") +def test_live_mysql(tmp_path): + _run_to( + tmp_path, + "mysql", + port=os.environ.get("LOGSTASHUI_LIVE_MYSQL_PORT", "53307"), + user=os.environ.get("LOGSTASHUI_LIVE_DB_USER", "root"), + ) From 24d8975869a55ac79c46804a552d22ae135115ea Mon Sep 17 00:00:00 2001 From: Aaron Mildenstein Date: Tue, 1 Sep 2026 18:46:11 -0600 Subject: [PATCH 15/62] docs: multi-engine env, systemd, Docker extras, CI matrix --- .github/workflows/test-databases.yml | 20 +++++++ CHANGELOG.md | 11 ++++ docker/Dockerfile | 2 +- .../logstashui/configuration/environment.md | 52 +++++++++++++++++-- docs/docs/logstashui/general/deploy.md | 6 +-- src/logstashui/LogstashUI/cli.py | 48 +++++++++++++++++ .../LogstashUI/packaging/logstashui.default | 13 ++++- src/logstashui/LogstashUI/tests/test_cli.py | 35 +++++++++++++ 8 files changed, 177 insertions(+), 10 deletions(-) create mode 100644 .github/workflows/test-databases.yml diff --git a/.github/workflows/test-databases.yml b/.github/workflows/test-databases.yml new file mode 100644 index 0000000..da53c0c --- /dev/null +++ b/.github/workflows/test-databases.yml @@ -0,0 +1,20 @@ +name: Database matrix + +on: + pull_request: + push: + branches: [main, master] + +jobs: + db: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: astral-sh/setup-uv@v4 + with: + python-version: "3.12" + - uses: actions/setup-python@v5 + with: + python-version: "3.12" + - name: Run database matrix + run: bash bin/test_databases.sh diff --git a/CHANGELOG.md b/CHANGELOG.md index 1b960a1..c5e14af 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,14 @@ +## [0.5.2] - Multi-database + +Package version remains **0.5.1** until release tagging; this documents the 0.5.2 database work. + +- `LOGSTASHUI_DB_ENGINE=sqlite|postgresql|mysql` (MariaDB uses `mysql`). Discrete `LOGSTASHUI_DB_HOST/PORT/NAME/USER/PASSWORD` plus SSL and `CONN_MAX_AGE`. No YAML, no `DATABASE_URL`. +- Default is still SQLite. `logstashui serve` warns when `LOGSTASHUI_WORKERS>1` on SQLite. +- Native extras: `LogstashUI[postgres]`, `[mysql]`, `[databases]`. Container image installs `[databases]`. +- BETA `logstashui migrate-engine --to postgresql|mysql --i-have-a-backup` copies SQLite → server (stops gunicorn; does not restart). +- `bin/test_databases.sh` runs the suite and dump/load against local Docker Postgres, MariaDB, and MySQL. + + ## [0.5.1] - Agent control plane, SNMP NMS, dual HTTPS - 08/31/2026 Package version is **0.5.1** (`pyproject.toml`). Preferred LogstashAgent version is **0.5.1**. diff --git a/docker/Dockerfile b/docker/Dockerfile index 4ac81fa..57deccb 100644 --- a/docker/Dockerfile +++ b/docker/Dockerfile @@ -26,7 +26,7 @@ WORKDIR /app/src/logstashui/theme/static_src RUN rm -rf node_modules && npm install && npm run build && rm -rf node_modules WORKDIR /app -RUN uv pip install --system --no-cache /app +RUN uv pip install --system --no-cache "/app[databases]" # Pinned uid so K8s runAsUser/fsGroup is stable. Entrypoint starts as root, # chowns only LOGSTASHUI_DATA_DIR when needed, then drops to this user (or PUID). diff --git a/docs/docs/logstashui/configuration/environment.md b/docs/docs/logstashui/configuration/environment.md index 7c9a18b..6c228c5 100644 --- a/docs/docs/logstashui/configuration/environment.md +++ b/docs/docs/logstashui/configuration/environment.md @@ -59,13 +59,55 @@ Booleans accept `true`/`false`, `1`/`0`, `yes`/`no`, `on`/`off`. --- -## Database (sqlite only) +## Database | Variable | Default | Purpose | |---|---|---| -| `LOGSTASHUI_DB_ENGINE` | `sqlite` | **Only `sqlite` is implemented** | +| `LOGSTASHUI_DB_ENGINE` | `sqlite` | `sqlite`, `postgresql`, or `mysql` (MariaDB uses `mysql`). Aliases: `sqlite3`, `postgres`, `mariadb`, `my` | +| `LOGSTASHUI_DB_NAME` | sqlite: `$LOGSTASHUI_DATA_DIR/db.sqlite3`; else `logstashui` | Database name / sqlite path | +| `LOGSTASHUI_DB_HOST` | empty | **Required** for postgresql/mysql | +| `LOGSTASHUI_DB_PORT` | `5432` / `3306` | | +| `LOGSTASHUI_DB_USER` | empty | **Required** for postgresql/mysql | +| `LOGSTASHUI_DB_PASSWORD` | empty | Put in a Secret / `chmod 640` EnvironmentFile | +| `LOGSTASHUI_DB_SSLMODE` | postgres: `prefer` | `disable` `allow` `prefer` `require` `verify-ca` `verify-full` | +| `LOGSTASHUI_DB_SSL_CA` | empty | CA file for mysql TLS and postgres `verify-*` | +| `LOGSTASHUI_DB_CONN_MAX_AGE` | `60` | Persistent connections (seconds); `0` closes per request | +| `LOGSTASHUI_DB_CONN_HEALTH_CHECKS` | `true` | Django `CONN_HEALTH_CHECKS` | -`postgresql` and `mysql` are reserved names: setting them **fails at startup** until those backends land. Keep a PVC on `LOGSTASHUI_DATA_DIR` even after that work (TLS material and secrets still live there). +Floors: PostgreSQL 14+, MariaDB 10.6+, MySQL 8.0+. Create MySQL/MariaDB as `utf8mb4` / `utf8mb4_bin` so unique names match SQLite/Postgres case-sensitivity. + +**Install extras (native pip/uv):** `uv pip install 'LogstashUI[postgres]'`, `'LogstashUI[mysql]'`, or `'LogstashUI[databases]'`. The Docker/K8s image already installs `[databases]`. Missing driver fails at startup with that extra name. + +`LOGSTASHUI_DATA_DIR` is still required when the database is remote (TLS, `.django_secret_key`, logs, staticfiles). + +**SQLite scale:** `logstashui serve` logs a warning when engine is sqlite and `LOGSTASHUI_WORKERS` > 1. Use PostgreSQL or MySQL/MariaDB for concurrent agents. Startup still succeeds. + +**Connections:** gunicorn remains gevent (`--worker-connections 1000`). Keep `LOGSTASHUI_WORKERS` × in-flight requests under the server `max_connections`. PgBouncer (or equivalent) is optional, not required. + +No `DATABASE_URL`. No YAML. + +### Offline migration (supported) + +1. `systemctl stop logstashui` (or stop the container). +2. Copy `$LOGSTASHUI_DATA_DIR/db.sqlite3` somewhere safe. Keep the rest of `DATA_DIR` (same Django secret key). +3. Create the server database (`utf8mb4_bin` on MySQL/MariaDB). +4. Set `LOGSTASHUI_DB_*` for the target. Native installs need the matching extra. +5. `logstashui manage dumpdata --natural-foreign --natural-primary -e contenttypes -e auth.permission -e sessions -o dump.json` while still on sqlite, **or** use the BETA CLI below. +6. `logstashui manage migrate --noinput && logstashui manage loaddata dump.json` +7. Postgres: `logstashui manage sqlsequencereset PipelineManager Management SNMP AI auth admin | logstashui manage dbshell` +8. Start LogstashUI. Log in again (sessions were not copied). + +### BETA CLI + +```bash +# env already points at the empty target server; sqlite file still in DATA_DIR +sudo systemctl stop logstashui # avoid Restart= racing SIGTERM +logstashui migrate-engine --to postgresql --i-have-a-backup +# optional: --write-env /etc/default/logstashui +sudo systemctl start logstashui +``` + +`--to mysql` covers MariaDB and MySQL. The command SIGTERMs gunicorn if `$LOGSTASHUI_DATA_DIR/gunicorn.pid` is live, checkpoints WAL, dump/load, and **does not** restart serve. --- @@ -73,8 +115,8 @@ Booleans accept `true`/`false`, `1`/`0`, `yes`/`no`, `on`/`off`. Minimum: -1. Deployment env from a ConfigMap + Secret -2. PVC mounted at `/var/lib/logstashui` +1. Deployment env from a ConfigMap (`LOGSTASHUI_DB_ENGINE` / `HOST` / `NAME` / `USER`) + Secret (`SECRET_KEY`, `LOGSTASHUI_DB_PASSWORD`) +2. PVC mounted at `/var/lib/logstashui` (still required when the database is external) 3. Container image `CMD` is `logstashui serve` (already the Docker default) 4. Optional ingress: `LOGSTASHUI_TLS=false` and `CSRF_TRUSTED_ORIGINS=https://` diff --git a/docs/docs/logstashui/general/deploy.md b/docs/docs/logstashui/general/deploy.md index 219b7c5..f0fb1bb 100644 --- a/docs/docs/logstashui/general/deploy.md +++ b/docs/docs/logstashui/general/deploy.md @@ -33,11 +33,11 @@ Then browse to `https://:8443`. This is **embedded mode** — two containers: **LogstashUI** (gunicorn HTTPS on **8443**) and **LogstashAgent** (uvicorn HTTPS on **9500**). There is **no nginx**. Configuration is [environment variables](/docs/docs/logstashui/configuration/environment.md). -**Data directory:** Runtime state (sqlite, TLS, secrets, logs) lives **outside** `src/`. From a git checkout, Docker Compose bind-mounts `/logstashui_data` to `/var/lib/logstashui` and sets `LOGSTASHUI_DATA_DIR`. Native CLI default (no env) is `$(pwd)/logstashui_data`. Do not store this under `src/logstashui/data`. +**Data directory:** Runtime state (sqlite, TLS, secrets, logs) lives **outside** `src/`. From a git checkout, Docker Compose bind-mounts `/logstashui_data` to `/var/lib/logstashui` and sets `LOGSTASHUI_DATA_DIR`. Native CLI default (no env) is `$(pwd)/logstashui_data`. Do not store this under `src/logstashui/data`. The database may be external Postgres/MySQL; the PVC/bind-mount is still required for TLS and secrets. On **Linux Docker**, that bind-mount keeps host file ownership. The image entrypoint starts as root, chowns **only** the data directory if it is not writable, then drops to `PUID`/`PGID` (from `start_logstashui.sh`: your uid/gid) or image user **appuser (10001)**. Gunicorn never stays root. Docker Desktop (macOS/Windows) usually maps UIDs already; the same path is a no-op chown. If the directory is still unwritable, startup exits before migrate so sqlite/TLS are not created as the wrong user. -**Kubernetes:** use a PVC at `/var/lib/logstashui`, `runAsUser: 10001`, `runAsNonRoot: true`, and `fsGroup: 10001`. The entrypoint skips chown when it is not root. +**Kubernetes:** use a PVC at `/var/lib/logstashui`, `runAsUser: 10001`, `runAsNonRoot: true`, and `fsGroup: 10001`. The entrypoint skips chown when it is not root. ConfigMap `LOGSTASHUI_DB_ENGINE` / `LOGSTASHUI_DB_HOST` / `LOGSTASHUI_DB_NAME` / `LOGSTASHUI_DB_USER`; Secret `LOGSTASHUI_DB_PASSWORD`. The PVC is still required when the database is external. **HTTPS / product CA:** On first start, LogstashUI writes a product CA and a UI server certificate under `$LOGSTASHUI_DATA_DIR/tls/` (`ui-server.crt` / `ui-server.key`). Gunicorn presents that cert on port **8443** (ports under 1000 would need root). The product leaf SANs include `localhost`, `logstashui`, **all non-loopback host IPs**, and **PTR reverse-DNS FQDNs** for those IPs when available (injected by `start_logstashui.sh` as `LOGSTASHUI_HOST_*` / `LOGSTASHUI_TLS_SANS`, because the container cannot see the host LAN addresses by itself). Bare short hostnames (common on macOS) are replaced by reverse-lookup FQDNs when PTR records exist. Changing the Agent callback URL or those env SANs **re-issues** the product leaf on next startup (or Settings save); restart the UI container so gunicorn reloads the file. Agents: @@ -72,7 +72,7 @@ sudo systemctl enable --now logstashui Set `LOGSTASHUI_DATA_DIR=/var/lib/logstashui` in `/etc/default/logstashui` (the generator does this). See [environment configuration](/docs/docs/logstashui/configuration/environment.md). -Kubernetes: same image as Option 1, env/ConfigMap only, PVC at `/var/lib/logstashui`. No YAML mount. +Kubernetes: same image as Option 1, env/ConfigMap (`LOGSTASHUI_DB_ENGINE` / `HOST` / `NAME` / `USER`) plus Secret (`LOGSTASHUI_DB_PASSWORD`), PVC at `/var/lib/logstashui` (still required when the database is external). No YAML mount. --- diff --git a/src/logstashui/LogstashUI/cli.py b/src/logstashui/LogstashUI/cli.py index 3a112a0..3c71716 100644 --- a/src/logstashui/LogstashUI/cli.py +++ b/src/logstashui/LogstashUI/cli.py @@ -111,6 +111,11 @@ def build_parser() -> argparse.ArgumentParser: default="false", choices=("true", "false"), ) + systemd.add_argument("--db-engine", default="") + systemd.add_argument("--db-host", default="") + systemd.add_argument("--db-name", default="") + systemd.add_argument("--db-user", default="") + systemd.add_argument("--db-port", default="") parser.set_defaults( command="serve", bind=os.environ.get("LOGSTASHUI_BIND", "0.0.0.0:8443"), @@ -148,6 +153,11 @@ def render_default_env( tls_sans: str, agent_ui_url: str, no_auth: str, + db_engine: str = "", + db_host: str = "", + db_name: str = "", + db_user: str = "", + db_port: str = "", ) -> str: sample = _packaging_file("logstashui.default") replacements = { @@ -173,6 +183,18 @@ def render_default_env( extras.append(f"LOGSTASHUI_TLS_SANS={tls_sans}") if agent_ui_url: extras.append(f"LOGSTASHUI_AGENT_UI_URL={agent_ui_url}") + from LogstashUI.database import canonical_engine + + if db_engine and canonical_engine(db_engine) != "sqlite": + extras.append(f"LOGSTASHUI_DB_ENGINE={canonical_engine(db_engine)}") + if db_host: + extras.append(f"LOGSTASHUI_DB_HOST={db_host}") + if db_port: + extras.append(f"LOGSTASHUI_DB_PORT={db_port}") + if db_name: + extras.append(f"LOGSTASHUI_DB_NAME={db_name}") + if db_user: + extras.append(f"LOGSTASHUI_DB_USER={db_user}") if extras: text = text.rstrip() + "\n\n# Values from logstashui systemd\n" + "\n".join(extras) + "\n" return text @@ -204,11 +226,18 @@ def install_systemd( tls_sans: str = "", agent_ui_url: str = "", no_auth: str = "false", + db_engine: str = "", + db_host: str = "", + db_name: str = "", + db_user: str = "", + db_port: str = "", dry_run: bool = False, print_only: bool = False, interactive: bool = False, ) -> dict: if interactive and not dry_run and output_dir is None: + from LogstashUI.database import canonical_engine + exec_start = _prompt( "Path to logstashui executable", exec_start or _default_exec_start(), @@ -226,6 +255,15 @@ def install_systemd( tls_sans = _prompt("LOGSTASHUI_TLS_SANS", tls_sans) agent_ui_url = _prompt("LOGSTASHUI_AGENT_UI_URL", agent_ui_url) no_auth = _prompt("LOGSTASHUI_NO_AUTH (true/false)", no_auth) + db_engine = _prompt( + "LOGSTASHUI_DB_ENGINE (sqlite/postgresql/mysql)", + db_engine or "sqlite", + ) + if canonical_engine(db_engine) != "sqlite": + db_host = _prompt("LOGSTASHUI_DB_HOST", db_host) + db_port = _prompt("LOGSTASHUI_DB_PORT", db_port) + db_name = _prompt("LOGSTASHUI_DB_NAME", db_name or "logstashui") + db_user = _prompt("LOGSTASHUI_DB_USER", db_user) if not exec_start: exec_start = _default_exec_start() @@ -248,6 +286,11 @@ def install_systemd( tls_sans=tls_sans, agent_ui_url=agent_ui_url, no_auth=no_auth, + db_engine=db_engine, + db_host=db_host, + db_name=db_name, + db_user=db_user, + db_port=db_port, ) if print_only: @@ -414,6 +457,11 @@ def cmd_systemd(args: argparse.Namespace) -> int: tls_sans=args.tls_sans, agent_ui_url=args.agent_ui_url, no_auth=args.no_auth, + db_engine=args.db_engine, + db_host=args.db_host, + db_name=args.db_name, + db_user=args.db_user, + db_port=args.db_port, dry_run=dry_run, print_only=args.print_only, interactive=interactive, diff --git a/src/logstashui/LogstashUI/packaging/logstashui.default b/src/logstashui/LogstashUI/packaging/logstashui.default index fc10229..a0ace29 100644 --- a/src/logstashui/LogstashUI/packaging/logstashui.default +++ b/src/logstashui/LogstashUI/packaging/logstashui.default @@ -43,10 +43,21 @@ LOGSTASHUI_INCLUDE_CA_FINGERPRINT=true # In-app markdown docs (wheel ships a copy; checkout uses the git tree) # LOGSTASHUI_DOCS_DIR= -# Future database backends (not implemented — sqlite only for now) +# Database. Unset engine = sqlite at $LOGSTASHUI_DATA_DIR/db.sqlite3. +# DATA_DIR is still required for TLS, secrets, logs, staticfiles. # LOGSTASHUI_DB_ENGINE=sqlite # LOGSTASHUI_DB_NAME=logstashui # LOGSTASHUI_DB_HOST= # LOGSTASHUI_DB_PORT= # LOGSTASHUI_DB_USER= # LOGSTASHUI_DB_PASSWORD= +# LOGSTASHUI_DB_SSLMODE=prefer +# LOGSTASHUI_DB_SSL_CA= +# LOGSTASHUI_DB_CONN_MAX_AGE=60 +# LOGSTASHUI_DB_CONN_HEALTH_CHECKS=true +# +# Native extras: uv pip install 'LogstashUI[postgres]' or 'LogstashUI[mysql]' +# or 'LogstashUI[databases]'. The container image already includes both drivers. +# +# Create MySQL/MariaDB with utf8mb4_bin: +# CREATE DATABASE logstashui CHARACTER SET utf8mb4 COLLATE utf8mb4_bin; diff --git a/src/logstashui/LogstashUI/tests/test_cli.py b/src/logstashui/LogstashUI/tests/test_cli.py index 18e2d42..d49f57b 100644 --- a/src/logstashui/LogstashUI/tests/test_cli.py +++ b/src/logstashui/LogstashUI/tests/test_cli.py @@ -55,6 +55,41 @@ def test_systemd_dry_run_writes_unit_and_default(tmp_path): assert "LOGSTASHUI_NO_AUTH=false" in env_text assert result["unit"] == unit assert result["default"] == envf + assert "LOGSTASHUI_DB_ENGINE=postgresql" not in env_text + assert "# LOGSTASHUI_DB_ENGINE=sqlite" in env_text + + +def test_systemd_env_includes_postgres_when_passed(tmp_path): + result = install_systemd( + output_dir=tmp_path, + exec_start="/usr/bin/logstashui serve", + user="logstashui", + group="logstashui", + data_dir="/var/lib/logstashui", + bind="0.0.0.0:8443", + workers=2, + allowed_hosts="*", + csrf_trusted_origins="", + tls="true", + host_hostname="", + host_ips="", + tls_sans="", + agent_ui_url="", + no_auth="false", + dry_run=True, + db_engine="postgresql", + db_host="db.example", + db_port="5432", + db_name="logstashui", + db_user="lsui", + ) + text = (tmp_path / "logstashui.default").read_text() + assert "LOGSTASHUI_DB_ENGINE=postgresql" in text + assert "LOGSTASHUI_DB_HOST=db.example" in text + assert "LOGSTASHUI_DB_PORT=5432" in text + assert "LOGSTASHUI_DB_NAME=logstashui" in text + assert "LOGSTASHUI_DB_USER=lsui" in text + assert result["default"] == tmp_path / "logstashui.default" def test_serve_snmp_commanderror_does_not_abort(monkeypatch): From 93316765f9ed9b1d55b14de63399bdf15fe31118 Mon Sep 17 00:00:00 2001 From: Aaron Mildenstein Date: Tue, 1 Sep 2026 18:51:00 -0600 Subject: [PATCH 16/62] docs: dump sqlite before switching LOGSTASHUI_DB_* --- docs/docs/logstashui/configuration/environment.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/docs/logstashui/configuration/environment.md b/docs/docs/logstashui/configuration/environment.md index 6c228c5..f31effa 100644 --- a/docs/docs/logstashui/configuration/environment.md +++ b/docs/docs/logstashui/configuration/environment.md @@ -90,9 +90,9 @@ No `DATABASE_URL`. No YAML. 1. `systemctl stop logstashui` (or stop the container). 2. Copy `$LOGSTASHUI_DATA_DIR/db.sqlite3` somewhere safe. Keep the rest of `DATA_DIR` (same Django secret key). -3. Create the server database (`utf8mb4_bin` on MySQL/MariaDB). -4. Set `LOGSTASHUI_DB_*` for the target. Native installs need the matching extra. -5. `logstashui manage dumpdata --natural-foreign --natural-primary -e contenttypes -e auth.permission -e sessions -o dump.json` while still on sqlite, **or** use the BETA CLI below. +3. Dump **from sqlite** while `LOGSTASHUI_DB_ENGINE` is still sqlite (or unset): `logstashui manage dumpdata --natural-foreign --natural-primary -e contenttypes -e auth.permission -e sessions -o dump.json`. Do not set target `LOGSTASHUI_DB_*` yet, or dumpdata will dump the empty server database. +4. Create the server database (`utf8mb4_bin` on MySQL/MariaDB). +5. **Then** set `LOGSTASHUI_DB_*` for the target. Native installs need the matching extra. 6. `logstashui manage migrate --noinput && logstashui manage loaddata dump.json` 7. Postgres: `logstashui manage sqlsequencereset PipelineManager Management SNMP AI auth admin | logstashui manage dbshell` 8. Start LogstashUI. Log in again (sessions were not copied). @@ -107,7 +107,7 @@ logstashui migrate-engine --to postgresql --i-have-a-backup sudo systemctl start logstashui ``` -`--to mysql` covers MariaDB and MySQL. The command SIGTERMs gunicorn if `$LOGSTASHUI_DATA_DIR/gunicorn.pid` is live, checkpoints WAL, dump/load, and **does not** restart serve. +`--to mysql` covers MariaDB and MySQL. The command SIGTERMs gunicorn if `$LOGSTASHUI_DATA_DIR/gunicorn.pid` is live, checkpoints WAL, dump/load from the sqlite file in `DATA_DIR` (regardless of target `LOGSTASHUI_DB_*`), and **does not** restart serve. --- From 470665ac1fc035b50f368a1a0654cc73e53d859e Mon Sep 17 00:00:00 2001 From: Aaron Mildenstein Date: Tue, 1 Sep 2026 19:07:45 -0600 Subject: [PATCH 17/62] fix: migrate-engine dump uses package path; reset sequences without psql --- src/logstashui/LogstashUI/migrate_engine.py | 45 ++++++++++--------- .../LogstashUI/tests/test_migrate_engine.py | 41 +++++++++++++++++ .../LogstashUI/tests/test_migrate_live.py | 1 + 3 files changed, 66 insertions(+), 21 deletions(-) diff --git a/src/logstashui/LogstashUI/migrate_engine.py b/src/logstashui/LogstashUI/migrate_engine.py index 7b36cd8..5ec6c75 100644 --- a/src/logstashui/LogstashUI/migrate_engine.py +++ b/src/logstashui/LogstashUI/migrate_engine.py @@ -23,10 +23,19 @@ _DUMPDATA_EXCLUDES = ("contenttypes", "auth.permission", "sessions") +def _with_package_pythonpath(env: dict[str, str]) -> dict[str, str]: + """Subprocesses must import this tree, not a stale site-packages copy.""" + pkg_root = str(Path(__file__).resolve().parent.parent) + existing = env.get("PYTHONPATH", "") + env["PYTHONPATH"] = pkg_root if not existing else os.pathsep.join([pkg_root, existing]) + return env + + def run_manage(argv: list[str], extra_env: dict[str, str]) -> None: env = os.environ.copy() env.update(extra_env) env.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") + _with_package_pythonpath(env) code = ( "import sys; from django.core.management import execute_from_command_line; " "execute_from_command_line(['logstashui'] + sys.argv[1:])" @@ -90,41 +99,35 @@ def write_env_file(path: Path, engine: str) -> None: fh.write(prefix + "\n".join(lines) + "\n") -def _sqlsequencereset_to_dbshell(extra_env: dict[str, str]) -> None: +def _reset_postgres_sequences(extra_env: dict[str, str]) -> None: + """Apply sqlsequencereset via the Django connection (no psql CLI).""" env = os.environ.copy() env.update(extra_env) env.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") + _with_package_pythonpath(env) reset_code = ( "import os, django\n" "os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'LogstashUI.settings')\n" "django.setup()\n" "from django.apps import apps\n" - "from django.core.management import call_command\n" - "labels = [c.label for c in apps.get_app_configs() if c.models_module]\n" - "call_command('sqlsequencereset', *labels)\n" + "from django.core.management.color import no_style\n" + "from django.db import connection\n" + "models = [\n" + " m for c in apps.get_app_configs() if c.models_module\n" + " for m in c.get_models(include_auto_created=True)\n" + "]\n" + "sql_list = connection.ops.sequence_reset_sql(no_style(), models)\n" + "with connection.cursor() as cursor:\n" + " for sql in sql_list:\n" + " cursor.execute(sql)\n" ) - reset = subprocess.run( + proc = subprocess.run( [sys.executable, "-c", reset_code], env=env, check=False, capture_output=True, text=True, ) - if reset.returncode != 0: - sys.stderr.write(reset.stderr or "") - raise SystemExit(reset.returncode) - dbshell_code = ( - "import sys; from django.core.management import execute_from_command_line; " - "execute_from_command_line(['logstashui'] + sys.argv[1:])" - ) - proc = subprocess.run( - [sys.executable, "-c", dbshell_code, "dbshell"], - env=env, - input=reset.stdout, - check=False, - text=True, - capture_output=True, - ) if proc.returncode != 0: sys.stderr.write(proc.stderr or "") raise SystemExit(proc.returncode) @@ -194,7 +197,7 @@ def cmd_migrate_engine(args: Namespace) -> int: run_manage(["migrate", "--noinput"], target_env) run_manage(["loaddata", str(dump_path)], target_env) if engine == "postgresql": - _sqlsequencereset_to_dbshell(target_env) + _reset_postgres_sequences(target_env) finally: dump_path.unlink(missing_ok=True) diff --git a/src/logstashui/LogstashUI/tests/test_migrate_engine.py b/src/logstashui/LogstashUI/tests/test_migrate_engine.py index 797d652..0447da5 100644 --- a/src/logstashui/LogstashUI/tests/test_migrate_engine.py +++ b/src/logstashui/LogstashUI/tests/test_migrate_engine.py @@ -67,3 +67,44 @@ def test_write_env_appends(tmp_path, monkeypatch): assert "LOGSTASHUI_DB_ENGINE=postgresql" in text assert "LOGSTASHUI_DB_HOST=db.example" in text assert "PASSWORD" not in text + + +def test_run_manage_sets_package_pythonpath(monkeypatch): + captured = {} + + def fake_run(cmd, env=None, check=False): + captured["env"] = env + class Result: + returncode = 0 + return Result() + + monkeypatch.setattr(me.subprocess, "run", fake_run) + me.run_manage(["migrate", "--noinput"], {"LOGSTASHUI_DB_ENGINE": "sqlite"}) + pythonpath = captured["env"]["PYTHONPATH"] + pkg_root = str(Path(me.__file__).resolve().parent.parent) + assert pythonpath.split(me.os.pathsep)[0] == pkg_root + + +def test_reset_postgres_sequences_does_not_require_psql(monkeypatch): + captured = {} + + def fake_run(cmd, env=None, check=False, capture_output=False, text=False): + captured["cmd"] = cmd + captured["env"] = env + class Result: + returncode = 0 + stdout = "" + stderr = "" + return Result() + + monkeypatch.setattr(me.subprocess, "run", fake_run) + me._reset_postgres_sequences({"LOGSTASHUI_DB_ENGINE": "postgresql"}) + assert captured["cmd"][0] == me.sys.executable + assert captured["cmd"][1] == "-c" + code = captured["cmd"][2] + assert "dbshell" not in code + assert "psql" not in code + assert "sequence_reset_sql" in code + assert "cursor.execute" in code + pkg_root = str(Path(me.__file__).resolve().parent.parent) + assert captured["env"]["PYTHONPATH"].split(me.os.pathsep)[0] == pkg_root diff --git a/src/logstashui/LogstashUI/tests/test_migrate_live.py b/src/logstashui/LogstashUI/tests/test_migrate_live.py index f54b7be..3d3a28d 100644 --- a/src/logstashui/LogstashUI/tests/test_migrate_live.py +++ b/src/logstashui/LogstashUI/tests/test_migrate_live.py @@ -66,6 +66,7 @@ def _run_python(code: str, extra_env: dict[str, str]) -> str: env = os.environ.copy() env.update(extra_env) env.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") + me._with_package_pythonpath(env) proc = subprocess.run( [sys.executable, "-c", code], env=env, From 13f07a619e8532dfc7bcb07042f0e6a30fc88123 Mon Sep 17 00:00:00 2001 From: Aaron Mildenstein Date: Tue, 1 Sep 2026 19:30:47 -0600 Subject: [PATCH 18/62] docs: expand 0.5.2 CHANGELOG for multi-database Record engines, extras, migrator, test matrix, and operator env in the same style as the 0.5.1 notes. --- CHANGELOG.md | 39 ++++++++++++++++++++++++++++++++------- 1 file changed, 32 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c5e14af..2085c55 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,12 +1,37 @@ -## [0.5.2] - Multi-database +## [0.5.2] - Multi-database - 09/01/2026 -Package version remains **0.5.1** until release tagging; this documents the 0.5.2 database work. +Package version in `pyproject.toml` remains **0.5.1** until release tagging. This documents the 0.5.2 database work. -- `LOGSTASHUI_DB_ENGINE=sqlite|postgresql|mysql` (MariaDB uses `mysql`). Discrete `LOGSTASHUI_DB_HOST/PORT/NAME/USER/PASSWORD` plus SSL and `CONN_MAX_AGE`. No YAML, no `DATABASE_URL`. -- Default is still SQLite. `logstashui serve` warns when `LOGSTASHUI_WORKERS>1` on SQLite. -- Native extras: `LogstashUI[postgres]`, `[mysql]`, `[databases]`. Container image installs `[databases]`. -- BETA `logstashui migrate-engine --to postgresql|mysql --i-have-a-backup` copies SQLite → server (stops gunicorn; does not restart). -- `bin/test_databases.sh` runs the suite and dump/load against local Docker Postgres, MariaDB, and MySQL. +SQLite does not scale under gunicorn/gevent. Operators can now run LogstashUI on **SQLite** (default), **PostgreSQL 14+**, or **MariaDB 10.6+ / MySQL 8.0+** without changing the Django ORM data model. + +### Database engines + +- `LOGSTASHUI_DB_ENGINE=sqlite|postgresql|mysql` (MariaDB uses `mysql`). Aliases: `sqlite3`, `postgres`, `mariadb`, `my`. +- Discrete env only: `LOGSTASHUI_DB_HOST`, `PORT`, `NAME`, `USER`, `PASSWORD`, plus `LOGSTASHUI_DB_SSLMODE` / `LOGSTASHUI_DB_SSL_CA`, `LOGSTASHUI_DB_CONN_MAX_AGE` (default 60), `LOGSTASHUI_DB_CONN_HEALTH_CHECKS` (default true). No YAML. No `DATABASE_URL`. +- Unset engine is still SQLite at `$LOGSTASHUI_DATA_DIR/db.sqlite3` (WAL + `busy_timeout` unchanged). +- `logstashui serve` logs a warning when engine is SQLite and `LOGSTASHUI_WORKERS>1`; it does not refuse to start. +- Fail-fast on unknown engine, missing driver extra, missing HOST/USER, or server below version floors. +- MySQL/MariaDB use `utf8mb4` / `utf8mb4_bin` so unique names match SQLite/Postgres case-sensitivity. Create the database with that collation. +- `LOGSTASHUI_DATA_DIR` is still required when the database is remote (TLS, Django secret, logs, staticfiles). +- gunicorn stays `--worker-class gevent`. PostgreSQL uses `psycopg[binary]`; MySQL/MariaDB use PyMySQL (not mysqlclient). Optional PgBouncer is documented, not required. + +### Packaging and Docker + +- Native extras: `LogstashUI[postgres]`, `LogstashUI[mysql]`, `LogstashUI[databases]`. Default wheel stays SQLite-only. +- Container image installs `LogstashUI[databases]` so Kubernetes only sets env. +- systemd generator prompts for engine/host/port/name/user; sample `/etc/default/logstashui` documents all `LOGSTASHUI_DB_*` keys. Set the password in the EnvironmentFile or a Secret (`chmod 640`). +- gunicorn writes `$LOGSTASHUI_DATA_DIR/gunicorn.pid`. + +### Migration off SQLite + +- **Supported offline path:** stop UI → back up `db.sqlite3` → `dumpdata` while still on SQLite → create the server database → set `LOGSTASHUI_DB_*` → `migrate` + `loaddata`. Keep `DATA_DIR` (same secret key) or encrypted keystore rows will not decrypt. Sessions are not copied; log in again. +- **BETA** `logstashui migrate-engine --to postgresql|mysql --i-have-a-backup` dumps the SQLite file, loads the target, and does **not** restart serve. It SIGTERMs gunicorn if a pidfile is live. Prefer `systemctl stop` first so `Restart=` does not race. Optional `--write-env` appends engine/host/name/user (never the password). + +### Testing + +- Default `pytest` stays SQLite (no Docker, no extras). +- `bin/test_databases.sh` / `bin/test_databases.bat` start local Docker Postgres 16, MariaDB 11, and MySQL 8.0, run the suite on each engine, and run live dump/load tests. CI workflow `.github/workflows/test-databases.yml` calls the same script. +- Smoke compose is still SQLite (product CA / PUID unchanged). ## [0.5.1] - Agent control plane, SNMP NMS, dual HTTPS - 08/31/2026 From d54f9f1c362363668da93657f84df9c5f3964aec Mon Sep 17 00:00:00 2001 From: Aaron Mildenstein Date: Tue, 1 Sep 2026 20:30:36 -0600 Subject: [PATCH 19/62] fix: fail-fast DB floors before migrate; review nits Check server version before migrate and on --skip-migrate. Strip passwords, upsert --write-env, register psycopg wait_select, seed JSONField in live dump/load, drop psql from offline docs. --- CHANGELOG.md | 3 +- .../logstashui/configuration/environment.md | 2 +- src/logstashui/LogstashUI/cli.py | 25 +++++++-- src/logstashui/LogstashUI/database.py | 29 ++++++++-- src/logstashui/LogstashUI/migrate_engine.py | 30 ++++++---- src/logstashui/LogstashUI/tests/test_cli.py | 55 ++++++++++++++++++- .../LogstashUI/tests/test_database.py | 46 ++++++++++++++++ .../LogstashUI/tests/test_migrate_engine.py | 3 +- .../LogstashUI/tests/test_migrate_live.py | 16 +++++- src/logstashui/LogstashUI/wsgi.py | 4 +- 10 files changed, 182 insertions(+), 31 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2085c55..0faaa96 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,8 @@ SQLite does not scale under gunicorn/gevent. Operators can now run LogstashUI on - Discrete env only: `LOGSTASHUI_DB_HOST`, `PORT`, `NAME`, `USER`, `PASSWORD`, plus `LOGSTASHUI_DB_SSLMODE` / `LOGSTASHUI_DB_SSL_CA`, `LOGSTASHUI_DB_CONN_MAX_AGE` (default 60), `LOGSTASHUI_DB_CONN_HEALTH_CHECKS` (default true). No YAML. No `DATABASE_URL`. - Unset engine is still SQLite at `$LOGSTASHUI_DATA_DIR/db.sqlite3` (WAL + `busy_timeout` unchanged). - `logstashui serve` logs a warning when engine is SQLite and `LOGSTASHUI_WORKERS>1`; it does not refuse to start. -- Fail-fast on unknown engine, missing driver extra, missing HOST/USER, or server below version floors. +- Fail-fast on unknown engine, missing driver extra, missing HOST/USER, or server below version floors. `logstashui serve` checks the server version **before** `migrate` and still checks when `--skip-migrate` is set. +- `logstashui migrate-engine --to` accepts `postgresql`, `mysql`, and `mariadb` (`mariadb` is an alias of `mysql`). - MySQL/MariaDB use `utf8mb4` / `utf8mb4_bin` so unique names match SQLite/Postgres case-sensitivity. Create the database with that collation. - `LOGSTASHUI_DATA_DIR` is still required when the database is remote (TLS, Django secret, logs, staticfiles). - gunicorn stays `--worker-class gevent`. PostgreSQL uses `psycopg[binary]`; MySQL/MariaDB use PyMySQL (not mysqlclient). Optional PgBouncer is documented, not required. diff --git a/docs/docs/logstashui/configuration/environment.md b/docs/docs/logstashui/configuration/environment.md index f31effa..4a18745 100644 --- a/docs/docs/logstashui/configuration/environment.md +++ b/docs/docs/logstashui/configuration/environment.md @@ -94,7 +94,7 @@ No `DATABASE_URL`. No YAML. 4. Create the server database (`utf8mb4_bin` on MySQL/MariaDB). 5. **Then** set `LOGSTASHUI_DB_*` for the target. Native installs need the matching extra. 6. `logstashui manage migrate --noinput && logstashui manage loaddata dump.json` -7. Postgres: `logstashui manage sqlsequencereset PipelineManager Management SNMP AI auth admin | logstashui manage dbshell` +7. Postgres sequences: the BETA CLI (`logstashui migrate-engine`) resets them through Django (no `psql`). The dump/load path above does not; use `migrate-engine` when you need sequence reset without a Postgres client. 8. Start LogstashUI. Log in again (sessions were not copied). ### BETA CLI diff --git a/src/logstashui/LogstashUI/cli.py b/src/logstashui/LogstashUI/cli.py index 3c71716..212ca96 100644 --- a/src/logstashui/LogstashUI/cli.py +++ b/src/logstashui/LogstashUI/cli.py @@ -54,7 +54,12 @@ def build_parser() -> argparse.ArgumentParser: "migrate-engine", help="BETA: copy SQLite data to PostgreSQL or MySQL (stops gunicorn)", ) - migrate.add_argument("--to", required=True, choices=("postgresql", "mysql")) + migrate.add_argument( + "--to", + required=True, + choices=("postgresql", "mysql", "mariadb"), + help="Target engine (mariadb is an alias of mysql)", + ) migrate.add_argument( "--i-have-a-backup", dest="i_have_a_backup", @@ -364,8 +369,19 @@ def _best_effort_call(name: str, **kwargs) -> None: print(f"Warning: {name} failed: {exc}", file=sys.stderr) +def _check_db_floor() -> None: + """Connect and enforce engine version floors before migrate or gunicorn bind.""" + _django_setup() + from django.db import connection + + from LogstashUI.database import check_server_version + + connection.ensure_connection() + check_server_version(connection) + + def cmd_serve(args: argparse.Namespace) -> int: - from LogstashUI.database import canonical_engine, check_server_version + from LogstashUI.database import canonical_engine from LogstashUI.paths import resolve_data_dir os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") @@ -381,12 +397,9 @@ def cmd_serve(args: argparse.Namespace) -> int: logger.warning(msg) print(msg, file=sys.stderr) + _check_db_floor() if not args.skip_migrate: _manage(["migrate", "--noinput"]) - _django_setup() - from django.db import connection - - check_server_version(connection) _best_effort_call("sync_snmp_official_data", cleanup=True) _best_effort_call("collectstatic", interactive=False) diff --git a/src/logstashui/LogstashUI/database.py b/src/logstashui/LogstashUI/database.py index f5b3d4f..a5201ca 100644 --- a/src/logstashui/LogstashUI/database.py +++ b/src/logstashui/LogstashUI/database.py @@ -56,7 +56,10 @@ def _env_int(name: str, default: int) -> int: raw = _env(name, "") if raw == "": return default - return int(raw) + try: + return int(raw) + except ValueError as exc: + raise RuntimeError(f"{name} must be an integer, not {raw!r}.") from exc def _require(names: list[str]) -> None: @@ -104,7 +107,7 @@ def build_databases(data_dir: Path) -> dict: "ENGINE": "django.db.backends.postgresql", "NAME": _env("LOGSTASHUI_DB_NAME", "logstashui") or "logstashui", "USER": _env("LOGSTASHUI_DB_USER"), - "PASSWORD": os.environ.get("LOGSTASHUI_DB_PASSWORD") or "", + "PASSWORD": _env("LOGSTASHUI_DB_PASSWORD"), "HOST": _env("LOGSTASHUI_DB_HOST"), "PORT": _env("LOGSTASHUI_DB_PORT", "5432") or "5432", "CONN_MAX_AGE": conn_max_age, @@ -135,7 +138,7 @@ def build_databases(data_dir: Path) -> dict: "ENGINE": "django.db.backends.mysql", "NAME": _env("LOGSTASHUI_DB_NAME", "logstashui") or "logstashui", "USER": _env("LOGSTASHUI_DB_USER"), - "PASSWORD": os.environ.get("LOGSTASHUI_DB_PASSWORD") or "", + "PASSWORD": _env("LOGSTASHUI_DB_PASSWORD"), "HOST": _env("LOGSTASHUI_DB_HOST"), "PORT": _env("LOGSTASHUI_DB_PORT", "3306") or "3306", "CONN_MAX_AGE": conn_max_age, @@ -154,7 +157,7 @@ def check_server_version(connection) -> None: vendor = getattr(connection, "vendor", "") if vendor == "postgresql": pg_version = int(getattr(connection, "pg_version", 0) or 0) - if pg_version and pg_version < 140000: + if pg_version < 140000: raise RuntimeError( f"PostgreSQL 14+ is required (server_version_num={pg_version})." ) @@ -178,3 +181,21 @@ def check_server_version(connection) -> None: raise RuntimeError( f"MySQL 8.0+ is required (server={getattr(connection, 'mysql_server_info', tup)})." ) + + +def ensure_psycopg_gevent(waiting_module=None): + """Use psycopg wait_select so gunicorn gevent's patched select is cooperative. + + gunicorn --worker-class gevent monkey-patches select before loading WSGI. + psycopg 3.1.14+ also auto-detects that; assigning wait_select makes it explicit. + """ + waiting = waiting_module + if waiting is None: + try: + from psycopg import waiting as waiting + except ImportError: + return None + wait_select = getattr(waiting, "wait_select", None) + if wait_select is not None: + waiting.wait = wait_select + return waiting diff --git a/src/logstashui/LogstashUI/migrate_engine.py b/src/logstashui/LogstashUI/migrate_engine.py index 5ec6c75..1da46aa 100644 --- a/src/logstashui/LogstashUI/migrate_engine.py +++ b/src/logstashui/LogstashUI/migrate_engine.py @@ -85,18 +85,26 @@ def stop_gunicorn(pidfile: Path) -> None: def write_env_file(path: Path, engine: str) -> None: - pairs = [ - ("LOGSTASHUI_DB_ENGINE", engine), - ("LOGSTASHUI_DB_NAME", os.environ.get("LOGSTASHUI_DB_NAME")), - ("LOGSTASHUI_DB_HOST", os.environ.get("LOGSTASHUI_DB_HOST")), - ("LOGSTASHUI_DB_PORT", os.environ.get("LOGSTASHUI_DB_PORT")), - ("LOGSTASHUI_DB_USER", os.environ.get("LOGSTASHUI_DB_USER")), - ] - lines = [f"{key}={value}" for key, value in pairs if value] + """Upsert LOGSTASHUI_DB_* keys (never password). Second run does not duplicate.""" + assignments = { + "LOGSTASHUI_DB_ENGINE": engine, + "LOGSTASHUI_DB_NAME": os.environ.get("LOGSTASHUI_DB_NAME") or "", + "LOGSTASHUI_DB_HOST": os.environ.get("LOGSTASHUI_DB_HOST") or "", + "LOGSTASHUI_DB_PORT": os.environ.get("LOGSTASHUI_DB_PORT") or "", + "LOGSTASHUI_DB_USER": os.environ.get("LOGSTASHUI_DB_USER") or "", + } + assignments = {key: value for key, value in assignments.items() if value} existing = path.read_text(encoding="utf-8") if path.is_file() else "" - prefix = "" if not existing or existing.endswith("\n") else "\n" - with path.open("a", encoding="utf-8") as fh: - fh.write(prefix + "\n".join(lines) + "\n") + kept = [] + for line in existing.splitlines(): + key = line.split("=", 1)[0] if "=" in line else "" + if key in assignments: + continue + kept.append(line) + while kept and kept[-1] == "": + kept.pop() + kept.extend(f"{key}={value}" for key, value in assignments.items()) + path.write_text("\n".join(kept) + "\n", encoding="utf-8") def _reset_postgres_sequences(extra_env: dict[str, str]) -> None: diff --git a/src/logstashui/LogstashUI/tests/test_cli.py b/src/logstashui/LogstashUI/tests/test_cli.py index d49f57b..b7c24c3 100644 --- a/src/logstashui/LogstashUI/tests/test_cli.py +++ b/src/logstashui/LogstashUI/tests/test_cli.py @@ -97,9 +97,7 @@ def test_serve_snmp_commanderror_does_not_abort(monkeypatch): from LogstashUI import cli monkeypatch.setattr(cli, "_manage", lambda argv: None) - monkeypatch.setattr( - "LogstashUI.database.check_server_version", lambda conn: None - ) + monkeypatch.setattr(cli, "_check_db_floor", lambda: None) monkeypatch.setenv("LOGSTASHUI_TLS", "false") def fake_call(name, *args, **kwargs): @@ -133,10 +131,61 @@ def test_parser_migrate_engine_requires_backup_flag(): assert ns.i_have_a_backup is False +def test_parser_migrate_engine_accepts_mariadb_alias(): + parser = build_parser() + ns = parser.parse_args(["migrate-engine", "--to", "mariadb", "--i-have-a-backup"]) + assert ns.to == "mariadb" + assert ns.i_have_a_backup is True + + +def test_serve_checks_version_before_migrate(monkeypatch): + from LogstashUI import cli + + order = [] + monkeypatch.setattr(cli, "_check_db_floor", lambda: order.append("check")) + monkeypatch.setattr(cli, "_manage", lambda argv: order.append(argv[0])) + monkeypatch.setattr(cli, "_best_effort_call", lambda *a, **k: None) + monkeypatch.setenv("LOGSTASHUI_TLS", "false") + + def fake_execvp(file, args): + raise SystemExit(0) + + monkeypatch.setattr(cli.os, "execvp", fake_execvp) + ns = Namespace(skip_migrate=False, no_tls=True, bind="127.0.0.1:8443", workers=1) + try: + cmd_serve(ns) + except SystemExit: + pass + assert order[0] == "check" + assert "migrate" in order + + +def test_serve_checks_version_when_skip_migrate(monkeypatch, tmp_path): + from LogstashUI import cli + + called = [] + monkeypatch.setattr(cli, "_check_db_floor", lambda: called.append(True)) + monkeypatch.setattr(cli, "_manage", lambda argv: None) + monkeypatch.setenv("LOGSTASHUI_TLS", "false") + monkeypatch.setenv("LOGSTASHUI_DATA_DIR", str(tmp_path)) + + def fake_execvp(file, args): + raise SystemExit(0) + + monkeypatch.setattr(cli.os, "execvp", fake_execvp) + ns = Namespace(skip_migrate=True, no_tls=True, bind="127.0.0.1:8443", workers=1) + try: + cmd_serve(ns) + except SystemExit: + pass + assert called == [True] + + def test_serve_adds_pidfile_and_warns_sqlite(monkeypatch, tmp_path, capsys): from LogstashUI import cli monkeypatch.setattr(cli, "_manage", lambda argv: None) + monkeypatch.setattr(cli, "_check_db_floor", lambda: None) monkeypatch.setenv("LOGSTASHUI_TLS", "false") monkeypatch.setenv("LOGSTASHUI_DATA_DIR", str(tmp_path)) monkeypatch.delenv("LOGSTASHUI_DB_ENGINE", raising=False) diff --git a/src/logstashui/LogstashUI/tests/test_database.py b/src/logstashui/LogstashUI/tests/test_database.py index 9a9cc89..5afb0de 100644 --- a/src/logstashui/LogstashUI/tests/test_database.py +++ b/src/logstashui/LogstashUI/tests/test_database.py @@ -165,6 +165,52 @@ class Conn: check_server_version(Conn()) +def test_check_server_version_postgres_zero_is_too_old(): + class Conn: + vendor = "postgresql" + pg_version = 0 + + with pytest.raises(RuntimeError, match="PostgreSQL 14"): + check_server_version(Conn()) + + +def test_conn_max_age_invalid_raises_runtimeerror(tmp_path, monkeypatch): + _clear_db_env(monkeypatch) + monkeypatch.setenv("LOGSTASHUI_DB_CONN_MAX_AGE", "nope") + with pytest.raises(RuntimeError, match="LOGSTASHUI_DB_CONN_MAX_AGE"): + build_databases(tmp_path) + + +def test_password_is_stripped(tmp_path, monkeypatch): + _clear_db_env(monkeypatch) + monkeypatch.setenv("LOGSTASHUI_DB_ENGINE", "postgresql") + monkeypatch.setenv("LOGSTASHUI_DB_HOST", "db.example") + monkeypatch.setenv("LOGSTASHUI_DB_USER", "lsui") + monkeypatch.setenv("LOGSTASHUI_DB_PASSWORD", " secret\n") + monkeypatch.setattr("LogstashUI.database._import_or_raise", lambda *a, **k: None) + db = build_databases(tmp_path)["default"] + assert db["PASSWORD"] == "secret" + + +def test_ensure_psycopg_gevent_assigns_wait_select(): + from types import SimpleNamespace + + from LogstashUI.database import ensure_psycopg_gevent + + def wait_select(*args, **kwargs): + return "select" + + waiting = SimpleNamespace(wait_select=wait_select, wait=None) + ensure_psycopg_gevent(waiting) + assert waiting.wait is wait_select + + +def test_ensure_psycopg_gevent_does_not_raise(): + from LogstashUI.database import ensure_psycopg_gevent + + ensure_psycopg_gevent() + + def test_check_server_version_mysql_and_mariadb(): class Mysql: vendor = "mysql" diff --git a/src/logstashui/LogstashUI/tests/test_migrate_engine.py b/src/logstashui/LogstashUI/tests/test_migrate_engine.py index 0447da5..c0008f7 100644 --- a/src/logstashui/LogstashUI/tests/test_migrate_engine.py +++ b/src/logstashui/LogstashUI/tests/test_migrate_engine.py @@ -63,8 +63,9 @@ def test_write_env_appends(tmp_path, monkeypatch): monkeypatch.setenv("LOGSTASHUI_DB_USER", "lsui") monkeypatch.setenv("LOGSTASHUI_DB_NAME", "logstashui") me.write_env_file(envf, "postgresql") + me.write_env_file(envf, "postgresql") text = envf.read_text() - assert "LOGSTASHUI_DB_ENGINE=postgresql" in text + assert text.count("LOGSTASHUI_DB_ENGINE=postgresql") == 1 assert "LOGSTASHUI_DB_HOST=db.example" in text assert "PASSWORD" not in text diff --git a/src/logstashui/LogstashUI/tests/test_migrate_live.py b/src/logstashui/LogstashUI/tests/test_migrate_live.py index 3d3a28d..ed4ee62 100644 --- a/src/logstashui/LogstashUI/tests/test_migrate_live.py +++ b/src/logstashui/LogstashUI/tests/test_migrate_live.py @@ -23,15 +23,22 @@ os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") django.setup() from django.contrib.auth import get_user_model -from PipelineManager.models import Policy +from PipelineManager.models import Connection, Policy User = get_user_model() User.objects.create_user(username="migrate-user", password="migrate-pass") -Policy.objects.create( +policy = Policy.objects.create( name="Migrate Policy", logstash_yml="http.host: 0.0.0.0", jvm_options="-Xms1g", log4j2_properties="status = error", ) +Connection.objects.create( + name="Migrate Conn", + connection_type=Connection.ConnectionType.AGENT, + host="127.0.0.1", + policy=policy, + status_blob={"health": "green", "n": 1}, +) """ _COUNT = """ @@ -41,13 +48,15 @@ os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") django.setup() from django.contrib.auth import get_user_model -from PipelineManager.models import Policy +from PipelineManager.models import Connection, Policy User = get_user_model() +conn = Connection.objects.filter(name="Migrate Conn").first() print(json.dumps({ "users": User.objects.count(), "policies": Policy.objects.count(), "migrate_user": User.objects.filter(username="migrate-user").count(), "migrate_policy": Policy.objects.filter(name="Migrate Policy").count(), + "status_blob": conn.status_blob if conn else None, })) """ @@ -126,6 +135,7 @@ def _run_to(tmp_path, engine: str, *, port: str, user: str) -> None: assert counts["policies"] >= 1 assert counts["migrate_user"] == 1 assert counts["migrate_policy"] == 1 + assert counts["status_blob"] == {"health": "green", "n": 1} def test_live_postgres(tmp_path): diff --git a/src/logstashui/LogstashUI/wsgi.py b/src/logstashui/LogstashUI/wsgi.py index 3a9e067..743ed77 100644 --- a/src/logstashui/LogstashUI/wsgi.py +++ b/src/logstashui/LogstashUI/wsgi.py @@ -19,9 +19,11 @@ os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'LogstashUI.settings') +from LogstashUI.database import ensure_psycopg_gevent + # gunicorn --worker-class gevent monkey-patches select before this module loads. -# psycopg 3.1.14+ detects that and waits cooperatively; do not use psycogreen. application = get_wsgi_application() +ensure_psycopg_gevent() # Quiet gevent/gunicorn spam: clients that reject the product CA (browser From 52db963d0a9849ff03de00ff21586639cc9968f2 Mon Sep 17 00:00:00 2001 From: Aaron Mildenstein Date: Tue, 1 Sep 2026 20:38:21 -0600 Subject: [PATCH 20/62] docs: note BETA migrate-engine is not atomic on the target If loaddata fails the target may be partial; drop/recreate and retry. A production migrator could wrap loaddata in transaction.atomic(). --- CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0faaa96..3353068 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -27,6 +27,7 @@ SQLite does not scale under gunicorn/gevent. Operators can now run LogstashUI on - **Supported offline path:** stop UI → back up `db.sqlite3` → `dumpdata` while still on SQLite → create the server database → set `LOGSTASHUI_DB_*` → `migrate` + `loaddata`. Keep `DATA_DIR` (same secret key) or encrypted keystore rows will not decrypt. Sessions are not copied; log in again. - **BETA** `logstashui migrate-engine --to postgresql|mysql --i-have-a-backup` dumps the SQLite file, loads the target, and does **not** restart serve. It SIGTERMs gunicorn if a pidfile is live. Prefer `systemctl stop` first so `Restart=` does not race. Optional `--write-env` appends engine/host/name/user (never the password). +- BETA `migrate-engine` is **not atomic** on the target: `dumpdata` → `migrate` → `loaddata` is three steps. If `loaddata` fails, the target may be partially populated. Drop or recreate the target database (SQLite is only WAL-checkpointed) and re-run. A later production migrator could wrap `loaddata` and Postgres `sequence_reset_sql` in `transaction.atomic()` after `migrate`; `migrate` itself applies DDL and cannot be one atomic unit on MySQL/MariaDB. ### Testing From fcacf59efa27706206436c212ed4147daa497d0c Mon Sep 17 00:00:00 2001 From: Aaron Mildenstein Date: Wed, 2 Sep 2026 10:25:52 -0600 Subject: [PATCH 21/62] Add examples & fix ALLOWED_HOSTS in k8s After testing the generated docker image it was discovered that it was not adding the pod's own IP to the ALLOWED_HOSTS. This made it so that aliveness checks were getting a 400 error. That has been addressed here. A few more edits to .gitignore for a scratch directory and relocating other lines All other changes are documentation. --- .gitignore | 3 +- CHANGELOG.md | 7 + docs/docs/index.md | 11 + .../logstashui/configuration/environment.md | 12 +- docs/docs/logstashui/configuration/index.md | 2 + .../logstashui/database/examples/README.md | 15 + .../database/examples/create-mariadb.sql | 13 + .../database/examples/create-mysql.sql | 12 + .../database/examples/create-postgresql.sql | 13 + .../database/examples/schema-mysql.sql | 502 +++++ .../database/examples/schema-postgresql.sql | 1709 +++++++++++++++++ docs/docs/logstashui/database/index.md | 79 + docs/docs/logstashui/database/migration.md | 75 + docs/docs/logstashui/general/deploy.md | 19 +- docs/docs/logstashui/general/index.md | 2 + docs/docs/logstashui/index.md | 2 + docs/docs/logstashui/kubernetes/cnpg.md | 98 + .../logstashui/kubernetes/envoy-gateway.md | 112 ++ .../logstashui/kubernetes/examples/README.md | 17 + .../kubernetes/examples/mysql/configmap.yaml | 24 + .../kubernetes/examples/mysql/ingress.yaml | 27 + .../kubernetes/examples/mysql/namespace.yaml | 4 + .../kubernetes/examples/mysql/secret.yaml | 9 + .../examples/mysql/statefulset.yaml | 98 + .../kubernetes/examples/postgresql/cnpg.yaml | 21 + .../examples/postgresql/configmap.yaml | 25 + .../examples/postgresql/ingress.yaml | 27 + .../examples/postgresql/namespace.yaml | 4 + .../examples/postgresql/secret.yaml | 9 + .../examples/postgresql/statefulset.yaml | 98 + .../kubernetes/examples/sqlite/configmap.yaml | 18 + .../kubernetes/examples/sqlite/ingress.yaml | 27 + .../kubernetes/examples/sqlite/namespace.yaml | 4 + .../kubernetes/examples/sqlite/secret.yaml | 8 + .../examples/sqlite/statefulset.yaml | 98 + docs/docs/logstashui/kubernetes/index.md | 113 ++ src/logstashui/LogstashUI/config.py | 32 + src/logstashui/LogstashUI/settings.py | 4 +- .../LogstashUI/tests/test_config.py | 22 +- 39 files changed, 3363 insertions(+), 12 deletions(-) create mode 100644 docs/docs/logstashui/database/examples/README.md create mode 100644 docs/docs/logstashui/database/examples/create-mariadb.sql create mode 100644 docs/docs/logstashui/database/examples/create-mysql.sql create mode 100644 docs/docs/logstashui/database/examples/create-postgresql.sql create mode 100644 docs/docs/logstashui/database/examples/schema-mysql.sql create mode 100644 docs/docs/logstashui/database/examples/schema-postgresql.sql create mode 100644 docs/docs/logstashui/database/index.md create mode 100644 docs/docs/logstashui/database/migration.md create mode 100644 docs/docs/logstashui/kubernetes/cnpg.md create mode 100644 docs/docs/logstashui/kubernetes/envoy-gateway.md create mode 100644 docs/docs/logstashui/kubernetes/examples/README.md create mode 100644 docs/docs/logstashui/kubernetes/examples/mysql/configmap.yaml create mode 100644 docs/docs/logstashui/kubernetes/examples/mysql/ingress.yaml create mode 100644 docs/docs/logstashui/kubernetes/examples/mysql/namespace.yaml create mode 100644 docs/docs/logstashui/kubernetes/examples/mysql/secret.yaml create mode 100644 docs/docs/logstashui/kubernetes/examples/mysql/statefulset.yaml create mode 100644 docs/docs/logstashui/kubernetes/examples/postgresql/cnpg.yaml create mode 100644 docs/docs/logstashui/kubernetes/examples/postgresql/configmap.yaml create mode 100644 docs/docs/logstashui/kubernetes/examples/postgresql/ingress.yaml create mode 100644 docs/docs/logstashui/kubernetes/examples/postgresql/namespace.yaml create mode 100644 docs/docs/logstashui/kubernetes/examples/postgresql/secret.yaml create mode 100644 docs/docs/logstashui/kubernetes/examples/postgresql/statefulset.yaml create mode 100644 docs/docs/logstashui/kubernetes/examples/sqlite/configmap.yaml create mode 100644 docs/docs/logstashui/kubernetes/examples/sqlite/ingress.yaml create mode 100644 docs/docs/logstashui/kubernetes/examples/sqlite/namespace.yaml create mode 100644 docs/docs/logstashui/kubernetes/examples/sqlite/secret.yaml create mode 100644 docs/docs/logstashui/kubernetes/examples/sqlite/statefulset.yaml create mode 100644 docs/docs/logstashui/kubernetes/index.md diff --git a/.gitignore b/.gitignore index 9e0234d..4ec32aa 100644 --- a/.gitignore +++ b/.gitignore @@ -223,7 +223,6 @@ __marimo__/ # LogstashUI /src/logstashui/staticfiles/ /src/logstashui/data/ -/.plans/ # Local config (use logstashui.example.yml) /src/logstashui/logstashui.yml @@ -233,6 +232,7 @@ __marimo__/ /docker/logstashui_data/ # Agentic / local tooling scratch (not product docs) +/.plans/ .grokignore /docs/superpowers/ .hermes.md @@ -243,6 +243,7 @@ CLAUDE.md IDEA.md graphify-out/ .github/copilot-instructions.md +/scratch/ # LogstashAgent (cloned from GitHub in host mode) /LogstashAgent/ diff --git a/CHANGELOG.md b/CHANGELOG.md index 3353068..9881c1b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -35,6 +35,13 @@ SQLite does not scale under gunicorn/gevent. Operators can now run LogstashUI on - `bin/test_databases.sh` / `bin/test_databases.bat` start local Docker Postgres 16, MariaDB 11, and MySQL 8.0, run the suite on each engine, and run live dump/load tests. CI workflow `.github/workflows/test-databases.yml` calls the same script. - Smoke compose is still SQLite (product CA / PUID unchanged). +### Kubernetes and database docs + +- Kubernetes subsection: StatefulSet + one PVC at `/var/lib/logstashui`, TLS kept on `:8443`, Ingress-nginx skip backend verify, Envoy Gateway `Backend` `insecureSkipVerify` (enable Backend API), CloudNativePG Cluster in the app namespace. +- K8s probes send `Host: logstashui` (kubelet otherwise uses the pod IP and Django returns 400). Downward API `status.podIP` → `LOGSTASHUI_HOST_IPS` for the product leaf; those IPs are appended to `ALLOWED_HOSTS` unless the list is `*`. +- Example manifests under `docs/docs/logstashui/kubernetes/examples/{sqlite,postgresql,mysql}/`. +- Database subsection: engines, every `LOGSTASHUI_DB_*` default, offline dump/load, BETA `migrate-engine`, CREATE DATABASE scripts (`utf8mb4_bin` for MySQL/MariaDB), schema snapshots from 0.5.2 `migrate`. + ## [0.5.1] - Agent control plane, SNMP NMS, dual HTTPS - 08/31/2026 diff --git a/docs/docs/index.md b/docs/docs/index.md index 3bc46a6..50991e8 100644 --- a/docs/docs/index.md +++ b/docs/docs/index.md @@ -63,6 +63,17 @@ New to LogstashUI? Start here: - **[Simulation Modes](/docs/docs/logstashui/configuration/simulation.md)** - Embedded vs Host mode - **[Host Mode Setup](/docs/docs/logstashui/configuration/host_mode.md)** - High-performance simulation setup +#### Database +- **[Database](/docs/docs/logstashui/database/index.md)** - SQLite default, PostgreSQL, MySQL/MariaDB + - **[Migration](/docs/docs/logstashui/database/migration.md)** - Offline dump/load and BETA `migrate-engine` + - **[SQL examples](/docs/docs/logstashui/database/examples/README.md)** - CREATE DATABASE and schema snapshots + +#### Kubernetes +- **[Kubernetes](/docs/docs/logstashui/kubernetes/index.md)** - StatefulSet, PVC, TLS on :8443 + - **[Envoy Gateway](/docs/docs/logstashui/kubernetes/envoy-gateway.md)** - Enable Backend API, skip backend cert verify + - **[CloudNativePG](/docs/docs/logstashui/kubernetes/cnpg.md)** - Cluster in the app namespace + - **[Example manifests](/docs/docs/logstashui/kubernetes/examples/README.md)** - SQLite, PostgreSQL, MySQL/MariaDB + #### SNMP Monitoring - **[SNMP Overview](/docs/docs/logstashui/SNMP/index.md)** - Network monitoring introduction - **[Quickstart Guide](/docs/docs/logstashui/SNMP/Quickstart.md)** - From zero to metrics diff --git a/docs/docs/logstashui/configuration/environment.md b/docs/docs/logstashui/configuration/environment.md index 4a18745..7454560 100644 --- a/docs/docs/logstashui/configuration/environment.md +++ b/docs/docs/logstashui/configuration/environment.md @@ -39,7 +39,7 @@ Relative values resolve from the process working directory. | `CSRF_TRUSTED_ORIGINS` | (dev localhost defaults) | comma-separated origins | | `SECRET_KEY` | auto in data dir | Django secret | -Set `LOGSTASHUI_TLS=false` when an ingress terminates TLS and the pod should speak HTTP. +Keep `LOGSTASHUI_TLS=true` (the default) in Kubernetes. Ingress/HTTPRoute should originate HTTPS to `:8443` and skip backend cert verify. See [Kubernetes](/docs/docs/logstashui/kubernetes/index.md). --- @@ -51,7 +51,7 @@ Set `LOGSTASHUI_TLS=false` when an ingress terminates TLS and the pod should spe | `LOGSTASHUI_AGENT_UI_URL` | empty | Prefill `--logstash-ui-url` (DB Settings wins if set) | | `LOGSTASHUI_INCLUDE_CA_FINGERPRINT` | `true` | Embed product CA fingerprint in enrollment tokens | | `LOGSTASH_AGENT_URL` | debug: `http://127.0.0.1:9500`; else `https://logstashagent:9500` | Embedded/compose agent API | -| `LOGSTASHUI_HOST_HOSTNAME` / `LOGSTASHUI_HOST_IPS` / `LOGSTASHUI_TLS_SANS` | empty | Extra SANs on the product UI cert | +| `LOGSTASHUI_HOST_HOSTNAME` / `LOGSTASHUI_HOST_IPS` / `LOGSTASHUI_TLS_SANS` | empty | Extra SANs on the product UI cert. Kubernetes: set `LOGSTASHUI_HOST_IPS` from `status.podIP` (Downward API). IPs are also appended to `ALLOWED_HOSTS` unless that list is `*`. | | `LOGSTASHUI_AGENT_CSR_SECRET` | empty | Compose/embedded agent CSR without enroll | | `LOGSTASHUI_DOCS_DIR` | checkout `docs/` or packaged copy | In-app documentation root | @@ -74,7 +74,7 @@ Booleans accept `true`/`false`, `1`/`0`, `yes`/`no`, `on`/`off`. | `LOGSTASHUI_DB_CONN_MAX_AGE` | `60` | Persistent connections (seconds); `0` closes per request | | `LOGSTASHUI_DB_CONN_HEALTH_CHECKS` | `true` | Django `CONN_HEALTH_CHECKS` | -Floors: PostgreSQL 14+, MariaDB 10.6+, MySQL 8.0+. Create MySQL/MariaDB as `utf8mb4` / `utf8mb4_bin` so unique names match SQLite/Postgres case-sensitivity. +Floors: PostgreSQL 14+, MariaDB 10.6+, MySQL 8.0+. Create MySQL/MariaDB as `utf8mb4` / `utf8mb4_bin` so unique names match SQLite/Postgres case-sensitivity. Full engine docs, env defaults, and SQL examples: [Database](/docs/docs/logstashui/database/index.md). Migration (offline + BETA CLI): [Migration](/docs/docs/logstashui/database/migration.md). **Install extras (native pip/uv):** `uv pip install 'LogstashUI[postgres]'`, `'LogstashUI[mysql]'`, or `'LogstashUI[databases]'`. The Docker/K8s image already installs `[databases]`. Missing driver fails at startup with that extra name. @@ -115,12 +115,12 @@ sudo systemctl start logstashui Minimum: -1. Deployment env from a ConfigMap (`LOGSTASHUI_DB_ENGINE` / `HOST` / `NAME` / `USER`) + Secret (`SECRET_KEY`, `LOGSTASHUI_DB_PASSWORD`) +1. StatefulSet `replicas: 1`, env from a ConfigMap (`LOGSTASHUI_DB_ENGINE` / `HOST` / `NAME` / `USER`) + Secret (`SECRET_KEY`, `LOGSTASHUI_DB_PASSWORD`) 2. PVC mounted at `/var/lib/logstashui` (still required when the database is external) 3. Container image `CMD` is `logstashui serve` (already the Docker default) -4. Optional ingress: `LOGSTASHUI_TLS=false` and `CSRF_TRUSTED_ORIGINS=https://` +4. Ingress or HTTPRoute: keep `LOGSTASHUI_TLS=true`, originate HTTPS to `:8443`, skip backend cert verify, set `CSRF_TRUSTED_ORIGINS=https://` -No ConfigMap file mount is required. +No ConfigMap file mount is required. Manifests and CloudNativePG: [Kubernetes](/docs/docs/logstashui/kubernetes/index.md). --- diff --git a/docs/docs/logstashui/configuration/index.md b/docs/docs/logstashui/configuration/index.md index 24720c5..db8931e 100644 --- a/docs/docs/logstashui/configuration/index.md +++ b/docs/docs/logstashui/configuration/index.md @@ -12,6 +12,8 @@ Data dir, TLS, bind address, `LOGSTASHUI_NO_AUTH`, agent URL, systemd, and Kuber **📖 [View environment configuration →](/docs/docs/logstashui/configuration/environment.md)** +Database engines and `LOGSTASHUI_DB_*`: **[Database](/docs/docs/logstashui/database/index.md)**. Kubernetes manifests: **[Kubernetes](/docs/docs/logstashui/kubernetes/index.md)**. + `logstashui.yml` is [removed](/docs/docs/logstashui/configuration/logstashui.yml.md). --- diff --git a/docs/docs/logstashui/database/examples/README.md b/docs/docs/logstashui/database/examples/README.md new file mode 100644 index 0000000..9211a8a --- /dev/null +++ b/docs/docs/logstashui/database/examples/README.md @@ -0,0 +1,15 @@ +# Database examples + +Apply a **create-*.sql** script to an empty server, then start LogstashUI (`migrate` creates tables). + +| File | Use | +|---|---| +| [create-postgresql.sql](create-postgresql.sql) | Role + database (UTF8) | +| [create-mysql.sql](create-mysql.sql) | MySQL 8.0+ `utf8mb4` / `utf8mb4_bin` | +| [create-mariadb.sql](create-mariadb.sql) | MariaDB 10.6+ same collation | +| [schema-postgresql.sql](schema-postgresql.sql) | Snapshot of `migrate` DDL (reference) | +| [schema-mysql.sql](schema-mysql.sql) | Snapshot of `migrate` DDL (reference) | + +`schema-*.sql` is generated from LogstashUI **0.5.2** migrations. It **will go stale**. Do not apply it instead of `logstashui manage migrate --noinput`. It exists so you can see tables, indexes, and the MySQL collation before you run migrate. + +CloudNativePG: bootstrap `database` / `owner` already creates the empty database — skip `create-postgresql.sql`, still run migrate. diff --git a/docs/docs/logstashui/database/examples/create-mariadb.sql b/docs/docs/logstashui/database/examples/create-mariadb.sql new file mode 100644 index 0000000..2685eff --- /dev/null +++ b/docs/docs/logstashui/database/examples/create-mariadb.sql @@ -0,0 +1,13 @@ +-- Empty MariaDB 10.6+ database for LogstashUI. +-- Engine is still LOGSTASHUI_DB_ENGINE=mysql. +-- utf8mb4_bin keeps unique names case-sensitive like SQLite/Postgres. +-- Tables are created by: logstashui manage migrate --noinput +-- Replace the password before running. + +CREATE DATABASE IF NOT EXISTS logstashui + CHARACTER SET utf8mb4 + COLLATE utf8mb4_bin; + +CREATE USER IF NOT EXISTS 'logstashui'@'%' IDENTIFIED BY 'change-me'; +GRANT ALL PRIVILEGES ON logstashui.* TO 'logstashui'@'%'; +FLUSH PRIVILEGES; diff --git a/docs/docs/logstashui/database/examples/create-mysql.sql b/docs/docs/logstashui/database/examples/create-mysql.sql new file mode 100644 index 0000000..fff304a --- /dev/null +++ b/docs/docs/logstashui/database/examples/create-mysql.sql @@ -0,0 +1,12 @@ +-- Empty MySQL 8.0+ database for LogstashUI. +-- utf8mb4_bin keeps unique names case-sensitive like SQLite/Postgres. +-- Tables are created by: logstashui manage migrate --noinput +-- Replace the password before running. + +CREATE DATABASE IF NOT EXISTS logstashui + CHARACTER SET utf8mb4 + COLLATE utf8mb4_bin; + +CREATE USER IF NOT EXISTS 'logstashui'@'%' IDENTIFIED BY 'change-me'; +GRANT ALL PRIVILEGES ON logstashui.* TO 'logstashui'@'%'; +FLUSH PRIVILEGES; diff --git a/docs/docs/logstashui/database/examples/create-postgresql.sql b/docs/docs/logstashui/database/examples/create-postgresql.sql new file mode 100644 index 0000000..b95f4e5 --- /dev/null +++ b/docs/docs/logstashui/database/examples/create-postgresql.sql @@ -0,0 +1,13 @@ +-- Empty PostgreSQL 14+ database for LogstashUI. +-- Tables are created by: logstashui manage migrate --noinput +-- Replace the password before running. + +CREATE USER logstashui WITH LOGIN PASSWORD 'change-me'; +CREATE DATABASE logstashui + OWNER logstashui + ENCODING 'UTF8' + TEMPLATE template0; + +\connect logstashui +GRANT ALL ON SCHEMA public TO logstashui; +ALTER DATABASE logstashui OWNER TO logstashui; diff --git a/docs/docs/logstashui/database/examples/schema-mysql.sql b/docs/docs/logstashui/database/examples/schema-mysql.sql new file mode 100644 index 0000000..833ea67 --- /dev/null +++ b/docs/docs/logstashui/database/examples/schema-mysql.sql @@ -0,0 +1,502 @@ +-- Snapshot of LogstashUI 0.5.2 `migrate` DDL on MySQL 8.0 (utf8mb4_bin). +-- Generated with mysqldump --no-data. Do NOT apply this instead of +-- `logstashui manage migrate --noinput`. It will go stale with new migrations. +-- Create the empty database first (see create-mysql.sql / create-mariadb.sql). + + +/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */; +/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */; +/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */; +/*!50503 SET NAMES utf8mb4 */; +/*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */; +/*!40103 SET TIME_ZONE='+00:00' */; +/*!40014 SET @OLD_UNIQUE_CHECKS=@@UNIQUE_CHECKS, UNIQUE_CHECKS=0 */; +/*!40014 SET @OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0 */; +/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='NO_AUTO_VALUE_ON_ZERO' */; +/*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */; +DROP TABLE IF EXISTS `Management_userprofile`; +/*!40101 SET @saved_cs_client = @@character_set_client */; +/*!50503 SET character_set_client = utf8mb4 */; +CREATE TABLE `Management_userprofile` ( + `id` bigint NOT NULL AUTO_INCREMENT, + `role` varchar(20) COLLATE utf8mb4_bin NOT NULL, + `user_id` int NOT NULL, + PRIMARY KEY (`id`), + UNIQUE KEY `user_id` (`user_id`), + CONSTRAINT `Management_userprofile_user_id_70f1a900_fk_auth_user_id` FOREIGN KEY (`user_id`) REFERENCES `auth_user` (`id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +/*!40101 SET character_set_client = @saved_cs_client */; +DROP TABLE IF EXISTS `PipelineManager_apikey`; +/*!40101 SET @saved_cs_client = @@character_set_client */; +/*!50503 SET character_set_client = utf8mb4 */; +CREATE TABLE `PipelineManager_apikey` ( + `id` bigint NOT NULL AUTO_INCREMENT, + `api_key` varchar(512) COLLATE utf8mb4_bin NOT NULL, + `connection_id` bigint NOT NULL, + PRIMARY KEY (`id`), + KEY `PipelineManager_apik_connection_id_27156847_fk_PipelineM` (`connection_id`), + CONSTRAINT `PipelineManager_apik_connection_id_27156847_fk_PipelineM` FOREIGN KEY (`connection_id`) REFERENCES `PipelineManager_connection` (`id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +/*!40101 SET character_set_client = @saved_cs_client */; +DROP TABLE IF EXISTS `PipelineManager_connection`; +/*!40101 SET @saved_cs_client = @@character_set_client */; +/*!50503 SET character_set_client = utf8mb4 */; +CREATE TABLE `PipelineManager_connection` ( + `id` bigint NOT NULL AUTO_INCREMENT, + `name` varchar(100) COLLATE utf8mb4_bin NOT NULL, + `connection_type` varchar(20) COLLATE utf8mb4_bin NOT NULL, + `host` varchar(255) COLLATE utf8mb4_bin DEFAULT NULL, + `port` int unsigned DEFAULT NULL, + `username` varchar(100) COLLATE utf8mb4_bin DEFAULT NULL, + `password` varchar(512) COLLATE utf8mb4_bin DEFAULT NULL, + `ssh_key` longtext COLLATE utf8mb4_bin, + `cloud_id` varchar(255) COLLATE utf8mb4_bin DEFAULT NULL, + `cloud_url` varchar(200) COLLATE utf8mb4_bin DEFAULT NULL, + `api_key` varchar(512) COLLATE utf8mb4_bin DEFAULT NULL, + `created_at` datetime(6) NOT NULL, + `updated_at` datetime(6) NOT NULL, + `is_active` tinyint(1) NOT NULL, + `policy_id` bigint DEFAULT NULL, + `agent_id` varchar(255) COLLATE utf8mb4_bin DEFAULT NULL, + `last_check_in` datetime(6) DEFAULT NULL, + `status_blob` json DEFAULT NULL, + `restart_on_next_checkin` tinyint(1) NOT NULL, + `desired_agent_version` varchar(20) COLLATE utf8mb4_bin DEFAULT NULL, + `agent_api_port` int unsigned DEFAULT NULL, + `instance_id` int unsigned DEFAULT NULL, + `last_selected_at` datetime(6) DEFAULT NULL, + `logstash_api_port` int unsigned DEFAULT NULL, + `logstash_version_resolved` varchar(64) COLLATE utf8mb4_bin NOT NULL, + PRIMARY KEY (`id`), + UNIQUE KEY `agent_id` (`agent_id`), + KEY `PipelineManager_conn_policy_id_819d316d_fk_PipelineM` (`policy_id`), + CONSTRAINT `PipelineManager_conn_policy_id_819d316d_fk_PipelineM` FOREIGN KEY (`policy_id`) REFERENCES `PipelineManager_policy` (`id`), + CONSTRAINT `PipelineManager_connection_chk_1` CHECK ((`port` >= 0)), + CONSTRAINT `PipelineManager_connection_chk_2` CHECK ((`agent_api_port` >= 0)), + CONSTRAINT `PipelineManager_connection_chk_3` CHECK ((`instance_id` >= 0)), + CONSTRAINT `PipelineManager_connection_chk_4` CHECK ((`logstash_api_port` >= 0)) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +/*!40101 SET character_set_client = @saved_cs_client */; +DROP TABLE IF EXISTS `PipelineManager_enrollmenttoken`; +/*!40101 SET @saved_cs_client = @@character_set_client */; +/*!50503 SET character_set_client = utf8mb4 */; +CREATE TABLE `PipelineManager_enrollmenttoken` ( + `id` bigint NOT NULL AUTO_INCREMENT, + `token` varchar(512) COLLATE utf8mb4_bin NOT NULL, + `policy_id` bigint NOT NULL, + `name` varchar(100) COLLATE utf8mb4_bin NOT NULL, + PRIMARY KEY (`id`), + UNIQUE KEY `unique_token_name_per_policy` (`policy_id`,`name`), + CONSTRAINT `PipelineManager_enro_policy_id_355d6dca_fk_PipelineM` FOREIGN KEY (`policy_id`) REFERENCES `PipelineManager_policy` (`id`) +) ENGINE=InnoDB AUTO_INCREMENT=4 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +/*!40101 SET character_set_client = @saved_cs_client */; +DROP TABLE IF EXISTS `PipelineManager_keystore`; +/*!40101 SET @saved_cs_client = @@character_set_client */; +/*!50503 SET character_set_client = utf8mb4 */; +CREATE TABLE `PipelineManager_keystore` ( + `id` bigint NOT NULL AUTO_INCREMENT, + `key_name` varchar(100) COLLATE utf8mb4_bin NOT NULL, + `key_value` varchar(512) COLLATE utf8mb4_bin NOT NULL, + `last_updated` datetime(6) NOT NULL, + `policy_id` bigint NOT NULL, + `revision_number` int NOT NULL, + `kv_hash` varchar(64) COLLATE utf8mb4_bin NOT NULL, + `managed_by` varchar(20) COLLATE utf8mb4_bin NOT NULL, + PRIMARY KEY (`id`), + UNIQUE KEY `unique_key_per_policy` (`policy_id`,`key_name`), + CONSTRAINT `PipelineManager_keys_policy_id_6309b699_fk_PipelineM` FOREIGN KEY (`policy_id`) REFERENCES `PipelineManager_policy` (`id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +/*!40101 SET character_set_client = @saved_cs_client */; +DROP TABLE IF EXISTS `PipelineManager_pipeline`; +/*!40101 SET @saved_cs_client = @@character_set_client */; +/*!50503 SET character_set_client = utf8mb4 */; +CREATE TABLE `PipelineManager_pipeline` ( + `id` bigint NOT NULL AUTO_INCREMENT, + `name` varchar(100) COLLATE utf8mb4_bin NOT NULL, + `description` longtext COLLATE utf8mb4_bin, + `lscl` longtext COLLATE utf8mb4_bin NOT NULL, + `lscl_hash` varchar(64) COLLATE utf8mb4_bin DEFAULT NULL, + `last_updated` datetime(6) NOT NULL, + `policy_id` bigint NOT NULL, + `revision_number` int NOT NULL, + `pipeline_batch_delay` int NOT NULL, + `pipeline_batch_size` int NOT NULL, + `pipeline_workers` int NOT NULL, + `queue_checkpoint_writes` int NOT NULL, + `queue_max_bytes` varchar(20) COLLATE utf8mb4_bin NOT NULL, + `queue_type` varchar(20) COLLATE utf8mb4_bin NOT NULL, + `pipeline_hash` varchar(64) COLLATE utf8mb4_bin NOT NULL, + `no_input` tinyint(1) NOT NULL, + `non_reloadable` tinyint(1) NOT NULL, + `managed_by` varchar(20) COLLATE utf8mb4_bin NOT NULL, + PRIMARY KEY (`id`), + UNIQUE KEY `unique_pipeline_per_policy` (`policy_id`,`name`), + CONSTRAINT `PipelineManager_pipe_policy_id_7c3a6fd9_fk_PipelineM` FOREIGN KEY (`policy_id`) REFERENCES `PipelineManager_policy` (`id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +/*!40101 SET character_set_client = @saved_cs_client */; +DROP TABLE IF EXISTS `PipelineManager_policy`; +/*!40101 SET @saved_cs_client = @@character_set_client */; +/*!50503 SET character_set_client = utf8mb4 */; +CREATE TABLE `PipelineManager_policy` ( + `id` bigint NOT NULL AUTO_INCREMENT, + `name` varchar(100) COLLATE utf8mb4_bin NOT NULL, + `settings_path` varchar(255) COLLATE utf8mb4_bin NOT NULL, + `logs_path` varchar(255) COLLATE utf8mb4_bin NOT NULL, + `logstash_yml` longtext COLLATE utf8mb4_bin NOT NULL, + `jvm_options` longtext COLLATE utf8mb4_bin NOT NULL, + `log4j2_properties` longtext COLLATE utf8mb4_bin NOT NULL, + `created_at` datetime(6) NOT NULL, + `updated_at` datetime(6) NOT NULL, + `current_revision_number` int NOT NULL, + `has_undeployed_changes` tinyint(1) NOT NULL, + `jvm_options_hash` varchar(64) COLLATE utf8mb4_bin NOT NULL, + `log4j2_properties_hash` varchar(64) COLLATE utf8mb4_bin NOT NULL, + `logstash_yml_hash` varchar(64) COLLATE utf8mb4_bin NOT NULL, + `binary_path` varchar(255) COLLATE utf8mb4_bin NOT NULL, + `keystore_password` varchar(512) COLLATE utf8mb4_bin DEFAULT NULL, + `keystore_password_hash` varchar(64) COLLATE utf8mb4_bin NOT NULL, + `last_deployed_at` datetime(6) DEFAULT NULL, + `agent_api_port` int unsigned NOT NULL, + `cloned_from_id` bigint DEFAULT NULL, + `data_path` varchar(255) COLLATE utf8mb4_bin NOT NULL, + `is_system` tinyint(1) NOT NULL, + `keystore_env_file` varchar(512) COLLATE utf8mb4_bin NOT NULL, + `logstash_api_port` int unsigned NOT NULL, + `logstash_download_dir` varchar(512) COLLATE utf8mb4_bin NOT NULL, + `logstash_source` varchar(20) COLLATE utf8mb4_bin NOT NULL, + `logstash_version` varchar(32) COLLATE utf8mb4_bin NOT NULL, + `policy_type` varchar(20) COLLATE utf8mb4_bin NOT NULL, + PRIMARY KEY (`id`), + UNIQUE KEY `name` (`name`), + KEY `PipelineManager_poli_cloned_from_id_4177e74a_fk_PipelineM` (`cloned_from_id`), + CONSTRAINT `PipelineManager_poli_cloned_from_id_4177e74a_fk_PipelineM` FOREIGN KEY (`cloned_from_id`) REFERENCES `PipelineManager_policy` (`id`), + CONSTRAINT `PipelineManager_policy_chk_1` CHECK ((`agent_api_port` >= 0)), + CONSTRAINT `PipelineManager_policy_chk_2` CHECK ((`logstash_api_port` >= 0)) +) ENGINE=InnoDB AUTO_INCREMENT=5 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +/*!40101 SET character_set_client = @saved_cs_client */; +DROP TABLE IF EXISTS `PipelineManager_revision`; +/*!40101 SET @saved_cs_client = @@character_set_client */; +/*!50503 SET character_set_client = utf8mb4 */; +CREATE TABLE `PipelineManager_revision` ( + `id` bigint NOT NULL AUTO_INCREMENT, + `revision_number` int NOT NULL, + `snapshot_json` json NOT NULL, + `created_at` datetime(6) NOT NULL, + `created_by` varchar(150) COLLATE utf8mb4_bin NOT NULL, + `policy_id` bigint NOT NULL, + PRIMARY KEY (`id`), + UNIQUE KEY `unique_revision_per_policy` (`policy_id`,`revision_number`), + CONSTRAINT `PipelineManager_revi_policy_id_99e7d146_fk_PipelineM` FOREIGN KEY (`policy_id`) REFERENCES `PipelineManager_policy` (`id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +/*!40101 SET character_set_client = @saved_cs_client */; +DROP TABLE IF EXISTS `SNMP_credential`; +/*!40101 SET @saved_cs_client = @@character_set_client */; +/*!50503 SET character_set_client = utf8mb4 */; +CREATE TABLE `SNMP_credential` ( + `id` bigint NOT NULL AUTO_INCREMENT, + `name` varchar(255) COLLATE utf8mb4_bin NOT NULL, + `description` longtext COLLATE utf8mb4_bin NOT NULL, + `version` varchar(2) COLLATE utf8mb4_bin NOT NULL, + `community` varchar(255) COLLATE utf8mb4_bin NOT NULL, + `security_name` varchar(255) COLLATE utf8mb4_bin NOT NULL, + `security_level` varchar(20) COLLATE utf8mb4_bin NOT NULL, + `auth_protocol` varchar(20) COLLATE utf8mb4_bin NOT NULL, + `auth_pass` varchar(255) COLLATE utf8mb4_bin NOT NULL, + `priv_protocol` varchar(20) COLLATE utf8mb4_bin NOT NULL, + `priv_pass` varchar(255) COLLATE utf8mb4_bin NOT NULL, + `created_at` datetime(6) NOT NULL, + `updated_at` datetime(6) NOT NULL, + PRIMARY KEY (`id`), + UNIQUE KEY `name` (`name`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +/*!40101 SET character_set_client = @saved_cs_client */; +DROP TABLE IF EXISTS `SNMP_device`; +/*!40101 SET @saved_cs_client = @@character_set_client */; +/*!50503 SET character_set_client = utf8mb4 */; +CREATE TABLE `SNMP_device` ( + `id` bigint NOT NULL AUTO_INCREMENT, + `name` varchar(255) COLLATE utf8mb4_bin NOT NULL, + `ip_address` varchar(255) COLLATE utf8mb4_bin DEFAULT NULL, + `port` int NOT NULL, + `retries` int NOT NULL, + `timeout` int NOT NULL, + `created_at` datetime(6) NOT NULL, + `updated_at` datetime(6) NOT NULL, + `credential_id` bigint DEFAULT NULL, + `network_id` bigint DEFAULT NULL, + `device_template_id` bigint DEFAULT NULL, + `hostname` varchar(255) COLLATE utf8mb4_bin DEFAULT NULL, + `building` varchar(255) COLLATE utf8mb4_bin DEFAULT NULL, + `latitude` decimal(15,10) DEFAULT NULL, + `longitude` decimal(15,10) DEFAULT NULL, + `metadata` json NOT NULL DEFAULT (_utf8mb4'{}'), + `room` varchar(255) COLLATE utf8mb4_bin DEFAULT NULL, + `site` varchar(255) COLLATE utf8mb4_bin DEFAULT NULL, + PRIMARY KEY (`id`), + UNIQUE KEY `name` (`name`), + KEY `SNMP_device_credential_id_f229ff4e_fk_SNMP_credential_id` (`credential_id`), + KEY `SNMP_device_name_087021_idx` (`name`), + KEY `SNMP_device_ip_addr_6a90e5_idx` (`ip_address`), + KEY `SNMP_device_created_becb56_idx` (`created_at` DESC), + KEY `SNMP_device_network_889f1c_idx` (`network_id`,`name`), + KEY `SNMP_device_device_template_id_ca143332_fk_SNMP_devi` (`device_template_id`), + KEY `SNMP_device_hostnam_fae88a_idx` (`hostname`), + CONSTRAINT `SNMP_device_credential_id_f229ff4e_fk_SNMP_credential_id` FOREIGN KEY (`credential_id`) REFERENCES `SNMP_credential` (`id`), + CONSTRAINT `SNMP_device_device_template_id_ca143332_fk_SNMP_devi` FOREIGN KEY (`device_template_id`) REFERENCES `SNMP_devicetemplate` (`id`), + CONSTRAINT `SNMP_device_network_id_4dea94fa_fk_SNMP_network_id` FOREIGN KEY (`network_id`) REFERENCES `SNMP_network` (`id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +/*!40101 SET character_set_client = @saved_cs_client */; +DROP TABLE IF EXISTS `SNMP_devicetemplate`; +/*!40101 SET @saved_cs_client = @@character_set_client */; +/*!50503 SET character_set_client = utf8mb4 */; +CREATE TABLE `SNMP_devicetemplate` ( + `id` bigint NOT NULL AUTO_INCREMENT, + `name` varchar(255) COLLATE utf8mb4_bin NOT NULL, + `description` longtext COLLATE utf8mb4_bin NOT NULL, + `vendor` varchar(100) COLLATE utf8mb4_bin NOT NULL, + `model` varchar(100) COLLATE utf8mb4_bin NOT NULL, + `matching_rules` json NOT NULL, + `official` tinyint(1) NOT NULL, + `created_at` datetime(6) NOT NULL, + `updated_at` datetime(6) NOT NULL, + `product` varchar(100) COLLATE utf8mb4_bin NOT NULL, + `type` varchar(100) COLLATE utf8mb4_bin NOT NULL, + `official_key` varchar(255) COLLATE utf8mb4_bin DEFAULT NULL, + PRIMARY KEY (`id`), + UNIQUE KEY `name` (`name`), + UNIQUE KEY `official_key` (`official_key`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +/*!40101 SET character_set_client = @saved_cs_client */; +DROP TABLE IF EXISTS `SNMP_devicetemplate_profiles`; +/*!40101 SET @saved_cs_client = @@character_set_client */; +/*!50503 SET character_set_client = utf8mb4 */; +CREATE TABLE `SNMP_devicetemplate_profiles` ( + `id` bigint NOT NULL AUTO_INCREMENT, + `devicetemplate_id` bigint NOT NULL, + `profile_id` bigint NOT NULL, + PRIMARY KEY (`id`), + UNIQUE KEY `SNMP_devicetemplate_prof_devicetemplate_id_profil_fb6396a5_uniq` (`devicetemplate_id`,`profile_id`), + KEY `SNMP_devicetemplate__profile_id_4cb23513_fk_SNMP_prof` (`profile_id`), + CONSTRAINT `SNMP_devicetemplate__devicetemplate_id_da70425d_fk_SNMP_devi` FOREIGN KEY (`devicetemplate_id`) REFERENCES `SNMP_devicetemplate` (`id`), + CONSTRAINT `SNMP_devicetemplate__profile_id_4cb23513_fk_SNMP_prof` FOREIGN KEY (`profile_id`) REFERENCES `SNMP_profile` (`id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +/*!40101 SET character_set_client = @saved_cs_client */; +DROP TABLE IF EXISTS `SNMP_network`; +/*!40101 SET @saved_cs_client = @@character_set_client */; +/*!50503 SET character_set_client = utf8mb4 */; +CREATE TABLE `SNMP_network` ( + `id` bigint NOT NULL AUTO_INCREMENT, + `name` varchar(255) COLLATE utf8mb4_bin NOT NULL, + `network_range` varchar(50) COLLATE utf8mb4_bin NOT NULL, + `discovery_enabled` tinyint(1) NOT NULL, + `traps_enabled` tinyint(1) NOT NULL, + `interval` int unsigned NOT NULL, + `created_at` datetime(6) NOT NULL, + `updated_at` datetime(6) NOT NULL, + `connection_id` bigint DEFAULT NULL, + `credential_id` bigint DEFAULT NULL, + `discovery_credential_id` bigint DEFAULT NULL, + `namespace` varchar(100) COLLATE utf8mb4_bin NOT NULL, + `namespace_from_device_template` tinyint(1) NOT NULL, + `agent_connection_id` bigint DEFAULT NULL, + `deployment_mode` varchar(20) COLLATE utf8mb4_bin NOT NULL, + `credential_mode` varchar(20) COLLATE utf8mb4_bin NOT NULL, + PRIMARY KEY (`id`), + UNIQUE KEY `name` (`name`), + KEY `SNMP_network_connection_id_432a2f88_fk_PipelineM` (`connection_id`), + KEY `SNMP_network_credential_id_3b15dfc1_fk_SNMP_credential_id` (`credential_id`), + KEY `SNMP_network_discovery_credential_d9229589_fk_SNMP_cred` (`discovery_credential_id`), + KEY `SNMP_network_agent_connection_id_cf646bb5_fk_PipelineM` (`agent_connection_id`), + CONSTRAINT `SNMP_network_agent_connection_id_cf646bb5_fk_PipelineM` FOREIGN KEY (`agent_connection_id`) REFERENCES `PipelineManager_connection` (`id`), + CONSTRAINT `SNMP_network_connection_id_432a2f88_fk_PipelineM` FOREIGN KEY (`connection_id`) REFERENCES `PipelineManager_connection` (`id`), + CONSTRAINT `SNMP_network_credential_id_3b15dfc1_fk_SNMP_credential_id` FOREIGN KEY (`credential_id`) REFERENCES `SNMP_credential` (`id`), + CONSTRAINT `SNMP_network_discovery_credential_d9229589_fk_SNMP_cred` FOREIGN KEY (`discovery_credential_id`) REFERENCES `SNMP_credential` (`id`), + CONSTRAINT `SNMP_network_chk_1` CHECK ((`interval` >= 0)) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +/*!40101 SET character_set_client = @saved_cs_client */; +DROP TABLE IF EXISTS `SNMP_profile`; +/*!40101 SET @saved_cs_client = @@character_set_client */; +/*!50503 SET character_set_client = utf8mb4 */; +CREATE TABLE `SNMP_profile` ( + `id` bigint NOT NULL AUTO_INCREMENT, + `name` varchar(255) COLLATE utf8mb4_bin NOT NULL, + `profile_data` json NOT NULL, + `description` longtext COLLATE utf8mb4_bin NOT NULL, + `vendor` varchar(100) COLLATE utf8mb4_bin NOT NULL, + `created_at` datetime(6) NOT NULL, + `updated_at` datetime(6) NOT NULL, + `product` varchar(100) COLLATE utf8mb4_bin NOT NULL, + `normalizers` json NOT NULL DEFAULT (_utf8mb4'[]'), + `official_key` varchar(255) COLLATE utf8mb4_bin DEFAULT NULL, + PRIMARY KEY (`id`), + UNIQUE KEY `name` (`name`), + UNIQUE KEY `official_key` (`official_key`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +/*!40101 SET character_set_client = @saved_cs_client */; +DROP TABLE IF EXISTS `auth_group`; +/*!40101 SET @saved_cs_client = @@character_set_client */; +/*!50503 SET character_set_client = utf8mb4 */; +CREATE TABLE `auth_group` ( + `id` int NOT NULL AUTO_INCREMENT, + `name` varchar(150) COLLATE utf8mb4_bin NOT NULL, + PRIMARY KEY (`id`), + UNIQUE KEY `name` (`name`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +/*!40101 SET character_set_client = @saved_cs_client */; +DROP TABLE IF EXISTS `auth_group_permissions`; +/*!40101 SET @saved_cs_client = @@character_set_client */; +/*!50503 SET character_set_client = utf8mb4 */; +CREATE TABLE `auth_group_permissions` ( + `id` bigint NOT NULL AUTO_INCREMENT, + `group_id` int NOT NULL, + `permission_id` int NOT NULL, + PRIMARY KEY (`id`), + UNIQUE KEY `auth_group_permissions_group_id_permission_id_0cd325b0_uniq` (`group_id`,`permission_id`), + KEY `auth_group_permissio_permission_id_84c5c92e_fk_auth_perm` (`permission_id`), + CONSTRAINT `auth_group_permissio_permission_id_84c5c92e_fk_auth_perm` FOREIGN KEY (`permission_id`) REFERENCES `auth_permission` (`id`), + CONSTRAINT `auth_group_permissions_group_id_b120cbf9_fk_auth_group_id` FOREIGN KEY (`group_id`) REFERENCES `auth_group` (`id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +/*!40101 SET character_set_client = @saved_cs_client */; +DROP TABLE IF EXISTS `auth_permission`; +/*!40101 SET @saved_cs_client = @@character_set_client */; +/*!50503 SET character_set_client = utf8mb4 */; +CREATE TABLE `auth_permission` ( + `id` int NOT NULL AUTO_INCREMENT, + `name` varchar(255) COLLATE utf8mb4_bin NOT NULL, + `content_type_id` int NOT NULL, + `codename` varchar(100) COLLATE utf8mb4_bin NOT NULL, + PRIMARY KEY (`id`), + UNIQUE KEY `auth_permission_content_type_id_codename_01ab375a_uniq` (`content_type_id`,`codename`), + CONSTRAINT `auth_permission_content_type_id_2f476e4b_fk_django_co` FOREIGN KEY (`content_type_id`) REFERENCES `django_content_type` (`id`) +) ENGINE=InnoDB AUTO_INCREMENT=85 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +/*!40101 SET character_set_client = @saved_cs_client */; +DROP TABLE IF EXISTS `auth_user`; +/*!40101 SET @saved_cs_client = @@character_set_client */; +/*!50503 SET character_set_client = utf8mb4 */; +CREATE TABLE `auth_user` ( + `id` int NOT NULL AUTO_INCREMENT, + `password` varchar(128) COLLATE utf8mb4_bin NOT NULL, + `last_login` datetime(6) DEFAULT NULL, + `is_superuser` tinyint(1) NOT NULL, + `username` varchar(150) COLLATE utf8mb4_bin NOT NULL, + `first_name` varchar(150) COLLATE utf8mb4_bin NOT NULL, + `last_name` varchar(150) COLLATE utf8mb4_bin NOT NULL, + `email` varchar(254) COLLATE utf8mb4_bin NOT NULL, + `is_staff` tinyint(1) NOT NULL, + `is_active` tinyint(1) NOT NULL, + `date_joined` datetime(6) NOT NULL, + PRIMARY KEY (`id`), + UNIQUE KEY `username` (`username`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +/*!40101 SET character_set_client = @saved_cs_client */; +DROP TABLE IF EXISTS `auth_user_groups`; +/*!40101 SET @saved_cs_client = @@character_set_client */; +/*!50503 SET character_set_client = utf8mb4 */; +CREATE TABLE `auth_user_groups` ( + `id` bigint NOT NULL AUTO_INCREMENT, + `user_id` int NOT NULL, + `group_id` int NOT NULL, + PRIMARY KEY (`id`), + UNIQUE KEY `auth_user_groups_user_id_group_id_94350c0c_uniq` (`user_id`,`group_id`), + KEY `auth_user_groups_group_id_97559544_fk_auth_group_id` (`group_id`), + CONSTRAINT `auth_user_groups_group_id_97559544_fk_auth_group_id` FOREIGN KEY (`group_id`) REFERENCES `auth_group` (`id`), + CONSTRAINT `auth_user_groups_user_id_6a12ed8b_fk_auth_user_id` FOREIGN KEY (`user_id`) REFERENCES `auth_user` (`id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +/*!40101 SET character_set_client = @saved_cs_client */; +DROP TABLE IF EXISTS `auth_user_user_permissions`; +/*!40101 SET @saved_cs_client = @@character_set_client */; +/*!50503 SET character_set_client = utf8mb4 */; +CREATE TABLE `auth_user_user_permissions` ( + `id` bigint NOT NULL AUTO_INCREMENT, + `user_id` int NOT NULL, + `permission_id` int NOT NULL, + PRIMARY KEY (`id`), + UNIQUE KEY `auth_user_user_permissions_user_id_permission_id_14a6b632_uniq` (`user_id`,`permission_id`), + KEY `auth_user_user_permi_permission_id_1fbb5f2c_fk_auth_perm` (`permission_id`), + CONSTRAINT `auth_user_user_permi_permission_id_1fbb5f2c_fk_auth_perm` FOREIGN KEY (`permission_id`) REFERENCES `auth_permission` (`id`), + CONSTRAINT `auth_user_user_permissions_user_id_a95ead1b_fk_auth_user_id` FOREIGN KEY (`user_id`) REFERENCES `auth_user` (`id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +/*!40101 SET character_set_client = @saved_cs_client */; +DROP TABLE IF EXISTS `django_admin_log`; +/*!40101 SET @saved_cs_client = @@character_set_client */; +/*!50503 SET character_set_client = utf8mb4 */; +CREATE TABLE `django_admin_log` ( + `id` int NOT NULL AUTO_INCREMENT, + `action_time` datetime(6) NOT NULL, + `object_id` longtext COLLATE utf8mb4_bin, + `object_repr` varchar(200) COLLATE utf8mb4_bin NOT NULL, + `action_flag` smallint unsigned NOT NULL, + `change_message` longtext COLLATE utf8mb4_bin NOT NULL, + `content_type_id` int DEFAULT NULL, + `user_id` int NOT NULL, + PRIMARY KEY (`id`), + KEY `django_admin_log_content_type_id_c4bce8eb_fk_django_co` (`content_type_id`), + KEY `django_admin_log_user_id_c564eba6_fk_auth_user_id` (`user_id`), + CONSTRAINT `django_admin_log_content_type_id_c4bce8eb_fk_django_co` FOREIGN KEY (`content_type_id`) REFERENCES `django_content_type` (`id`), + CONSTRAINT `django_admin_log_user_id_c564eba6_fk_auth_user_id` FOREIGN KEY (`user_id`) REFERENCES `auth_user` (`id`), + CONSTRAINT `django_admin_log_chk_1` CHECK ((`action_flag` >= 0)) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +/*!40101 SET character_set_client = @saved_cs_client */; +DROP TABLE IF EXISTS `django_content_type`; +/*!40101 SET @saved_cs_client = @@character_set_client */; +/*!50503 SET character_set_client = utf8mb4 */; +CREATE TABLE `django_content_type` ( + `id` int NOT NULL AUTO_INCREMENT, + `app_label` varchar(100) COLLATE utf8mb4_bin NOT NULL, + `model` varchar(100) COLLATE utf8mb4_bin NOT NULL, + PRIMARY KEY (`id`), + UNIQUE KEY `django_content_type_app_label_model_76bd3d3b_uniq` (`app_label`,`model`) +) ENGINE=InnoDB AUTO_INCREMENT=22 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +/*!40101 SET character_set_client = @saved_cs_client */; +DROP TABLE IF EXISTS `django_migrations`; +/*!40101 SET @saved_cs_client = @@character_set_client */; +/*!50503 SET character_set_client = utf8mb4 */; +CREATE TABLE `django_migrations` ( + `id` bigint NOT NULL AUTO_INCREMENT, + `app` varchar(255) COLLATE utf8mb4_bin NOT NULL, + `name` varchar(255) COLLATE utf8mb4_bin NOT NULL, + `applied` datetime(6) NOT NULL, + PRIMARY KEY (`id`) +) ENGINE=InnoDB AUTO_INCREMENT=76 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +/*!40101 SET character_set_client = @saved_cs_client */; +DROP TABLE IF EXISTS `django_session`; +/*!40101 SET @saved_cs_client = @@character_set_client */; +/*!50503 SET character_set_client = utf8mb4 */; +CREATE TABLE `django_session` ( + `session_key` varchar(40) COLLATE utf8mb4_bin NOT NULL, + `session_data` longtext COLLATE utf8mb4_bin NOT NULL, + `expire_date` datetime(6) NOT NULL, + PRIMARY KEY (`session_key`), + KEY `django_session_expire_date_a5c62663` (`expire_date`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +/*!40101 SET character_set_client = @saved_cs_client */; +DROP TABLE IF EXISTS `settings`; +/*!40101 SET @saved_cs_client = @@character_set_client */; +/*!50503 SET character_set_client = utf8mb4 */; +CREATE TABLE `settings` ( + `id` bigint NOT NULL AUTO_INCREMENT, + `experimental_mode` tinyint(1) NOT NULL, + `agent_ui_url` varchar(512) COLLATE utf8mb4_bin NOT NULL, + PRIMARY KEY (`id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +/*!40101 SET character_set_client = @saved_cs_client */; +DROP TABLE IF EXISTS `snmp_deployment_state`; +/*!40101 SET @saved_cs_client = @@character_set_client */; +/*!50503 SET character_set_client = utf8mb4 */; +CREATE TABLE `snmp_deployment_state` ( + `id` bigint NOT NULL AUTO_INCREMENT, + `last_deployment` datetime(6) DEFAULT NULL, + `last_config_change` datetime(6) DEFAULT NULL, + PRIMARY KEY (`id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +/*!40101 SET character_set_client = @saved_cs_client */; +/*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */; + +/*!40101 SET SQL_MODE=@OLD_SQL_MODE */; +/*!40014 SET FOREIGN_KEY_CHECKS=@OLD_FOREIGN_KEY_CHECKS */; +/*!40014 SET UNIQUE_CHECKS=@OLD_UNIQUE_CHECKS */; +/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */; +/*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */; +/*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */; +/*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */; + diff --git a/docs/docs/logstashui/database/examples/schema-postgresql.sql b/docs/docs/logstashui/database/examples/schema-postgresql.sql new file mode 100644 index 0000000..b168a7e --- /dev/null +++ b/docs/docs/logstashui/database/examples/schema-postgresql.sql @@ -0,0 +1,1709 @@ +-- Snapshot of LogstashUI 0.5.2 `migrate` DDL on PostgreSQL 16. +-- Generated with pg_dump --schema-only. Do NOT apply this instead of +-- `logstashui manage migrate --noinput`. It will go stale with new migrations. +-- Create the empty database first (see create-postgresql.sql). + +-- +-- PostgreSQL database dump +-- + + +-- Dumped from database version 16.15 (Debian 16.15-1.pgdg13+2) +-- Dumped by pg_dump version 16.15 (Debian 16.15-1.pgdg13+2) + +SET statement_timeout = 0; +SET lock_timeout = 0; +SET idle_in_transaction_session_timeout = 0; +SET client_encoding = 'UTF8'; +SET standard_conforming_strings = on; +SELECT pg_catalog.set_config('search_path', '', false); +SET check_function_bodies = false; +SET xmloption = content; +SET client_min_messages = warning; +SET row_security = off; + +SET default_tablespace = ''; + +SET default_table_access_method = heap; + +-- +-- Name: Management_userprofile; Type: TABLE; Schema: public; Owner: - +-- + +CREATE TABLE public."Management_userprofile" ( + id bigint NOT NULL, + role character varying(20) NOT NULL, + user_id integer NOT NULL +); + + +-- +-- Name: Management_userprofile_id_seq; Type: SEQUENCE; Schema: public; Owner: - +-- + +ALTER TABLE public."Management_userprofile" ALTER COLUMN id ADD GENERATED BY DEFAULT AS IDENTITY ( + SEQUENCE NAME public."Management_userprofile_id_seq" + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1 +); + + +-- +-- Name: PipelineManager_apikey; Type: TABLE; Schema: public; Owner: - +-- + +CREATE TABLE public."PipelineManager_apikey" ( + id bigint NOT NULL, + api_key character varying(512) NOT NULL, + connection_id bigint NOT NULL +); + + +-- +-- Name: PipelineManager_apikey_id_seq; Type: SEQUENCE; Schema: public; Owner: - +-- + +ALTER TABLE public."PipelineManager_apikey" ALTER COLUMN id ADD GENERATED BY DEFAULT AS IDENTITY ( + SEQUENCE NAME public."PipelineManager_apikey_id_seq" + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1 +); + + +-- +-- Name: PipelineManager_connection; Type: TABLE; Schema: public; Owner: - +-- + +CREATE TABLE public."PipelineManager_connection" ( + id bigint NOT NULL, + name character varying(100) NOT NULL, + connection_type character varying(20) NOT NULL, + host character varying(255), + port integer, + username character varying(100), + password character varying(512), + ssh_key text, + cloud_id character varying(255), + cloud_url character varying(200), + api_key character varying(512), + created_at timestamp with time zone NOT NULL, + updated_at timestamp with time zone NOT NULL, + is_active boolean NOT NULL, + policy_id bigint, + agent_id character varying(255), + last_check_in timestamp with time zone, + status_blob jsonb, + restart_on_next_checkin boolean NOT NULL, + desired_agent_version character varying(20), + agent_api_port integer, + instance_id integer, + last_selected_at timestamp with time zone, + logstash_api_port integer, + logstash_version_resolved character varying(64) NOT NULL, + CONSTRAINT "PipelineManager_connection_agent_api_port_check" CHECK ((agent_api_port >= 0)), + CONSTRAINT "PipelineManager_connection_instance_id_check" CHECK ((instance_id >= 0)), + CONSTRAINT "PipelineManager_connection_logstash_api_port_check" CHECK ((logstash_api_port >= 0)), + CONSTRAINT "PipelineManager_connection_port_check" CHECK ((port >= 0)) +); + + +-- +-- Name: PipelineManager_connection_id_seq; Type: SEQUENCE; Schema: public; Owner: - +-- + +ALTER TABLE public."PipelineManager_connection" ALTER COLUMN id ADD GENERATED BY DEFAULT AS IDENTITY ( + SEQUENCE NAME public."PipelineManager_connection_id_seq" + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1 +); + + +-- +-- Name: PipelineManager_enrollmenttoken; Type: TABLE; Schema: public; Owner: - +-- + +CREATE TABLE public."PipelineManager_enrollmenttoken" ( + id bigint NOT NULL, + token character varying(512) NOT NULL, + policy_id bigint NOT NULL, + name character varying(100) NOT NULL +); + + +-- +-- Name: PipelineManager_enrollmenttoken_id_seq; Type: SEQUENCE; Schema: public; Owner: - +-- + +ALTER TABLE public."PipelineManager_enrollmenttoken" ALTER COLUMN id ADD GENERATED BY DEFAULT AS IDENTITY ( + SEQUENCE NAME public."PipelineManager_enrollmenttoken_id_seq" + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1 +); + + +-- +-- Name: PipelineManager_keystore; Type: TABLE; Schema: public; Owner: - +-- + +CREATE TABLE public."PipelineManager_keystore" ( + id bigint NOT NULL, + key_name character varying(100) NOT NULL, + key_value character varying(512) NOT NULL, + last_updated timestamp with time zone NOT NULL, + policy_id bigint NOT NULL, + revision_number integer NOT NULL, + kv_hash character varying(64) NOT NULL, + managed_by character varying(20) NOT NULL +); + + +-- +-- Name: PipelineManager_keystore_id_seq; Type: SEQUENCE; Schema: public; Owner: - +-- + +ALTER TABLE public."PipelineManager_keystore" ALTER COLUMN id ADD GENERATED BY DEFAULT AS IDENTITY ( + SEQUENCE NAME public."PipelineManager_keystore_id_seq" + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1 +); + + +-- +-- Name: PipelineManager_pipeline; Type: TABLE; Schema: public; Owner: - +-- + +CREATE TABLE public."PipelineManager_pipeline" ( + id bigint NOT NULL, + name character varying(100) NOT NULL, + description text, + lscl text NOT NULL, + lscl_hash character varying(64), + last_updated timestamp with time zone NOT NULL, + policy_id bigint NOT NULL, + revision_number integer NOT NULL, + pipeline_batch_delay integer NOT NULL, + pipeline_batch_size integer NOT NULL, + pipeline_workers integer NOT NULL, + queue_checkpoint_writes integer NOT NULL, + queue_max_bytes character varying(20) NOT NULL, + queue_type character varying(20) NOT NULL, + pipeline_hash character varying(64) NOT NULL, + no_input boolean NOT NULL, + non_reloadable boolean NOT NULL, + managed_by character varying(20) NOT NULL +); + + +-- +-- Name: PipelineManager_pipeline_id_seq; Type: SEQUENCE; Schema: public; Owner: - +-- + +ALTER TABLE public."PipelineManager_pipeline" ALTER COLUMN id ADD GENERATED BY DEFAULT AS IDENTITY ( + SEQUENCE NAME public."PipelineManager_pipeline_id_seq" + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1 +); + + +-- +-- Name: PipelineManager_policy; Type: TABLE; Schema: public; Owner: - +-- + +CREATE TABLE public."PipelineManager_policy" ( + id bigint NOT NULL, + name character varying(100) NOT NULL, + settings_path character varying(255) NOT NULL, + logs_path character varying(255) NOT NULL, + logstash_yml text NOT NULL, + jvm_options text NOT NULL, + log4j2_properties text NOT NULL, + created_at timestamp with time zone NOT NULL, + updated_at timestamp with time zone NOT NULL, + current_revision_number integer NOT NULL, + has_undeployed_changes boolean NOT NULL, + jvm_options_hash character varying(64) NOT NULL, + log4j2_properties_hash character varying(64) NOT NULL, + logstash_yml_hash character varying(64) NOT NULL, + binary_path character varying(255) NOT NULL, + keystore_password character varying(512), + keystore_password_hash character varying(64) NOT NULL, + last_deployed_at timestamp with time zone, + agent_api_port integer NOT NULL, + cloned_from_id bigint, + data_path character varying(255) NOT NULL, + is_system boolean NOT NULL, + keystore_env_file character varying(512) NOT NULL, + logstash_api_port integer NOT NULL, + logstash_download_dir character varying(512) NOT NULL, + logstash_source character varying(20) NOT NULL, + logstash_version character varying(32) NOT NULL, + policy_type character varying(20) NOT NULL, + CONSTRAINT "PipelineManager_policy_agent_api_port_check" CHECK ((agent_api_port >= 0)), + CONSTRAINT "PipelineManager_policy_logstash_api_port_check" CHECK ((logstash_api_port >= 0)) +); + + +-- +-- Name: PipelineManager_policy_id_seq; Type: SEQUENCE; Schema: public; Owner: - +-- + +ALTER TABLE public."PipelineManager_policy" ALTER COLUMN id ADD GENERATED BY DEFAULT AS IDENTITY ( + SEQUENCE NAME public."PipelineManager_policy_id_seq" + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1 +); + + +-- +-- Name: PipelineManager_revision; Type: TABLE; Schema: public; Owner: - +-- + +CREATE TABLE public."PipelineManager_revision" ( + id bigint NOT NULL, + revision_number integer NOT NULL, + snapshot_json jsonb NOT NULL, + created_at timestamp with time zone NOT NULL, + created_by character varying(150) NOT NULL, + policy_id bigint NOT NULL +); + + +-- +-- Name: PipelineManager_revision_id_seq; Type: SEQUENCE; Schema: public; Owner: - +-- + +ALTER TABLE public."PipelineManager_revision" ALTER COLUMN id ADD GENERATED BY DEFAULT AS IDENTITY ( + SEQUENCE NAME public."PipelineManager_revision_id_seq" + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1 +); + + +-- +-- Name: SNMP_credential; Type: TABLE; Schema: public; Owner: - +-- + +CREATE TABLE public."SNMP_credential" ( + id bigint NOT NULL, + name character varying(255) NOT NULL, + description text NOT NULL, + version character varying(2) NOT NULL, + community character varying(255) NOT NULL, + security_name character varying(255) NOT NULL, + security_level character varying(20) NOT NULL, + auth_protocol character varying(20) NOT NULL, + auth_pass character varying(255) NOT NULL, + priv_protocol character varying(20) NOT NULL, + priv_pass character varying(255) NOT NULL, + created_at timestamp with time zone NOT NULL, + updated_at timestamp with time zone NOT NULL +); + + +-- +-- Name: SNMP_credential_id_seq; Type: SEQUENCE; Schema: public; Owner: - +-- + +ALTER TABLE public."SNMP_credential" ALTER COLUMN id ADD GENERATED BY DEFAULT AS IDENTITY ( + SEQUENCE NAME public."SNMP_credential_id_seq" + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1 +); + + +-- +-- Name: SNMP_device; Type: TABLE; Schema: public; Owner: - +-- + +CREATE TABLE public."SNMP_device" ( + id bigint NOT NULL, + name character varying(255) NOT NULL, + ip_address character varying(255), + port integer NOT NULL, + retries integer NOT NULL, + timeout integer NOT NULL, + created_at timestamp with time zone NOT NULL, + updated_at timestamp with time zone NOT NULL, + credential_id bigint, + network_id bigint, + device_template_id bigint, + hostname character varying(255), + building character varying(255), + latitude numeric(15,10), + longitude numeric(15,10), + metadata jsonb NOT NULL, + room character varying(255), + site character varying(255) +); + + +-- +-- Name: SNMP_device_id_seq; Type: SEQUENCE; Schema: public; Owner: - +-- + +ALTER TABLE public."SNMP_device" ALTER COLUMN id ADD GENERATED BY DEFAULT AS IDENTITY ( + SEQUENCE NAME public."SNMP_device_id_seq" + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1 +); + + +-- +-- Name: SNMP_devicetemplate; Type: TABLE; Schema: public; Owner: - +-- + +CREATE TABLE public."SNMP_devicetemplate" ( + id bigint NOT NULL, + name character varying(255) NOT NULL, + description text NOT NULL, + vendor character varying(100) NOT NULL, + model character varying(100) NOT NULL, + matching_rules jsonb NOT NULL, + official boolean NOT NULL, + created_at timestamp with time zone NOT NULL, + updated_at timestamp with time zone NOT NULL, + product character varying(100) NOT NULL, + type character varying(100) NOT NULL, + official_key character varying(255) +); + + +-- +-- Name: SNMP_devicetemplate_id_seq; Type: SEQUENCE; Schema: public; Owner: - +-- + +ALTER TABLE public."SNMP_devicetemplate" ALTER COLUMN id ADD GENERATED BY DEFAULT AS IDENTITY ( + SEQUENCE NAME public."SNMP_devicetemplate_id_seq" + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1 +); + + +-- +-- Name: SNMP_devicetemplate_profiles; Type: TABLE; Schema: public; Owner: - +-- + +CREATE TABLE public."SNMP_devicetemplate_profiles" ( + id bigint NOT NULL, + devicetemplate_id bigint NOT NULL, + profile_id bigint NOT NULL +); + + +-- +-- Name: SNMP_devicetemplate_profiles_id_seq; Type: SEQUENCE; Schema: public; Owner: - +-- + +ALTER TABLE public."SNMP_devicetemplate_profiles" ALTER COLUMN id ADD GENERATED BY DEFAULT AS IDENTITY ( + SEQUENCE NAME public."SNMP_devicetemplate_profiles_id_seq" + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1 +); + + +-- +-- Name: SNMP_network; Type: TABLE; Schema: public; Owner: - +-- + +CREATE TABLE public."SNMP_network" ( + id bigint NOT NULL, + name character varying(255) NOT NULL, + network_range character varying(50) NOT NULL, + discovery_enabled boolean NOT NULL, + traps_enabled boolean NOT NULL, + "interval" integer NOT NULL, + created_at timestamp with time zone NOT NULL, + updated_at timestamp with time zone NOT NULL, + connection_id bigint, + credential_id bigint, + discovery_credential_id bigint, + namespace character varying(100) NOT NULL, + namespace_from_device_template boolean NOT NULL, + agent_connection_id bigint, + deployment_mode character varying(20) NOT NULL, + credential_mode character varying(20) NOT NULL, + CONSTRAINT "SNMP_network_interval_check" CHECK (("interval" >= 0)) +); + + +-- +-- Name: SNMP_network_id_seq; Type: SEQUENCE; Schema: public; Owner: - +-- + +ALTER TABLE public."SNMP_network" ALTER COLUMN id ADD GENERATED BY DEFAULT AS IDENTITY ( + SEQUENCE NAME public."SNMP_network_id_seq" + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1 +); + + +-- +-- Name: SNMP_profile; Type: TABLE; Schema: public; Owner: - +-- + +CREATE TABLE public."SNMP_profile" ( + id bigint NOT NULL, + name character varying(255) NOT NULL, + profile_data jsonb NOT NULL, + description text NOT NULL, + vendor character varying(100) NOT NULL, + created_at timestamp with time zone NOT NULL, + updated_at timestamp with time zone NOT NULL, + product character varying(100) NOT NULL, + normalizers jsonb NOT NULL, + official_key character varying(255) +); + + +-- +-- Name: SNMP_profile_id_seq; Type: SEQUENCE; Schema: public; Owner: - +-- + +ALTER TABLE public."SNMP_profile" ALTER COLUMN id ADD GENERATED BY DEFAULT AS IDENTITY ( + SEQUENCE NAME public."SNMP_profile_id_seq" + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1 +); + + +-- +-- Name: auth_group; Type: TABLE; Schema: public; Owner: - +-- + +CREATE TABLE public.auth_group ( + id integer NOT NULL, + name character varying(150) NOT NULL +); + + +-- +-- Name: auth_group_id_seq; Type: SEQUENCE; Schema: public; Owner: - +-- + +ALTER TABLE public.auth_group ALTER COLUMN id ADD GENERATED BY DEFAULT AS IDENTITY ( + SEQUENCE NAME public.auth_group_id_seq + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1 +); + + +-- +-- Name: auth_group_permissions; Type: TABLE; Schema: public; Owner: - +-- + +CREATE TABLE public.auth_group_permissions ( + id bigint NOT NULL, + group_id integer NOT NULL, + permission_id integer NOT NULL +); + + +-- +-- Name: auth_group_permissions_id_seq; Type: SEQUENCE; Schema: public; Owner: - +-- + +ALTER TABLE public.auth_group_permissions ALTER COLUMN id ADD GENERATED BY DEFAULT AS IDENTITY ( + SEQUENCE NAME public.auth_group_permissions_id_seq + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1 +); + + +-- +-- Name: auth_permission; Type: TABLE; Schema: public; Owner: - +-- + +CREATE TABLE public.auth_permission ( + id integer NOT NULL, + name character varying(255) NOT NULL, + content_type_id integer NOT NULL, + codename character varying(100) NOT NULL +); + + +-- +-- Name: auth_permission_id_seq; Type: SEQUENCE; Schema: public; Owner: - +-- + +ALTER TABLE public.auth_permission ALTER COLUMN id ADD GENERATED BY DEFAULT AS IDENTITY ( + SEQUENCE NAME public.auth_permission_id_seq + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1 +); + + +-- +-- Name: auth_user; Type: TABLE; Schema: public; Owner: - +-- + +CREATE TABLE public.auth_user ( + id integer NOT NULL, + password character varying(128) NOT NULL, + last_login timestamp with time zone, + is_superuser boolean NOT NULL, + username character varying(150) NOT NULL, + first_name character varying(150) NOT NULL, + last_name character varying(150) NOT NULL, + email character varying(254) NOT NULL, + is_staff boolean NOT NULL, + is_active boolean NOT NULL, + date_joined timestamp with time zone NOT NULL +); + + +-- +-- Name: auth_user_groups; Type: TABLE; Schema: public; Owner: - +-- + +CREATE TABLE public.auth_user_groups ( + id bigint NOT NULL, + user_id integer NOT NULL, + group_id integer NOT NULL +); + + +-- +-- Name: auth_user_groups_id_seq; Type: SEQUENCE; Schema: public; Owner: - +-- + +ALTER TABLE public.auth_user_groups ALTER COLUMN id ADD GENERATED BY DEFAULT AS IDENTITY ( + SEQUENCE NAME public.auth_user_groups_id_seq + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1 +); + + +-- +-- Name: auth_user_id_seq; Type: SEQUENCE; Schema: public; Owner: - +-- + +ALTER TABLE public.auth_user ALTER COLUMN id ADD GENERATED BY DEFAULT AS IDENTITY ( + SEQUENCE NAME public.auth_user_id_seq + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1 +); + + +-- +-- Name: auth_user_user_permissions; Type: TABLE; Schema: public; Owner: - +-- + +CREATE TABLE public.auth_user_user_permissions ( + id bigint NOT NULL, + user_id integer NOT NULL, + permission_id integer NOT NULL +); + + +-- +-- Name: auth_user_user_permissions_id_seq; Type: SEQUENCE; Schema: public; Owner: - +-- + +ALTER TABLE public.auth_user_user_permissions ALTER COLUMN id ADD GENERATED BY DEFAULT AS IDENTITY ( + SEQUENCE NAME public.auth_user_user_permissions_id_seq + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1 +); + + +-- +-- Name: django_admin_log; Type: TABLE; Schema: public; Owner: - +-- + +CREATE TABLE public.django_admin_log ( + id integer NOT NULL, + action_time timestamp with time zone NOT NULL, + object_id text, + object_repr character varying(200) NOT NULL, + action_flag smallint NOT NULL, + change_message text NOT NULL, + content_type_id integer, + user_id integer NOT NULL, + CONSTRAINT django_admin_log_action_flag_check CHECK ((action_flag >= 0)) +); + + +-- +-- Name: django_admin_log_id_seq; Type: SEQUENCE; Schema: public; Owner: - +-- + +ALTER TABLE public.django_admin_log ALTER COLUMN id ADD GENERATED BY DEFAULT AS IDENTITY ( + SEQUENCE NAME public.django_admin_log_id_seq + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1 +); + + +-- +-- Name: django_content_type; Type: TABLE; Schema: public; Owner: - +-- + +CREATE TABLE public.django_content_type ( + id integer NOT NULL, + app_label character varying(100) NOT NULL, + model character varying(100) NOT NULL +); + + +-- +-- Name: django_content_type_id_seq; Type: SEQUENCE; Schema: public; Owner: - +-- + +ALTER TABLE public.django_content_type ALTER COLUMN id ADD GENERATED BY DEFAULT AS IDENTITY ( + SEQUENCE NAME public.django_content_type_id_seq + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1 +); + + +-- +-- Name: django_migrations; Type: TABLE; Schema: public; Owner: - +-- + +CREATE TABLE public.django_migrations ( + id bigint NOT NULL, + app character varying(255) NOT NULL, + name character varying(255) NOT NULL, + applied timestamp with time zone NOT NULL +); + + +-- +-- Name: django_migrations_id_seq; Type: SEQUENCE; Schema: public; Owner: - +-- + +ALTER TABLE public.django_migrations ALTER COLUMN id ADD GENERATED BY DEFAULT AS IDENTITY ( + SEQUENCE NAME public.django_migrations_id_seq + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1 +); + + +-- +-- Name: django_session; Type: TABLE; Schema: public; Owner: - +-- + +CREATE TABLE public.django_session ( + session_key character varying(40) NOT NULL, + session_data text NOT NULL, + expire_date timestamp with time zone NOT NULL +); + + +-- +-- Name: settings; Type: TABLE; Schema: public; Owner: - +-- + +CREATE TABLE public.settings ( + id bigint NOT NULL, + experimental_mode boolean NOT NULL, + agent_ui_url character varying(512) NOT NULL +); + + +-- +-- Name: settings_id_seq; Type: SEQUENCE; Schema: public; Owner: - +-- + +ALTER TABLE public.settings ALTER COLUMN id ADD GENERATED BY DEFAULT AS IDENTITY ( + SEQUENCE NAME public.settings_id_seq + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1 +); + + +-- +-- Name: snmp_deployment_state; Type: TABLE; Schema: public; Owner: - +-- + +CREATE TABLE public.snmp_deployment_state ( + id bigint NOT NULL, + last_deployment timestamp with time zone, + last_config_change timestamp with time zone +); + + +-- +-- Name: snmp_deployment_state_id_seq; Type: SEQUENCE; Schema: public; Owner: - +-- + +ALTER TABLE public.snmp_deployment_state ALTER COLUMN id ADD GENERATED BY DEFAULT AS IDENTITY ( + SEQUENCE NAME public.snmp_deployment_state_id_seq + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1 +); + + +-- +-- Name: Management_userprofile Management_userprofile_pkey; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."Management_userprofile" + ADD CONSTRAINT "Management_userprofile_pkey" PRIMARY KEY (id); + + +-- +-- Name: Management_userprofile Management_userprofile_user_id_key; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."Management_userprofile" + ADD CONSTRAINT "Management_userprofile_user_id_key" UNIQUE (user_id); + + +-- +-- Name: PipelineManager_apikey PipelineManager_apikey_pkey; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."PipelineManager_apikey" + ADD CONSTRAINT "PipelineManager_apikey_pkey" PRIMARY KEY (id); + + +-- +-- Name: PipelineManager_connection PipelineManager_connection_agent_id_key; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."PipelineManager_connection" + ADD CONSTRAINT "PipelineManager_connection_agent_id_key" UNIQUE (agent_id); + + +-- +-- Name: PipelineManager_connection PipelineManager_connection_pkey; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."PipelineManager_connection" + ADD CONSTRAINT "PipelineManager_connection_pkey" PRIMARY KEY (id); + + +-- +-- Name: PipelineManager_enrollmenttoken PipelineManager_enrollmenttoken_pkey; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."PipelineManager_enrollmenttoken" + ADD CONSTRAINT "PipelineManager_enrollmenttoken_pkey" PRIMARY KEY (id); + + +-- +-- Name: PipelineManager_keystore PipelineManager_keystore_pkey; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."PipelineManager_keystore" + ADD CONSTRAINT "PipelineManager_keystore_pkey" PRIMARY KEY (id); + + +-- +-- Name: PipelineManager_pipeline PipelineManager_pipeline_pkey; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."PipelineManager_pipeline" + ADD CONSTRAINT "PipelineManager_pipeline_pkey" PRIMARY KEY (id); + + +-- +-- Name: PipelineManager_policy PipelineManager_policy_name_key; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."PipelineManager_policy" + ADD CONSTRAINT "PipelineManager_policy_name_key" UNIQUE (name); + + +-- +-- Name: PipelineManager_policy PipelineManager_policy_pkey; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."PipelineManager_policy" + ADD CONSTRAINT "PipelineManager_policy_pkey" PRIMARY KEY (id); + + +-- +-- Name: PipelineManager_revision PipelineManager_revision_pkey; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."PipelineManager_revision" + ADD CONSTRAINT "PipelineManager_revision_pkey" PRIMARY KEY (id); + + +-- +-- Name: SNMP_credential SNMP_credential_name_key; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."SNMP_credential" + ADD CONSTRAINT "SNMP_credential_name_key" UNIQUE (name); + + +-- +-- Name: SNMP_credential SNMP_credential_pkey; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."SNMP_credential" + ADD CONSTRAINT "SNMP_credential_pkey" PRIMARY KEY (id); + + +-- +-- Name: SNMP_device SNMP_device_name_key; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."SNMP_device" + ADD CONSTRAINT "SNMP_device_name_key" UNIQUE (name); + + +-- +-- Name: SNMP_device SNMP_device_pkey; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."SNMP_device" + ADD CONSTRAINT "SNMP_device_pkey" PRIMARY KEY (id); + + +-- +-- Name: SNMP_devicetemplate SNMP_devicetemplate_name_key; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."SNMP_devicetemplate" + ADD CONSTRAINT "SNMP_devicetemplate_name_key" UNIQUE (name); + + +-- +-- Name: SNMP_devicetemplate SNMP_devicetemplate_official_key_key; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."SNMP_devicetemplate" + ADD CONSTRAINT "SNMP_devicetemplate_official_key_key" UNIQUE (official_key); + + +-- +-- Name: SNMP_devicetemplate SNMP_devicetemplate_pkey; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."SNMP_devicetemplate" + ADD CONSTRAINT "SNMP_devicetemplate_pkey" PRIMARY KEY (id); + + +-- +-- Name: SNMP_devicetemplate_profiles SNMP_devicetemplate_prof_devicetemplate_id_profil_fb6396a5_uniq; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."SNMP_devicetemplate_profiles" + ADD CONSTRAINT "SNMP_devicetemplate_prof_devicetemplate_id_profil_fb6396a5_uniq" UNIQUE (devicetemplate_id, profile_id); + + +-- +-- Name: SNMP_devicetemplate_profiles SNMP_devicetemplate_profiles_pkey; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."SNMP_devicetemplate_profiles" + ADD CONSTRAINT "SNMP_devicetemplate_profiles_pkey" PRIMARY KEY (id); + + +-- +-- Name: SNMP_network SNMP_network_name_key; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."SNMP_network" + ADD CONSTRAINT "SNMP_network_name_key" UNIQUE (name); + + +-- +-- Name: SNMP_network SNMP_network_pkey; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."SNMP_network" + ADD CONSTRAINT "SNMP_network_pkey" PRIMARY KEY (id); + + +-- +-- Name: SNMP_profile SNMP_profile_name_key; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."SNMP_profile" + ADD CONSTRAINT "SNMP_profile_name_key" UNIQUE (name); + + +-- +-- Name: SNMP_profile SNMP_profile_official_key_key; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."SNMP_profile" + ADD CONSTRAINT "SNMP_profile_official_key_key" UNIQUE (official_key); + + +-- +-- Name: SNMP_profile SNMP_profile_pkey; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."SNMP_profile" + ADD CONSTRAINT "SNMP_profile_pkey" PRIMARY KEY (id); + + +-- +-- Name: auth_group auth_group_name_key; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public.auth_group + ADD CONSTRAINT auth_group_name_key UNIQUE (name); + + +-- +-- Name: auth_group_permissions auth_group_permissions_group_id_permission_id_0cd325b0_uniq; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public.auth_group_permissions + ADD CONSTRAINT auth_group_permissions_group_id_permission_id_0cd325b0_uniq UNIQUE (group_id, permission_id); + + +-- +-- Name: auth_group_permissions auth_group_permissions_pkey; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public.auth_group_permissions + ADD CONSTRAINT auth_group_permissions_pkey PRIMARY KEY (id); + + +-- +-- Name: auth_group auth_group_pkey; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public.auth_group + ADD CONSTRAINT auth_group_pkey PRIMARY KEY (id); + + +-- +-- Name: auth_permission auth_permission_content_type_id_codename_01ab375a_uniq; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public.auth_permission + ADD CONSTRAINT auth_permission_content_type_id_codename_01ab375a_uniq UNIQUE (content_type_id, codename); + + +-- +-- Name: auth_permission auth_permission_pkey; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public.auth_permission + ADD CONSTRAINT auth_permission_pkey PRIMARY KEY (id); + + +-- +-- Name: auth_user_groups auth_user_groups_pkey; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public.auth_user_groups + ADD CONSTRAINT auth_user_groups_pkey PRIMARY KEY (id); + + +-- +-- Name: auth_user_groups auth_user_groups_user_id_group_id_94350c0c_uniq; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public.auth_user_groups + ADD CONSTRAINT auth_user_groups_user_id_group_id_94350c0c_uniq UNIQUE (user_id, group_id); + + +-- +-- Name: auth_user auth_user_pkey; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public.auth_user + ADD CONSTRAINT auth_user_pkey PRIMARY KEY (id); + + +-- +-- Name: auth_user_user_permissions auth_user_user_permissions_pkey; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public.auth_user_user_permissions + ADD CONSTRAINT auth_user_user_permissions_pkey PRIMARY KEY (id); + + +-- +-- Name: auth_user_user_permissions auth_user_user_permissions_user_id_permission_id_14a6b632_uniq; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public.auth_user_user_permissions + ADD CONSTRAINT auth_user_user_permissions_user_id_permission_id_14a6b632_uniq UNIQUE (user_id, permission_id); + + +-- +-- Name: auth_user auth_user_username_key; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public.auth_user + ADD CONSTRAINT auth_user_username_key UNIQUE (username); + + +-- +-- Name: django_admin_log django_admin_log_pkey; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public.django_admin_log + ADD CONSTRAINT django_admin_log_pkey PRIMARY KEY (id); + + +-- +-- Name: django_content_type django_content_type_app_label_model_76bd3d3b_uniq; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public.django_content_type + ADD CONSTRAINT django_content_type_app_label_model_76bd3d3b_uniq UNIQUE (app_label, model); + + +-- +-- Name: django_content_type django_content_type_pkey; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public.django_content_type + ADD CONSTRAINT django_content_type_pkey PRIMARY KEY (id); + + +-- +-- Name: django_migrations django_migrations_pkey; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public.django_migrations + ADD CONSTRAINT django_migrations_pkey PRIMARY KEY (id); + + +-- +-- Name: django_session django_session_pkey; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public.django_session + ADD CONSTRAINT django_session_pkey PRIMARY KEY (session_key); + + +-- +-- Name: settings settings_pkey; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public.settings + ADD CONSTRAINT settings_pkey PRIMARY KEY (id); + + +-- +-- Name: snmp_deployment_state snmp_deployment_state_pkey; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public.snmp_deployment_state + ADD CONSTRAINT snmp_deployment_state_pkey PRIMARY KEY (id); + + +-- +-- Name: PipelineManager_keystore unique_key_per_policy; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."PipelineManager_keystore" + ADD CONSTRAINT unique_key_per_policy UNIQUE (policy_id, key_name); + + +-- +-- Name: PipelineManager_pipeline unique_pipeline_per_policy; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."PipelineManager_pipeline" + ADD CONSTRAINT unique_pipeline_per_policy UNIQUE (policy_id, name); + + +-- +-- Name: PipelineManager_revision unique_revision_per_policy; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."PipelineManager_revision" + ADD CONSTRAINT unique_revision_per_policy UNIQUE (policy_id, revision_number); + + +-- +-- Name: PipelineManager_enrollmenttoken unique_token_name_per_policy; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."PipelineManager_enrollmenttoken" + ADD CONSTRAINT unique_token_name_per_policy UNIQUE (policy_id, name); + + +-- +-- Name: PipelineManager_apikey_connection_id_27156847; Type: INDEX; Schema: public; Owner: - +-- + +CREATE INDEX "PipelineManager_apikey_connection_id_27156847" ON public."PipelineManager_apikey" USING btree (connection_id); + + +-- +-- Name: PipelineManager_connection_agent_id_d5ec5cd5_like; Type: INDEX; Schema: public; Owner: - +-- + +CREATE INDEX "PipelineManager_connection_agent_id_d5ec5cd5_like" ON public."PipelineManager_connection" USING btree (agent_id varchar_pattern_ops); + + +-- +-- Name: PipelineManager_connection_policy_id_819d316d; Type: INDEX; Schema: public; Owner: - +-- + +CREATE INDEX "PipelineManager_connection_policy_id_819d316d" ON public."PipelineManager_connection" USING btree (policy_id); + + +-- +-- Name: PipelineManager_enrollmenttoken_policy_id_355d6dca; Type: INDEX; Schema: public; Owner: - +-- + +CREATE INDEX "PipelineManager_enrollmenttoken_policy_id_355d6dca" ON public."PipelineManager_enrollmenttoken" USING btree (policy_id); + + +-- +-- Name: PipelineManager_keystore_policy_id_6309b699; Type: INDEX; Schema: public; Owner: - +-- + +CREATE INDEX "PipelineManager_keystore_policy_id_6309b699" ON public."PipelineManager_keystore" USING btree (policy_id); + + +-- +-- Name: PipelineManager_pipeline_policy_id_7c3a6fd9; Type: INDEX; Schema: public; Owner: - +-- + +CREATE INDEX "PipelineManager_pipeline_policy_id_7c3a6fd9" ON public."PipelineManager_pipeline" USING btree (policy_id); + + +-- +-- Name: PipelineManager_policy_cloned_from_id_4177e74a; Type: INDEX; Schema: public; Owner: - +-- + +CREATE INDEX "PipelineManager_policy_cloned_from_id_4177e74a" ON public."PipelineManager_policy" USING btree (cloned_from_id); + + +-- +-- Name: PipelineManager_policy_name_ec6c8a56_like; Type: INDEX; Schema: public; Owner: - +-- + +CREATE INDEX "PipelineManager_policy_name_ec6c8a56_like" ON public."PipelineManager_policy" USING btree (name varchar_pattern_ops); + + +-- +-- Name: PipelineManager_revision_policy_id_99e7d146; Type: INDEX; Schema: public; Owner: - +-- + +CREATE INDEX "PipelineManager_revision_policy_id_99e7d146" ON public."PipelineManager_revision" USING btree (policy_id); + + +-- +-- Name: SNMP_credential_name_2e8e0977_like; Type: INDEX; Schema: public; Owner: - +-- + +CREATE INDEX "SNMP_credential_name_2e8e0977_like" ON public."SNMP_credential" USING btree (name varchar_pattern_ops); + + +-- +-- Name: SNMP_device_created_becb56_idx; Type: INDEX; Schema: public; Owner: - +-- + +CREATE INDEX "SNMP_device_created_becb56_idx" ON public."SNMP_device" USING btree (created_at DESC); + + +-- +-- Name: SNMP_device_credential_id_f229ff4e; Type: INDEX; Schema: public; Owner: - +-- + +CREATE INDEX "SNMP_device_credential_id_f229ff4e" ON public."SNMP_device" USING btree (credential_id); + + +-- +-- Name: SNMP_device_device_template_id_ca143332; Type: INDEX; Schema: public; Owner: - +-- + +CREATE INDEX "SNMP_device_device_template_id_ca143332" ON public."SNMP_device" USING btree (device_template_id); + + +-- +-- Name: SNMP_device_hostnam_fae88a_idx; Type: INDEX; Schema: public; Owner: - +-- + +CREATE INDEX "SNMP_device_hostnam_fae88a_idx" ON public."SNMP_device" USING btree (hostname); + + +-- +-- Name: SNMP_device_ip_addr_6a90e5_idx; Type: INDEX; Schema: public; Owner: - +-- + +CREATE INDEX "SNMP_device_ip_addr_6a90e5_idx" ON public."SNMP_device" USING btree (ip_address); + + +-- +-- Name: SNMP_device_name_087021_idx; Type: INDEX; Schema: public; Owner: - +-- + +CREATE INDEX "SNMP_device_name_087021_idx" ON public."SNMP_device" USING btree (name); + + +-- +-- Name: SNMP_device_name_e0c9483b_like; Type: INDEX; Schema: public; Owner: - +-- + +CREATE INDEX "SNMP_device_name_e0c9483b_like" ON public."SNMP_device" USING btree (name varchar_pattern_ops); + + +-- +-- Name: SNMP_device_network_889f1c_idx; Type: INDEX; Schema: public; Owner: - +-- + +CREATE INDEX "SNMP_device_network_889f1c_idx" ON public."SNMP_device" USING btree (network_id, name); + + +-- +-- Name: SNMP_device_network_id_4dea94fa; Type: INDEX; Schema: public; Owner: - +-- + +CREATE INDEX "SNMP_device_network_id_4dea94fa" ON public."SNMP_device" USING btree (network_id); + + +-- +-- Name: SNMP_devicetemplate_name_9c02299b_like; Type: INDEX; Schema: public; Owner: - +-- + +CREATE INDEX "SNMP_devicetemplate_name_9c02299b_like" ON public."SNMP_devicetemplate" USING btree (name varchar_pattern_ops); + + +-- +-- Name: SNMP_devicetemplate_official_key_65a0ae02_like; Type: INDEX; Schema: public; Owner: - +-- + +CREATE INDEX "SNMP_devicetemplate_official_key_65a0ae02_like" ON public."SNMP_devicetemplate" USING btree (official_key varchar_pattern_ops); + + +-- +-- Name: SNMP_devicetemplate_profiles_devicetemplate_id_da70425d; Type: INDEX; Schema: public; Owner: - +-- + +CREATE INDEX "SNMP_devicetemplate_profiles_devicetemplate_id_da70425d" ON public."SNMP_devicetemplate_profiles" USING btree (devicetemplate_id); + + +-- +-- Name: SNMP_devicetemplate_profiles_profile_id_4cb23513; Type: INDEX; Schema: public; Owner: - +-- + +CREATE INDEX "SNMP_devicetemplate_profiles_profile_id_4cb23513" ON public."SNMP_devicetemplate_profiles" USING btree (profile_id); + + +-- +-- Name: SNMP_network_agent_connection_id_cf646bb5; Type: INDEX; Schema: public; Owner: - +-- + +CREATE INDEX "SNMP_network_agent_connection_id_cf646bb5" ON public."SNMP_network" USING btree (agent_connection_id); + + +-- +-- Name: SNMP_network_connection_id_432a2f88; Type: INDEX; Schema: public; Owner: - +-- + +CREATE INDEX "SNMP_network_connection_id_432a2f88" ON public."SNMP_network" USING btree (connection_id); + + +-- +-- Name: SNMP_network_credential_id_3b15dfc1; Type: INDEX; Schema: public; Owner: - +-- + +CREATE INDEX "SNMP_network_credential_id_3b15dfc1" ON public."SNMP_network" USING btree (credential_id); + + +-- +-- Name: SNMP_network_discovery_credential_id_d9229589; Type: INDEX; Schema: public; Owner: - +-- + +CREATE INDEX "SNMP_network_discovery_credential_id_d9229589" ON public."SNMP_network" USING btree (discovery_credential_id); + + +-- +-- Name: SNMP_network_name_83546472_like; Type: INDEX; Schema: public; Owner: - +-- + +CREATE INDEX "SNMP_network_name_83546472_like" ON public."SNMP_network" USING btree (name varchar_pattern_ops); + + +-- +-- Name: SNMP_profile_name_8a9d3c9d_like; Type: INDEX; Schema: public; Owner: - +-- + +CREATE INDEX "SNMP_profile_name_8a9d3c9d_like" ON public."SNMP_profile" USING btree (name varchar_pattern_ops); + + +-- +-- Name: SNMP_profile_official_key_de922dd3_like; Type: INDEX; Schema: public; Owner: - +-- + +CREATE INDEX "SNMP_profile_official_key_de922dd3_like" ON public."SNMP_profile" USING btree (official_key varchar_pattern_ops); + + +-- +-- Name: auth_group_name_a6ea08ec_like; Type: INDEX; Schema: public; Owner: - +-- + +CREATE INDEX auth_group_name_a6ea08ec_like ON public.auth_group USING btree (name varchar_pattern_ops); + + +-- +-- Name: auth_group_permissions_group_id_b120cbf9; Type: INDEX; Schema: public; Owner: - +-- + +CREATE INDEX auth_group_permissions_group_id_b120cbf9 ON public.auth_group_permissions USING btree (group_id); + + +-- +-- Name: auth_group_permissions_permission_id_84c5c92e; Type: INDEX; Schema: public; Owner: - +-- + +CREATE INDEX auth_group_permissions_permission_id_84c5c92e ON public.auth_group_permissions USING btree (permission_id); + + +-- +-- Name: auth_permission_content_type_id_2f476e4b; Type: INDEX; Schema: public; Owner: - +-- + +CREATE INDEX auth_permission_content_type_id_2f476e4b ON public.auth_permission USING btree (content_type_id); + + +-- +-- Name: auth_user_groups_group_id_97559544; Type: INDEX; Schema: public; Owner: - +-- + +CREATE INDEX auth_user_groups_group_id_97559544 ON public.auth_user_groups USING btree (group_id); + + +-- +-- Name: auth_user_groups_user_id_6a12ed8b; Type: INDEX; Schema: public; Owner: - +-- + +CREATE INDEX auth_user_groups_user_id_6a12ed8b ON public.auth_user_groups USING btree (user_id); + + +-- +-- Name: auth_user_user_permissions_permission_id_1fbb5f2c; Type: INDEX; Schema: public; Owner: - +-- + +CREATE INDEX auth_user_user_permissions_permission_id_1fbb5f2c ON public.auth_user_user_permissions USING btree (permission_id); + + +-- +-- Name: auth_user_user_permissions_user_id_a95ead1b; Type: INDEX; Schema: public; Owner: - +-- + +CREATE INDEX auth_user_user_permissions_user_id_a95ead1b ON public.auth_user_user_permissions USING btree (user_id); + + +-- +-- Name: auth_user_username_6821ab7c_like; Type: INDEX; Schema: public; Owner: - +-- + +CREATE INDEX auth_user_username_6821ab7c_like ON public.auth_user USING btree (username varchar_pattern_ops); + + +-- +-- Name: django_admin_log_content_type_id_c4bce8eb; Type: INDEX; Schema: public; Owner: - +-- + +CREATE INDEX django_admin_log_content_type_id_c4bce8eb ON public.django_admin_log USING btree (content_type_id); + + +-- +-- Name: django_admin_log_user_id_c564eba6; Type: INDEX; Schema: public; Owner: - +-- + +CREATE INDEX django_admin_log_user_id_c564eba6 ON public.django_admin_log USING btree (user_id); + + +-- +-- Name: django_session_expire_date_a5c62663; Type: INDEX; Schema: public; Owner: - +-- + +CREATE INDEX django_session_expire_date_a5c62663 ON public.django_session USING btree (expire_date); + + +-- +-- Name: django_session_session_key_c0390e0f_like; Type: INDEX; Schema: public; Owner: - +-- + +CREATE INDEX django_session_session_key_c0390e0f_like ON public.django_session USING btree (session_key varchar_pattern_ops); + + +-- +-- Name: Management_userprofile Management_userprofile_user_id_70f1a900_fk_auth_user_id; Type: FK CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."Management_userprofile" + ADD CONSTRAINT "Management_userprofile_user_id_70f1a900_fk_auth_user_id" FOREIGN KEY (user_id) REFERENCES public.auth_user(id) DEFERRABLE INITIALLY DEFERRED; + + +-- +-- Name: PipelineManager_apikey PipelineManager_apik_connection_id_27156847_fk_PipelineM; Type: FK CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."PipelineManager_apikey" + ADD CONSTRAINT "PipelineManager_apik_connection_id_27156847_fk_PipelineM" FOREIGN KEY (connection_id) REFERENCES public."PipelineManager_connection"(id) DEFERRABLE INITIALLY DEFERRED; + + +-- +-- Name: PipelineManager_connection PipelineManager_conn_policy_id_819d316d_fk_PipelineM; Type: FK CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."PipelineManager_connection" + ADD CONSTRAINT "PipelineManager_conn_policy_id_819d316d_fk_PipelineM" FOREIGN KEY (policy_id) REFERENCES public."PipelineManager_policy"(id) DEFERRABLE INITIALLY DEFERRED; + + +-- +-- Name: PipelineManager_enrollmenttoken PipelineManager_enro_policy_id_355d6dca_fk_PipelineM; Type: FK CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."PipelineManager_enrollmenttoken" + ADD CONSTRAINT "PipelineManager_enro_policy_id_355d6dca_fk_PipelineM" FOREIGN KEY (policy_id) REFERENCES public."PipelineManager_policy"(id) DEFERRABLE INITIALLY DEFERRED; + + +-- +-- Name: PipelineManager_keystore PipelineManager_keys_policy_id_6309b699_fk_PipelineM; Type: FK CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."PipelineManager_keystore" + ADD CONSTRAINT "PipelineManager_keys_policy_id_6309b699_fk_PipelineM" FOREIGN KEY (policy_id) REFERENCES public."PipelineManager_policy"(id) DEFERRABLE INITIALLY DEFERRED; + + +-- +-- Name: PipelineManager_pipeline PipelineManager_pipe_policy_id_7c3a6fd9_fk_PipelineM; Type: FK CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."PipelineManager_pipeline" + ADD CONSTRAINT "PipelineManager_pipe_policy_id_7c3a6fd9_fk_PipelineM" FOREIGN KEY (policy_id) REFERENCES public."PipelineManager_policy"(id) DEFERRABLE INITIALLY DEFERRED; + + +-- +-- Name: PipelineManager_policy PipelineManager_poli_cloned_from_id_4177e74a_fk_PipelineM; Type: FK CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."PipelineManager_policy" + ADD CONSTRAINT "PipelineManager_poli_cloned_from_id_4177e74a_fk_PipelineM" FOREIGN KEY (cloned_from_id) REFERENCES public."PipelineManager_policy"(id) DEFERRABLE INITIALLY DEFERRED; + + +-- +-- Name: PipelineManager_revision PipelineManager_revi_policy_id_99e7d146_fk_PipelineM; Type: FK CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."PipelineManager_revision" + ADD CONSTRAINT "PipelineManager_revi_policy_id_99e7d146_fk_PipelineM" FOREIGN KEY (policy_id) REFERENCES public."PipelineManager_policy"(id) DEFERRABLE INITIALLY DEFERRED; + + +-- +-- Name: SNMP_device SNMP_device_credential_id_f229ff4e_fk_SNMP_credential_id; Type: FK CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."SNMP_device" + ADD CONSTRAINT "SNMP_device_credential_id_f229ff4e_fk_SNMP_credential_id" FOREIGN KEY (credential_id) REFERENCES public."SNMP_credential"(id) DEFERRABLE INITIALLY DEFERRED; + + +-- +-- Name: SNMP_device SNMP_device_device_template_id_ca143332_fk_SNMP_devi; Type: FK CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."SNMP_device" + ADD CONSTRAINT "SNMP_device_device_template_id_ca143332_fk_SNMP_devi" FOREIGN KEY (device_template_id) REFERENCES public."SNMP_devicetemplate"(id) DEFERRABLE INITIALLY DEFERRED; + + +-- +-- Name: SNMP_device SNMP_device_network_id_4dea94fa_fk_SNMP_network_id; Type: FK CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."SNMP_device" + ADD CONSTRAINT "SNMP_device_network_id_4dea94fa_fk_SNMP_network_id" FOREIGN KEY (network_id) REFERENCES public."SNMP_network"(id) DEFERRABLE INITIALLY DEFERRED; + + +-- +-- Name: SNMP_devicetemplate_profiles SNMP_devicetemplate__devicetemplate_id_da70425d_fk_SNMP_devi; Type: FK CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."SNMP_devicetemplate_profiles" + ADD CONSTRAINT "SNMP_devicetemplate__devicetemplate_id_da70425d_fk_SNMP_devi" FOREIGN KEY (devicetemplate_id) REFERENCES public."SNMP_devicetemplate"(id) DEFERRABLE INITIALLY DEFERRED; + + +-- +-- Name: SNMP_devicetemplate_profiles SNMP_devicetemplate__profile_id_4cb23513_fk_SNMP_prof; Type: FK CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."SNMP_devicetemplate_profiles" + ADD CONSTRAINT "SNMP_devicetemplate__profile_id_4cb23513_fk_SNMP_prof" FOREIGN KEY (profile_id) REFERENCES public."SNMP_profile"(id) DEFERRABLE INITIALLY DEFERRED; + + +-- +-- Name: SNMP_network SNMP_network_agent_connection_id_cf646bb5_fk_PipelineM; Type: FK CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."SNMP_network" + ADD CONSTRAINT "SNMP_network_agent_connection_id_cf646bb5_fk_PipelineM" FOREIGN KEY (agent_connection_id) REFERENCES public."PipelineManager_connection"(id) DEFERRABLE INITIALLY DEFERRED; + + +-- +-- Name: SNMP_network SNMP_network_connection_id_432a2f88_fk_PipelineM; Type: FK CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."SNMP_network" + ADD CONSTRAINT "SNMP_network_connection_id_432a2f88_fk_PipelineM" FOREIGN KEY (connection_id) REFERENCES public."PipelineManager_connection"(id) DEFERRABLE INITIALLY DEFERRED; + + +-- +-- Name: SNMP_network SNMP_network_credential_id_3b15dfc1_fk_SNMP_credential_id; Type: FK CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."SNMP_network" + ADD CONSTRAINT "SNMP_network_credential_id_3b15dfc1_fk_SNMP_credential_id" FOREIGN KEY (credential_id) REFERENCES public."SNMP_credential"(id) DEFERRABLE INITIALLY DEFERRED; + + +-- +-- Name: SNMP_network SNMP_network_discovery_credential_d9229589_fk_SNMP_cred; Type: FK CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."SNMP_network" + ADD CONSTRAINT "SNMP_network_discovery_credential_d9229589_fk_SNMP_cred" FOREIGN KEY (discovery_credential_id) REFERENCES public."SNMP_credential"(id) DEFERRABLE INITIALLY DEFERRED; + + +-- +-- Name: auth_group_permissions auth_group_permissio_permission_id_84c5c92e_fk_auth_perm; Type: FK CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public.auth_group_permissions + ADD CONSTRAINT auth_group_permissio_permission_id_84c5c92e_fk_auth_perm FOREIGN KEY (permission_id) REFERENCES public.auth_permission(id) DEFERRABLE INITIALLY DEFERRED; + + +-- +-- Name: auth_group_permissions auth_group_permissions_group_id_b120cbf9_fk_auth_group_id; Type: FK CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public.auth_group_permissions + ADD CONSTRAINT auth_group_permissions_group_id_b120cbf9_fk_auth_group_id FOREIGN KEY (group_id) REFERENCES public.auth_group(id) DEFERRABLE INITIALLY DEFERRED; + + +-- +-- Name: auth_permission auth_permission_content_type_id_2f476e4b_fk_django_co; Type: FK CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public.auth_permission + ADD CONSTRAINT auth_permission_content_type_id_2f476e4b_fk_django_co FOREIGN KEY (content_type_id) REFERENCES public.django_content_type(id) DEFERRABLE INITIALLY DEFERRED; + + +-- +-- Name: auth_user_groups auth_user_groups_group_id_97559544_fk_auth_group_id; Type: FK CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public.auth_user_groups + ADD CONSTRAINT auth_user_groups_group_id_97559544_fk_auth_group_id FOREIGN KEY (group_id) REFERENCES public.auth_group(id) DEFERRABLE INITIALLY DEFERRED; + + +-- +-- Name: auth_user_groups auth_user_groups_user_id_6a12ed8b_fk_auth_user_id; Type: FK CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public.auth_user_groups + ADD CONSTRAINT auth_user_groups_user_id_6a12ed8b_fk_auth_user_id FOREIGN KEY (user_id) REFERENCES public.auth_user(id) DEFERRABLE INITIALLY DEFERRED; + + +-- +-- Name: auth_user_user_permissions auth_user_user_permi_permission_id_1fbb5f2c_fk_auth_perm; Type: FK CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public.auth_user_user_permissions + ADD CONSTRAINT auth_user_user_permi_permission_id_1fbb5f2c_fk_auth_perm FOREIGN KEY (permission_id) REFERENCES public.auth_permission(id) DEFERRABLE INITIALLY DEFERRED; + + +-- +-- Name: auth_user_user_permissions auth_user_user_permissions_user_id_a95ead1b_fk_auth_user_id; Type: FK CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public.auth_user_user_permissions + ADD CONSTRAINT auth_user_user_permissions_user_id_a95ead1b_fk_auth_user_id FOREIGN KEY (user_id) REFERENCES public.auth_user(id) DEFERRABLE INITIALLY DEFERRED; + + +-- +-- Name: django_admin_log django_admin_log_content_type_id_c4bce8eb_fk_django_co; Type: FK CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public.django_admin_log + ADD CONSTRAINT django_admin_log_content_type_id_c4bce8eb_fk_django_co FOREIGN KEY (content_type_id) REFERENCES public.django_content_type(id) DEFERRABLE INITIALLY DEFERRED; + + +-- +-- Name: django_admin_log django_admin_log_user_id_c564eba6_fk_auth_user_id; Type: FK CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public.django_admin_log + ADD CONSTRAINT django_admin_log_user_id_c564eba6_fk_auth_user_id FOREIGN KEY (user_id) REFERENCES public.auth_user(id) DEFERRABLE INITIALLY DEFERRED; + + +-- +-- PostgreSQL database dump complete +-- + + diff --git a/docs/docs/logstashui/database/index.md b/docs/docs/logstashui/database/index.md new file mode 100644 index 0000000..b4af631 --- /dev/null +++ b/docs/docs/logstashui/database/index.md @@ -0,0 +1,79 @@ +# Database + +LogstashUI stores operator data (policies, connections, pipelines, SNMP, users) in a SQL database. The Django ORM is the only CRUD layer. Choose the engine with discrete environment variables. There is no `DATABASE_URL` and no YAML. + +| Engine | `LOGSTASHUI_DB_ENGINE` | Server floor | When to use | +|---|---|---|---| +| **SQLite** (default) | `sqlite` (aliases: `sqlite3`, empty) | — | Single replica, lab, default Docker | +| **PostgreSQL** | `postgresql` (alias: `postgres`) | 14+ | Concurrent agents, Kubernetes | +| **MariaDB / MySQL** | `mysql` (aliases: `mariadb`, `my`) | MariaDB 10.6+ / MySQL 8.0+ | Same as Postgres; one engine covers both | + +`logstashui serve` logs a **warning** when the engine is SQLite and `LOGSTASHUI_WORKERS` > 1. It still starts. SQLite does not scale under gunicorn/gevent with concurrent agents. + +The container image already installs `LogstashUI[databases]` (both drivers). Native pip/uv: + +```bash +uv pip install 'LogstashUI[postgres]' +uv pip install 'LogstashUI[mysql]' +uv pip install 'LogstashUI[databases]' # both +``` + +Missing driver fails at startup with the extra name. Unknown engine, missing `HOST`/`USER` on server engines, or a server below the floor fails **before** `migrate`. + +Create the empty server database yourself ([examples](examples/)). Tables come from `logstashui manage migrate` (also run by `logstashui serve`). Do not apply the schema snapshots instead of migrate. + +MySQL/MariaDB **must** be `utf8mb4` / `utf8mb4_bin` so unique names match SQLite/Postgres case-sensitivity. The Django connection also sets `NAMES utf8mb4 COLLATE utf8mb4_bin`. + +--- + +## Environment variables + +Unset keys use the default in the **Default** column. Empty string after strip is treated as unset for name/port/sslmode. + +### Data directory (still required when the database is remote) + +| Variable | Default | Notes | +|---|---|---| +| `LOGSTASHUI_DATA_DIR` | `$(pwd)/logstashui_data` | Docker/K8s/systemd **must** set `/var/lib/logstashui`. TLS, `.django_secret_key`, logs, `staticfiles/`, SQLite file. | +| `LOGSTASHUI_LOGS_DIR` | `$LOGSTASHUI_DATA_DIR/logs` | | + +### Engine and connection + +| Variable | Default | Notes | +|---|---|---| +| `LOGSTASHUI_DB_ENGINE` | `sqlite` | `sqlite`, `postgresql`, `mysql`. Aliases: `sqlite3`, `postgres`, `mariadb`, `my`. | +| `LOGSTASHUI_DB_NAME` | sqlite: `$LOGSTASHUI_DATA_DIR/db.sqlite3`; else `logstashui` | Database name, or SQLite file path. | +| `LOGSTASHUI_DB_HOST` | empty | **Required** for postgresql/mysql. | +| `LOGSTASHUI_DB_PORT` | postgresql: `5432`; mysql: `3306` | | +| `LOGSTASHUI_DB_USER` | empty | **Required** for postgresql/mysql. | +| `LOGSTASHUI_DB_PASSWORD` | empty | Secret / `chmod 640` EnvironmentFile. Never in a ConfigMap. | +| `LOGSTASHUI_DB_SSLMODE` | postgresql: `prefer` | `disable` `allow` `prefer` `require` `verify-ca` `verify-full`. Ignored for mysql/sqlite. | +| `LOGSTASHUI_DB_SSL_CA` | empty | CA file. Postgres `sslrootcert` when `verify-*`; MySQL `ssl.ca`. | +| `LOGSTASHUI_DB_CONN_MAX_AGE` | `60` | Seconds. `0` closes per request. Non-integer → startup error. | +| `LOGSTASHUI_DB_CONN_HEALTH_CHECKS` | `true` | Django `CONN_HEALTH_CHECKS`. | + +SQLite opens WAL (`PRAGMA journal_mode=WAL`) and `busy_timeout=20000`. + +gunicorn stays `--worker-class gevent` (`--worker-connections 1000`). Keep `LOGSTASHUI_WORKERS` × in-flight requests under the server `max_connections`. PgBouncer is optional. + +Related server knobs (not DB-specific, but they interact): + +| Variable | Default | Notes | +|---|---|---| +| `LOGSTASHUI_WORKERS` | `2` | SQLite + workers > 1 logs a warning. | +| `LOGSTASHUI_TLS` | `true` | Keep `true` in Kubernetes. | +| `SECRET_KEY` | auto in `DATA_DIR/.django_secret_key` | Keep `DATA_DIR` across engine switches or encrypted rows will not decrypt. | + +--- + +## Kubernetes + +PVC at `/var/lib/logstashui` for every engine. ConfigMap: `LOGSTASHUI_DB_ENGINE` / `HOST` / `PORT` / `NAME` / `USER` (and `SSLMODE` for Postgres). Secret: `LOGSTASHUI_DB_PASSWORD`. CloudNativePG: [CNPG](/docs/docs/logstashui/kubernetes/cnpg.md). + +Manifests: [kubernetes/examples](/docs/docs/logstashui/kubernetes/examples/). + +--- + +## Moving data + +[Migration](migration.md) — offline `dumpdata`/`loaddata` (supported) and BETA `logstashui migrate-engine`. diff --git a/docs/docs/logstashui/database/migration.md b/docs/docs/logstashui/database/migration.md new file mode 100644 index 0000000..e5f4e2c --- /dev/null +++ b/docs/docs/logstashui/database/migration.md @@ -0,0 +1,75 @@ +# Migrating the database + +Two paths copy **SQLite** data onto PostgreSQL or MySQL/MariaDB. Both require a **stopped** UI so nothing writes during dump. Keep `$LOGSTASHUI_DATA_DIR` (same Django secret key) or encrypted keystore rows will not decrypt. Sessions are not copied; log in again. + +Create the empty target first ([examples](examples/)). MySQL/MariaDB: `utf8mb4` / `utf8mb4_bin`. Native installs need `LogstashUI[postgres]` or `LogstashUI[mysql]`. The container image already has both drivers. + +BETA `migrate-engine` is **not atomic** on the target: `dumpdata` → `migrate` → `loaddata` is three steps. If `loaddata` fails, drop or recreate the target and re-run. SQLite is only WAL-checkpointed. + +--- + +## Offline dump / load (supported) + +Do this while `LOGSTASHUI_DB_ENGINE` is still sqlite (or unset). If you point env at the empty server first, `dumpdata` dumps the empty server. + +1. Stop the UI. + - systemd: `sudo systemctl stop logstashui` (avoid `Restart=` racing SIGTERM). + - Docker: stop the container (`docker compose stop logstashui`). + - Kubernetes: `kubectl -n logstashui scale statefulset/logstashui --replicas=0` and wait for the pod to disappear. +2. Copy `$LOGSTASHUI_DATA_DIR/db.sqlite3` (and `-wal` / `-shm` if present) somewhere safe. Keep the rest of `DATA_DIR`. +3. Dump from SQLite: + + ```bash + logstashui manage dumpdata --natural-foreign --natural-primary \ + -e contenttypes -e auth.permission -e sessions \ + -o dump.json + ``` + + In a container: `docker exec -it logstashui manage dumpdata ...` **before** changing env, or exec with `LOGSTASHUI_DB_ENGINE=sqlite` and `LOGSTASHUI_DB_NAME` pointing at the sqlite file. + +4. Create the server database (`utf8mb4_bin` on MySQL/MariaDB). +5. Set `LOGSTASHUI_DB_*` for the target (`ENGINE`, `HOST`, `PORT`, `NAME`, `USER`, `PASSWORD`, plus `SSLMODE` / `SSL_CA` as needed). +6. Schema + load: + + ```bash + logstashui manage migrate --noinput + logstashui manage loaddata dump.json + ``` + +7. Postgres sequences: this path does **not** reset them. Use BETA `migrate-engine` if you need `sequence_reset_sql` without a `psql` client. Django will still insert; sequences can drift until reset. +8. Start LogstashUI. Log in again. + +Kubernetes after env change: patch ConfigMap/Secret, then `scale --replicas=1` (or delete the pod). First start runs `migrate` again (no-op if already applied) — **loaddata is not automatic**. Run loaddata from a one-shot `kubectl exec` after migrate, or run steps 6–7 from a job with the same env and a copy of `dump.json`. + +--- + +## BETA CLI + +```bash +# Target LOGSTASHUI_DB_* already set; sqlite file still in DATA_DIR +sudo systemctl stop logstashui +logstashui migrate-engine --to postgresql --i-have-a-backup +# --to mysql | --to mariadb (mariadb is an alias of mysql) +# --write-env /etc/default/logstashui # upserts engine/host/name/user; never password +sudo systemctl start logstashui +``` + +`--i-have-a-backup` is required. + +The command: + +1. SIGTERMs gunicorn if `$LOGSTASHUI_DATA_DIR/gunicorn.pid` is live (does not restart serve). +2. WAL checkpoint on the sqlite file in `DATA_DIR`. +3. `dumpdata` from that sqlite file (ignores target `LOGSTASHUI_DB_*` for the dump). +4. `migrate --noinput` on the target, then `loaddata`. +5. PostgreSQL: `sequence_reset_sql` through the Django connection (no `psql`). + +Kubernetes: scale to 0, `kubectl exec` is the wrong place if the pod is gone. Run `migrate-engine` from a debug pod / CI job that mounts the same PVC (`DATA_DIR`) and has target `LOGSTASHUI_DB_*`. Then scale to 1. + +If `loaddata` fails: drop or recreate the **target** database and re-run. Do not restore SQLite unless you also lost the backup. + +--- + +## What is not copied + +`contenttypes`, `auth.permission`, and `sessions` are excluded from dump/load. Permissions recreate on migrate. Sessions mean everyone logs in again. diff --git a/docs/docs/logstashui/general/deploy.md b/docs/docs/logstashui/general/deploy.md index f0fb1bb..2b4798d 100644 --- a/docs/docs/logstashui/general/deploy.md +++ b/docs/docs/logstashui/general/deploy.md @@ -9,6 +9,7 @@ The ways to deploy LogstashUI, from the standard Docker install to running from - [Option 2: Host-backed Simulation](#option-2-host-backed-simulation) - [Option 3: pip / uv + systemd](#option-3-pip--uv--systemd) - [Option 4: Source Development Setup](#option-4-source-development-setup) +- [Option 5: Kubernetes](#option-5-kubernetes) --- @@ -37,7 +38,7 @@ This is **embedded mode** — two containers: **LogstashUI** (gunicorn HTTPS on On **Linux Docker**, that bind-mount keeps host file ownership. The image entrypoint starts as root, chowns **only** the data directory if it is not writable, then drops to `PUID`/`PGID` (from `start_logstashui.sh`: your uid/gid) or image user **appuser (10001)**. Gunicorn never stays root. Docker Desktop (macOS/Windows) usually maps UIDs already; the same path is a no-op chown. If the directory is still unwritable, startup exits before migrate so sqlite/TLS are not created as the wrong user. -**Kubernetes:** use a PVC at `/var/lib/logstashui`, `runAsUser: 10001`, `runAsNonRoot: true`, and `fsGroup: 10001`. The entrypoint skips chown when it is not root. ConfigMap `LOGSTASHUI_DB_ENGINE` / `LOGSTASHUI_DB_HOST` / `LOGSTASHUI_DB_NAME` / `LOGSTASHUI_DB_USER`; Secret `LOGSTASHUI_DB_PASSWORD`. The PVC is still required when the database is external. +**Kubernetes:** see [Option 5](#option-5-kubernetes) and the [Kubernetes subsection](/docs/docs/logstashui/kubernetes/index.md). PVC at `/var/lib/logstashui`, `runAsUser: 10001`, `runAsNonRoot: true`, `fsGroup: 10001`. Keep `LOGSTASHUI_TLS` on; the Ingress/HTTPRoute originates HTTPS to `:8443` and skips backend cert verify. **HTTPS / product CA:** On first start, LogstashUI writes a product CA and a UI server certificate under `$LOGSTASHUI_DATA_DIR/tls/` (`ui-server.crt` / `ui-server.key`). Gunicorn presents that cert on port **8443** (ports under 1000 would need root). The product leaf SANs include `localhost`, `logstashui`, **all non-loopback host IPs**, and **PTR reverse-DNS FQDNs** for those IPs when available (injected by `start_logstashui.sh` as `LOGSTASHUI_HOST_*` / `LOGSTASHUI_TLS_SANS`, because the container cannot see the host LAN addresses by itself). Bare short hostnames (common on macOS) are replaced by reverse-lookup FQDNs when PTR records exist. Changing the Agent callback URL or those env SANs **re-issues** the product leaf on next startup (or Settings save); restart the UI container so gunicorn reloads the file. Agents: @@ -72,7 +73,7 @@ sudo systemctl enable --now logstashui Set `LOGSTASHUI_DATA_DIR=/var/lib/logstashui` in `/etc/default/logstashui` (the generator does this). See [environment configuration](/docs/docs/logstashui/configuration/environment.md). -Kubernetes: same image as Option 1, env/ConfigMap (`LOGSTASHUI_DB_ENGINE` / `HOST` / `NAME` / `USER`) plus Secret (`LOGSTASHUI_DB_PASSWORD`), PVC at `/var/lib/logstashui` (still required when the database is external). No YAML mount. +Kubernetes: [Option 5](#option-5-kubernetes). Same image as Option 1; env/ConfigMap plus Secret; PVC at `/var/lib/logstashui`. Keep TLS on. No YAML mount. --- @@ -86,9 +87,23 @@ Use this when you want to run LogstashUI directly from source for development or --- +## Option 5: Kubernetes + +One-replica StatefulSet, PVC at `/var/lib/logstashui`, image `codyjackson032/logstashui:latest` (or a tag you built). Gunicorn keeps HTTPS on **8443** (`LOGSTASHUI_TLS` stays true). Ingress-nginx or Envoy Gateway originates HTTPS to the pod and skips verification of the product self-signed leaf. + +- Guide: [Kubernetes](/docs/docs/logstashui/kubernetes/index.md) +- Manifests: [examples](/docs/docs/logstashui/kubernetes/examples/README.md) (SQLite, PostgreSQL, MySQL/MariaDB) +- CloudNativePG: [cnpg.md](/docs/docs/logstashui/kubernetes/cnpg.md) +- Envoy Gateway Backend API: [envoy-gateway.md](/docs/docs/logstashui/kubernetes/envoy-gateway.md) +- Database env and migration: [Database](/docs/docs/logstashui/database/index.md) + +--- + ## Related Documentation - **[Getting Started](/docs/docs/getting_started.md)** - Step-by-step standard install +- **[Kubernetes](/docs/docs/logstashui/kubernetes/index.md)** - StatefulSet, PVC, Ingress, Envoy Gateway, CNPG +- **[Database](/docs/docs/logstashui/database/index.md)** - SQLite, PostgreSQL, MySQL/MariaDB - **[Building LogstashUI from Source](/docs/docs/logstashui/general/build.md)** - Source builds and local development - **[Host Mode Setup](/docs/docs/logstashui/configuration/host_mode.md)** - High-performance simulation setup - **[Updating LogstashUI](/docs/docs/logstashui/general/updating.md)** - Keeping your deployment current diff --git a/docs/docs/logstashui/general/index.md b/docs/docs/logstashui/general/index.md index e227fc2..11227c6 100644 --- a/docs/docs/logstashui/general/index.md +++ b/docs/docs/logstashui/general/index.md @@ -11,6 +11,8 @@ All the ways to deploy LogstashUI. **Covers:** - Standard Docker deployment (recommended) - Host-backed simulation +- pip / uv + systemd +- Kubernetes (see also the [Kubernetes](/docs/docs/logstashui/kubernetes/index.md) subsection) - Source development setup **📖 [View deployment guide →](/docs/docs/logstashui/general/deploy.md)** diff --git a/docs/docs/logstashui/index.md b/docs/docs/logstashui/index.md index a3d8df3..2f15ebb 100644 --- a/docs/docs/logstashui/index.md +++ b/docs/docs/logstashui/index.md @@ -42,6 +42,8 @@ Configure polling, traps, and discovery through a web interface. - **[Architecture](/docs/docs/logstashui/architecture.md)** - System architecture - **[Compatibility](/docs/docs/logstashui/compatibility.md)** - Logstash version compatibility and requirements - **[Configuration](/docs/docs/logstashui/configuration/index.md)** - Configuration options and settings for LogstashUI +- **[Database](/docs/docs/logstashui/database/index.md)** - SQLite, PostgreSQL, MySQL/MariaDB, and migration +- **[Kubernetes](/docs/docs/logstashui/kubernetes/index.md)** - StatefulSet, PVC, Ingress, Envoy Gateway, CloudNativePG - **[SNMP Monitoring](/docs/docs/logstashui/SNMP/index.md)** - Network monitoring with SNMP polling, traps, and discovery - **[General](/docs/docs/logstashui/general/index.md)** - Build, update, and deployment guides diff --git a/docs/docs/logstashui/kubernetes/cnpg.md b/docs/docs/logstashui/kubernetes/cnpg.md new file mode 100644 index 0000000..cebf61f --- /dev/null +++ b/docs/docs/logstashui/kubernetes/cnpg.md @@ -0,0 +1,98 @@ +# CloudNativePG + +Use this when a [CloudNativePG](https://cloudnative-pg.io/) operator is already installed (commonly in namespace `cnpg-system`). Create the Postgres **Cluster in the LogstashUI namespace**. LogstashUI talks to the Cluster's read-write Service; it does not talk to the operator. + +PostgreSQL **14+** is required (16 is a good default). The container image already has `psycopg`. + +A sample Cluster is in [examples/postgresql/cnpg.yaml](examples/postgresql/cnpg.yaml). + +--- + +## Cluster + +```yaml +apiVersion: postgresql.cnpg.io/v1 +kind: Cluster +metadata: + name: logstashui-pg + namespace: logstashui +spec: + instances: 1 # 3 for HA + imageName: ghcr.io/cloudnative-pg/postgresql:16 + bootstrap: + initdb: + database: logstashui + owner: logstashui + encoding: UTF8 + storage: + size: 10Gi +``` + +Do **not** put Django `CREATE TABLE` in `postInitApplicationSQL`. `logstashui serve` runs `migrate` on first start. + +Wait until Ready: + +```bash +kubectl -n logstashui wait --for=condition=Ready cluster/logstashui-pg --timeout=300s +``` + +--- + +## Connection (what CNPG creates) + +| Object | Purpose | +|---|---| +| Service `logstashui-pg-rw` | Primary, port 5432 | +| Secret `logstashui-pg-app` | `host`, `port`, `dbname`, `username`, `password`, `uri` | +| Secret `logstashui-pg-ca` | Cluster CA (`ca.crt`) if you want `verify-full` | + +Wire LogstashUI from that Secret (do not copy the password into your own Secret): + +```yaml +env: + - name: LOGSTASHUI_DB_ENGINE + value: postgresql + - name: LOGSTASHUI_DB_SSLMODE + value: require + - name: LOGSTASHUI_DB_HOST + valueFrom: + secretKeyRef: + name: logstashui-pg-app + key: host + - name: LOGSTASHUI_DB_PORT + valueFrom: + secretKeyRef: + name: logstashui-pg-app + key: port + - name: LOGSTASHUI_DB_NAME + valueFrom: + secretKeyRef: + name: logstashui-pg-app + key: dbname + - name: LOGSTASHUI_DB_USER + valueFrom: + secretKeyRef: + name: logstashui-pg-app + key: username + - name: LOGSTASHUI_DB_PASSWORD + valueFrom: + secretKeyRef: + name: logstashui-pg-app + key: password +``` + +`require` is enough for CNPG's server TLS. For `verify-full`, mount `logstashui-pg-ca` `ca.crt` and set `LOGSTASHUI_DB_SSL_CA` to that path. + +Keep `LOGSTASHUI_DATA_DIR` on a PVC (TLS and secrets). An init container can `pg_isready -h logstashui-pg-rw` before the UI starts. + +--- + +## Existing Cluster in another namespace + +Point `LOGSTASHUI_DB_HOST` at that Cluster's `-rw` Service FQDN (`-rw..svc.cluster.local`). Copy or ExternalSecret the app password into `logstashui`. NetworkPolicy must allow 5432 from the UI pods. + +--- + +## SQLite → CNPG + +Create the Cluster and empty database, set `LOGSTASHUI_DB_*`, then follow [migration](/docs/docs/logstashui/database/migration.md). Do not `loaddata` until `migrate` has created tables. diff --git a/docs/docs/logstashui/kubernetes/envoy-gateway.md b/docs/docs/logstashui/kubernetes/envoy-gateway.md new file mode 100644 index 0000000..5e87404 --- /dev/null +++ b/docs/docs/logstashui/kubernetes/envoy-gateway.md @@ -0,0 +1,112 @@ +# Envoy Gateway + +Keep LogstashUI on HTTPS `:8443`. The HTTPRoute must send **HTTPS** to the pod and **skip verification** of the product self-signed leaf. + +Envoy Gateway documents that skip on the **`Backend` CR**, not on `BackendTrafficPolicy`. `BackendTrafficPolicy` is retries, timeouts, and circuit breaking. `tls.insecureSkipVerify` exists only on `Backend`. + +The Backend API is **disabled by default**. Enable it, then apply a `Backend` + `HTTPRoute`. + +--- + +## 1. Enable the Backend API + +Helm values: + +```yaml +config: + envoyGateway: + extensionApis: + enableBackend: true +``` + +Upgrade: + +```bash +helm upgrade eg oci://docker.io/envoyproxy/gateway-helm \ + -n envoy-gateway-system \ + --reuse-values \ + --set config.envoyGateway.extensionApis.enableBackend=true +``` + +If you manage an `EnvoyGateway` config ConfigMap instead of Helm values, merge: + +```yaml +apiVersion: gateway.envoyproxy.io/v1alpha1 +kind: EnvoyGateway +gateway: + controllerName: gateway.envoyproxy.io/gatewayclass-controller +extensionApis: + enableBackend: true +``` + +Restart the Envoy Gateway controller after the change. Confirm the `Backend` CRD is served: + +```bash +kubectl api-resources | grep -i backend +``` + +--- + +## 2. HTTPRoute + Backend + +The HTTPRoute `backendRefs` must use `group: gateway.envoyproxy.io` and `kind: Backend`. Point the `Backend` at the LogstashUI Service FQDN on port **8443**. + +```yaml +apiVersion: gateway.envoyproxy.io/v1alpha1 +kind: Backend +metadata: + name: logstashui + namespace: logstashui +spec: + endpoints: + - fqdn: + hostname: logstashui.logstashui.svc.cluster.local + port: 8443 + tls: + insecureSkipVerify: true +--- +apiVersion: gateway.networking.k8s.io/v1 +kind: HTTPRoute +metadata: + name: logstashui + namespace: logstashui +spec: + parentRefs: + - group: gateway.networking.k8s.io + kind: Gateway + name: public # your Gateway + namespace: envoygw # Gateway namespace + hostnames: + - logstashui.example.com + rules: + - backendRefs: + - group: gateway.envoyproxy.io + kind: Backend + name: logstashui + matches: + - path: + type: PathPrefix + value: / +``` + +Skip-verify is for **testing / product-CA backends**. Prefer a `BackendTLSPolicy` with the product CA in production. + +--- + +## 3. Gateway `allowedRoutes` + +If the Gateway is in another namespace, its listener must allow HTTPRoutes from `logstashui`: + +```yaml +allowedRoutes: + namespaces: + from: All +``` + +or a label selector that includes the `logstashui` namespace. + +--- + +## 4. LogstashUI env + +Do **not** set `LOGSTASHUI_TLS=false`. Set `CSRF_TRUSTED_ORIGINS=https://` and `ALLOWED_HOSTS` to the same host. See [Kubernetes](index.md). diff --git a/docs/docs/logstashui/kubernetes/examples/README.md b/docs/docs/logstashui/kubernetes/examples/README.md new file mode 100644 index 0000000..1ab324e --- /dev/null +++ b/docs/docs/logstashui/kubernetes/examples/README.md @@ -0,0 +1,17 @@ +# Example manifests + +Apply **one** tree. All of them keep LogstashUI TLS on (`:8443`) and use Ingress-nginx with `backend-protocol: HTTPS` and `proxy-ssl-verify: "off"`. Image: `codyjackson032/logstashui:latest`. Namespace: `logstashui`. Replicas: **1**. One PVC at `/var/lib/logstashui`. + +| Directory | Database | +|---|---| +| [sqlite/](sqlite/) | SQLite file on the data PVC. Simplest. Not for concurrent agents. | +| [postgresql/](postgresql/) | External PostgreSQL 14+. Optional [cnpg.yaml](postgresql/cnpg.yaml) if you run CloudNativePG. | +| [mysql/](mysql/) | MariaDB 10.6+ or MySQL 8.0+. Engine is `mysql`. Create the schema as `utf8mb4` / `utf8mb4_bin`. | + +Replace `logstashui.example.com` and `SECRET_KEY` before apply. For Postgres/MySQL, create the empty database first — [SQL examples](/docs/docs/logstashui/database/examples/). + +Envoy Gateway instead of Ingress: [envoy-gateway.md](../envoy-gateway.md) (enable the Backend API, then `Backend` + `HTTPRoute`). + +```bash +kubectl apply -f docs/docs/logstashui/kubernetes/examples/sqlite/ +``` diff --git a/docs/docs/logstashui/kubernetes/examples/mysql/configmap.yaml b/docs/docs/logstashui/kubernetes/examples/mysql/configmap.yaml new file mode 100644 index 0000000..92d002f --- /dev/null +++ b/docs/docs/logstashui/kubernetes/examples/mysql/configmap.yaml @@ -0,0 +1,24 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: logstashui + namespace: logstashui +data: + LOGSTASHUI_DATA_DIR: /var/lib/logstashui + LOGSTASHUI_BIND: 0.0.0.0:8443 + LOGSTASHUI_TLS: "true" + LOGSTASHUI_WORKERS: "2" + DEBUG: "false" + ALLOWED_HOSTS: logstashui.example.com,logstashui,logstashui.logstashui.svc,logstashui.logstashui.svc.cluster.local + CSRF_TRUSTED_ORIGINS: https://logstashui.example.com + LOGSTASHUI_TLS_SANS: logstashui.example.com,logstashui.logstashui.svc.cluster.local + LOGSTASHUI_HOST_HOSTNAME: logstashui.example.com + LOGSTASHUI_AGENT_UI_URL: https://logstashui.example.com + # MariaDB uses the same engine. Create the database as utf8mb4 / utf8mb4_bin. + LOGSTASHUI_DB_ENGINE: mysql + LOGSTASHUI_DB_HOST: mysql.example.svc.cluster.local + LOGSTASHUI_DB_PORT: "3306" + LOGSTASHUI_DB_NAME: logstashui + LOGSTASHUI_DB_USER: logstashui + # Optional CA file path if the server requires TLS: + # LOGSTASHUI_DB_SSL_CA: /etc/ssl/certs/mysql-ca.pem diff --git a/docs/docs/logstashui/kubernetes/examples/mysql/ingress.yaml b/docs/docs/logstashui/kubernetes/examples/mysql/ingress.yaml new file mode 100644 index 0000000..1cad21e --- /dev/null +++ b/docs/docs/logstashui/kubernetes/examples/mysql/ingress.yaml @@ -0,0 +1,27 @@ +# Ingress-nginx: originate HTTPS to the pod and skip verify of the product leaf. +# Do not set LOGSTASHUI_TLS=false on the container. +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: logstashui + namespace: logstashui + annotations: + nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" + nginx.ingress.kubernetes.io/proxy-ssl-verify: "off" +spec: + ingressClassName: nginx + # tls: + # - hosts: + # - logstashui.example.com + # secretName: logstashui-tls + rules: + - host: logstashui.example.com + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: logstashui + port: + number: 8443 diff --git a/docs/docs/logstashui/kubernetes/examples/mysql/namespace.yaml b/docs/docs/logstashui/kubernetes/examples/mysql/namespace.yaml new file mode 100644 index 0000000..4444b7c --- /dev/null +++ b/docs/docs/logstashui/kubernetes/examples/mysql/namespace.yaml @@ -0,0 +1,4 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: logstashui diff --git a/docs/docs/logstashui/kubernetes/examples/mysql/secret.yaml b/docs/docs/logstashui/kubernetes/examples/mysql/secret.yaml new file mode 100644 index 0000000..19ef69e --- /dev/null +++ b/docs/docs/logstashui/kubernetes/examples/mysql/secret.yaml @@ -0,0 +1,9 @@ +apiVersion: v1 +kind: Secret +metadata: + name: logstashui + namespace: logstashui +type: Opaque +stringData: + SECRET_KEY: CHANGE-ME-generate-a-django-secret-key + LOGSTASHUI_DB_PASSWORD: CHANGE-ME diff --git a/docs/docs/logstashui/kubernetes/examples/mysql/statefulset.yaml b/docs/docs/logstashui/kubernetes/examples/mysql/statefulset.yaml new file mode 100644 index 0000000..d614205 --- /dev/null +++ b/docs/docs/logstashui/kubernetes/examples/mysql/statefulset.yaml @@ -0,0 +1,98 @@ +apiVersion: v1 +kind: Service +metadata: + name: logstashui + namespace: logstashui + labels: + app: logstashui +spec: + selector: + app: logstashui + ports: + - name: https + port: 8443 + targetPort: https +--- +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: logstashui + namespace: logstashui +spec: + serviceName: logstashui + replicas: 1 + selector: + matchLabels: + app: logstashui + template: + metadata: + labels: + app: logstashui + spec: + securityContext: + fsGroup: 10001 + fsGroupChangePolicy: OnRootMismatch + containers: + - name: logstashui + image: codyjackson032/logstashui:latest + imagePullPolicy: IfNotPresent + ports: + - name: https + containerPort: 8443 + envFrom: + - configMapRef: + name: logstashui + - secretRef: + name: logstashui + env: + - name: LOGSTASHUI_HOST_IPS + valueFrom: + fieldRef: + fieldPath: status.podIP + securityContext: + runAsUser: 10001 + runAsGroup: 10001 + runAsNonRoot: true + allowPrivilegeEscalation: false + capabilities: + drop: ["ALL"] + volumeMounts: + - name: data + mountPath: /var/lib/logstashui + startupProbe: + httpGet: + path: / + port: https + scheme: HTTPS + httpHeaders: + - name: Host + value: logstashui + periodSeconds: 5 + failureThreshold: 60 + readinessProbe: + httpGet: + path: / + port: https + scheme: HTTPS + httpHeaders: + - name: Host + value: logstashui + periodSeconds: 10 + livenessProbe: + httpGet: + path: / + port: https + scheme: HTTPS + httpHeaders: + - name: Host + value: logstashui + periodSeconds: 30 + failureThreshold: 6 + volumeClaimTemplates: + - metadata: + name: data + spec: + accessModes: ["ReadWriteOnce"] + resources: + requests: + storage: 10Gi diff --git a/docs/docs/logstashui/kubernetes/examples/postgresql/cnpg.yaml b/docs/docs/logstashui/kubernetes/examples/postgresql/cnpg.yaml new file mode 100644 index 0000000..305c5be --- /dev/null +++ b/docs/docs/logstashui/kubernetes/examples/postgresql/cnpg.yaml @@ -0,0 +1,21 @@ +# Optional. Requires the CloudNativePG operator (often in cnpg-system). +# See ../../cnpg.md. After Ready, point LogstashUI at Secret logstashui-pg-app +# instead of the HOST/USER/PASSWORD in configmap.yaml / secret.yaml. +apiVersion: postgresql.cnpg.io/v1 +kind: Cluster +metadata: + name: logstashui-pg + namespace: logstashui +spec: + instances: 1 + imageName: ghcr.io/cloudnative-pg/postgresql:16 + bootstrap: + initdb: + database: logstashui + owner: logstashui + encoding: UTF8 + storage: + size: 10Gi + postgresql: + parameters: + max_connections: "200" diff --git a/docs/docs/logstashui/kubernetes/examples/postgresql/configmap.yaml b/docs/docs/logstashui/kubernetes/examples/postgresql/configmap.yaml new file mode 100644 index 0000000..49cb8d0 --- /dev/null +++ b/docs/docs/logstashui/kubernetes/examples/postgresql/configmap.yaml @@ -0,0 +1,25 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: logstashui + namespace: logstashui +data: + LOGSTASHUI_DATA_DIR: /var/lib/logstashui + LOGSTASHUI_BIND: 0.0.0.0:8443 + LOGSTASHUI_TLS: "true" + LOGSTASHUI_WORKERS: "2" + DEBUG: "false" + ALLOWED_HOSTS: logstashui.example.com,logstashui,logstashui.logstashui.svc,logstashui.logstashui.svc.cluster.local + CSRF_TRUSTED_ORIGINS: https://logstashui.example.com + LOGSTASHUI_TLS_SANS: logstashui.example.com,logstashui.logstashui.svc.cluster.local + LOGSTASHUI_HOST_HOSTNAME: logstashui.example.com + LOGSTASHUI_AGENT_UI_URL: https://logstashui.example.com + LOGSTASHUI_DB_ENGINE: postgresql + LOGSTASHUI_DB_HOST: postgres.example.svc.cluster.local + LOGSTASHUI_DB_PORT: "5432" + LOGSTASHUI_DB_NAME: logstashui + LOGSTASHUI_DB_USER: logstashui + LOGSTASHUI_DB_SSLMODE: prefer + # PASSWORD is in secret.yaml. For CloudNativePG, prefer secretKeyRef from + # -app instead of these HOST/USER/NAME keys — see cnpg.yaml and + # ../../cnpg.md. diff --git a/docs/docs/logstashui/kubernetes/examples/postgresql/ingress.yaml b/docs/docs/logstashui/kubernetes/examples/postgresql/ingress.yaml new file mode 100644 index 0000000..1cad21e --- /dev/null +++ b/docs/docs/logstashui/kubernetes/examples/postgresql/ingress.yaml @@ -0,0 +1,27 @@ +# Ingress-nginx: originate HTTPS to the pod and skip verify of the product leaf. +# Do not set LOGSTASHUI_TLS=false on the container. +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: logstashui + namespace: logstashui + annotations: + nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" + nginx.ingress.kubernetes.io/proxy-ssl-verify: "off" +spec: + ingressClassName: nginx + # tls: + # - hosts: + # - logstashui.example.com + # secretName: logstashui-tls + rules: + - host: logstashui.example.com + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: logstashui + port: + number: 8443 diff --git a/docs/docs/logstashui/kubernetes/examples/postgresql/namespace.yaml b/docs/docs/logstashui/kubernetes/examples/postgresql/namespace.yaml new file mode 100644 index 0000000..4444b7c --- /dev/null +++ b/docs/docs/logstashui/kubernetes/examples/postgresql/namespace.yaml @@ -0,0 +1,4 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: logstashui diff --git a/docs/docs/logstashui/kubernetes/examples/postgresql/secret.yaml b/docs/docs/logstashui/kubernetes/examples/postgresql/secret.yaml new file mode 100644 index 0000000..19ef69e --- /dev/null +++ b/docs/docs/logstashui/kubernetes/examples/postgresql/secret.yaml @@ -0,0 +1,9 @@ +apiVersion: v1 +kind: Secret +metadata: + name: logstashui + namespace: logstashui +type: Opaque +stringData: + SECRET_KEY: CHANGE-ME-generate-a-django-secret-key + LOGSTASHUI_DB_PASSWORD: CHANGE-ME diff --git a/docs/docs/logstashui/kubernetes/examples/postgresql/statefulset.yaml b/docs/docs/logstashui/kubernetes/examples/postgresql/statefulset.yaml new file mode 100644 index 0000000..d614205 --- /dev/null +++ b/docs/docs/logstashui/kubernetes/examples/postgresql/statefulset.yaml @@ -0,0 +1,98 @@ +apiVersion: v1 +kind: Service +metadata: + name: logstashui + namespace: logstashui + labels: + app: logstashui +spec: + selector: + app: logstashui + ports: + - name: https + port: 8443 + targetPort: https +--- +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: logstashui + namespace: logstashui +spec: + serviceName: logstashui + replicas: 1 + selector: + matchLabels: + app: logstashui + template: + metadata: + labels: + app: logstashui + spec: + securityContext: + fsGroup: 10001 + fsGroupChangePolicy: OnRootMismatch + containers: + - name: logstashui + image: codyjackson032/logstashui:latest + imagePullPolicy: IfNotPresent + ports: + - name: https + containerPort: 8443 + envFrom: + - configMapRef: + name: logstashui + - secretRef: + name: logstashui + env: + - name: LOGSTASHUI_HOST_IPS + valueFrom: + fieldRef: + fieldPath: status.podIP + securityContext: + runAsUser: 10001 + runAsGroup: 10001 + runAsNonRoot: true + allowPrivilegeEscalation: false + capabilities: + drop: ["ALL"] + volumeMounts: + - name: data + mountPath: /var/lib/logstashui + startupProbe: + httpGet: + path: / + port: https + scheme: HTTPS + httpHeaders: + - name: Host + value: logstashui + periodSeconds: 5 + failureThreshold: 60 + readinessProbe: + httpGet: + path: / + port: https + scheme: HTTPS + httpHeaders: + - name: Host + value: logstashui + periodSeconds: 10 + livenessProbe: + httpGet: + path: / + port: https + scheme: HTTPS + httpHeaders: + - name: Host + value: logstashui + periodSeconds: 30 + failureThreshold: 6 + volumeClaimTemplates: + - metadata: + name: data + spec: + accessModes: ["ReadWriteOnce"] + resources: + requests: + storage: 10Gi diff --git a/docs/docs/logstashui/kubernetes/examples/sqlite/configmap.yaml b/docs/docs/logstashui/kubernetes/examples/sqlite/configmap.yaml new file mode 100644 index 0000000..72042fa --- /dev/null +++ b/docs/docs/logstashui/kubernetes/examples/sqlite/configmap.yaml @@ -0,0 +1,18 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: logstashui + namespace: logstashui +data: + LOGSTASHUI_DATA_DIR: /var/lib/logstashui + LOGSTASHUI_BIND: 0.0.0.0:8443 + LOGSTASHUI_TLS: "true" + LOGSTASHUI_WORKERS: "2" + DEBUG: "false" + ALLOWED_HOSTS: logstashui.example.com,logstashui,logstashui.logstashui.svc,logstashui.logstashui.svc.cluster.local + CSRF_TRUSTED_ORIGINS: https://logstashui.example.com + LOGSTASHUI_TLS_SANS: logstashui.example.com,logstashui.logstashui.svc.cluster.local + LOGSTASHUI_HOST_HOSTNAME: logstashui.example.com + LOGSTASHUI_AGENT_UI_URL: https://logstashui.example.com + LOGSTASHUI_DB_ENGINE: sqlite + # db.sqlite3 is created at $LOGSTASHUI_DATA_DIR/db.sqlite3 on the PVC. diff --git a/docs/docs/logstashui/kubernetes/examples/sqlite/ingress.yaml b/docs/docs/logstashui/kubernetes/examples/sqlite/ingress.yaml new file mode 100644 index 0000000..1cad21e --- /dev/null +++ b/docs/docs/logstashui/kubernetes/examples/sqlite/ingress.yaml @@ -0,0 +1,27 @@ +# Ingress-nginx: originate HTTPS to the pod and skip verify of the product leaf. +# Do not set LOGSTASHUI_TLS=false on the container. +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: logstashui + namespace: logstashui + annotations: + nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" + nginx.ingress.kubernetes.io/proxy-ssl-verify: "off" +spec: + ingressClassName: nginx + # tls: + # - hosts: + # - logstashui.example.com + # secretName: logstashui-tls + rules: + - host: logstashui.example.com + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: logstashui + port: + number: 8443 diff --git a/docs/docs/logstashui/kubernetes/examples/sqlite/namespace.yaml b/docs/docs/logstashui/kubernetes/examples/sqlite/namespace.yaml new file mode 100644 index 0000000..4444b7c --- /dev/null +++ b/docs/docs/logstashui/kubernetes/examples/sqlite/namespace.yaml @@ -0,0 +1,4 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: logstashui diff --git a/docs/docs/logstashui/kubernetes/examples/sqlite/secret.yaml b/docs/docs/logstashui/kubernetes/examples/sqlite/secret.yaml new file mode 100644 index 0000000..1e9afcc --- /dev/null +++ b/docs/docs/logstashui/kubernetes/examples/sqlite/secret.yaml @@ -0,0 +1,8 @@ +apiVersion: v1 +kind: Secret +metadata: + name: logstashui + namespace: logstashui +type: Opaque +stringData: + SECRET_KEY: CHANGE-ME-generate-a-django-secret-key diff --git a/docs/docs/logstashui/kubernetes/examples/sqlite/statefulset.yaml b/docs/docs/logstashui/kubernetes/examples/sqlite/statefulset.yaml new file mode 100644 index 0000000..d614205 --- /dev/null +++ b/docs/docs/logstashui/kubernetes/examples/sqlite/statefulset.yaml @@ -0,0 +1,98 @@ +apiVersion: v1 +kind: Service +metadata: + name: logstashui + namespace: logstashui + labels: + app: logstashui +spec: + selector: + app: logstashui + ports: + - name: https + port: 8443 + targetPort: https +--- +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: logstashui + namespace: logstashui +spec: + serviceName: logstashui + replicas: 1 + selector: + matchLabels: + app: logstashui + template: + metadata: + labels: + app: logstashui + spec: + securityContext: + fsGroup: 10001 + fsGroupChangePolicy: OnRootMismatch + containers: + - name: logstashui + image: codyjackson032/logstashui:latest + imagePullPolicy: IfNotPresent + ports: + - name: https + containerPort: 8443 + envFrom: + - configMapRef: + name: logstashui + - secretRef: + name: logstashui + env: + - name: LOGSTASHUI_HOST_IPS + valueFrom: + fieldRef: + fieldPath: status.podIP + securityContext: + runAsUser: 10001 + runAsGroup: 10001 + runAsNonRoot: true + allowPrivilegeEscalation: false + capabilities: + drop: ["ALL"] + volumeMounts: + - name: data + mountPath: /var/lib/logstashui + startupProbe: + httpGet: + path: / + port: https + scheme: HTTPS + httpHeaders: + - name: Host + value: logstashui + periodSeconds: 5 + failureThreshold: 60 + readinessProbe: + httpGet: + path: / + port: https + scheme: HTTPS + httpHeaders: + - name: Host + value: logstashui + periodSeconds: 10 + livenessProbe: + httpGet: + path: / + port: https + scheme: HTTPS + httpHeaders: + - name: Host + value: logstashui + periodSeconds: 30 + failureThreshold: 6 + volumeClaimTemplates: + - metadata: + name: data + spec: + accessModes: ["ReadWriteOnce"] + resources: + requests: + storage: 10Gi diff --git a/docs/docs/logstashui/kubernetes/index.md b/docs/docs/logstashui/kubernetes/index.md new file mode 100644 index 0000000..dac7296 --- /dev/null +++ b/docs/docs/logstashui/kubernetes/index.md @@ -0,0 +1,113 @@ +# Kubernetes + +Run LogstashUI as a one-replica StatefulSet. The container image is `codyjackson032/logstashui:latest` (or a tag you built). Configuration is environment variables only — ConfigMap for non-secrets, Secret for `SECRET_KEY` and `LOGSTASHUI_DB_PASSWORD`. + +Copy-paste manifests: + +- [SQLite](examples/sqlite/) — database file on the data PVC (fine for a single replica; not for concurrent agents) +- [PostgreSQL](examples/postgresql/) — external Postgres 14+, optional [CloudNativePG](cnpg.md) +- [MySQL / MariaDB](examples/mysql/) — one engine (`mysql`); create the database as `utf8mb4` / `utf8mb4_bin` + +Envoy Gateway users: [skip backend TLS verify](envoy-gateway.md) (`Backend` CR). Ingress-nginx users: see the `ingress.yaml` in each examples folder. + +--- + +## Image and command + +The image already installs `LogstashUI[databases]`. `CMD` is `logstashui serve`. Do not override the command unless you are debugging. + +```bash +docker build -f docker/Dockerfile -t logstashui:0.5.2-dev . +# Apple Silicon → amd64 cluster: +docker build --platform linux/amd64 -f docker/Dockerfile -t logstashui:0.5.2-dev . +``` + +Build context is the **repository root**. + +--- + +## Data directory (required) + +Set `LOGSTASHUI_DATA_DIR=/var/lib/logstashui` and mount a PVC there. The PVC holds TLS (`tls/`), the Django secret file, logs, `staticfiles/`, and (SQLite only) `db.sqlite3`. Keep the PVC when the database is Postgres or MySQL — it is not optional. + +Use **one** `volumeClaimTemplate` named `data`. Splitting logs onto a second claim is not useful: `LOGSTASHUI_LOGS_DIR` still lives under `DATA_DIR` unless you override it, and TLS/secrets cannot move. + +**Replicas: 1.** LogstashUI is not horizontally scalable (product CA, gunicorn pidfile, SQLite). + +--- + +## Security context + +Image user is **appuser (uid/gid 10001)**. The entrypoint chowns `DATA_DIR` only when it is root. Kubernetes should not run the container as root: + +```yaml +spec: + template: + spec: + securityContext: + fsGroup: 10001 + fsGroupChangePolicy: OnRootMismatch + containers: + - name: logstashui + securityContext: + runAsUser: 10001 + runAsGroup: 10001 + runAsNonRoot: true + allowPrivilegeEscalation: false + capabilities: + drop: ["ALL"] +``` + +--- + +## TLS (keep it on) + +Leave `LOGSTASHUI_TLS` unset or `true`. Gunicorn serves HTTPS on **8443** with the product CA under `$LOGSTASHUI_DATA_DIR/tls/`. Do **not** set `LOGSTASHUI_TLS=false` for an ingress or HTTPRoute. + +The gateway or Ingress **originates HTTPS** to the pod and **skips backend certificate verification** because the product leaf is self-signed: + +- Ingress-nginx: `backend-protocol: HTTPS` and `proxy-ssl-verify: "off"` (see examples) +- Envoy Gateway: [`Backend` with `tls.insecureSkipVerify: true`](envoy-gateway.md) + +Set `CSRF_TRUSTED_ORIGINS` and `ALLOWED_HOSTS` to the public hostname (keep `logstashui` in `ALLOWED_HOSTS`). `LOGSTASHUI_TLS_SANS` / `LOGSTASHUI_HOST_HOSTNAME` add that name to the product leaf. `LOGSTASHUI_AGENT_UI_URL` is the URL agents should use. + +Inject the pod IP so the product leaf SAN matches Kubernetes (compose does this with `LOGSTASHUI_HOST_IPS` on the host; the UDP-to-8.8.8.8 trick often has no egress in a cluster): + +```yaml +env: + - name: LOGSTASHUI_HOST_IPS + valueFrom: + fieldRef: + fieldPath: status.podIP +``` + +Django also appends `LOGSTASHUI_HOST_IPS` / `POD_IP` to `ALLOWED_HOSTS` unless that list is `*`. + +Probes: `httpGet` `scheme: HTTPS` on port 8443. kubelet does not verify the pod certificate, but it **does** send `Host: :8443`. Set the probe Host to a name already in `ALLOWED_HOSTS` (examples use `logstashui`): + +```yaml +httpGet: + path: / + port: https + scheme: HTTPS + httpHeaders: + - name: Host + value: logstashui +``` + +--- + +## Database + +Default engine is SQLite on the PVC. For Postgres or MySQL/MariaDB, set the discrete `LOGSTASHUI_DB_*` keys — see [Database](/docs/docs/logstashui/database/index.md). Put the password in a Secret. CloudNativePG: [CNPG](cnpg.md). + +--- + +## Apply (generic) + +```bash +kubectl apply -f docs/docs/logstashui/kubernetes/examples/sqlite/ +# or postgresql/ or mysql/ +``` + +Replace `logstashui.example.com` in ConfigMap, Secret, and Ingress. Replace `SECRET_KEY`. For Postgres/MySQL, create the empty database first ([SQL examples](/docs/docs/logstashui/database/examples/)), then apply the StatefulSet. First start runs `migrate`. diff --git a/src/logstashui/LogstashUI/config.py b/src/logstashui/LogstashUI/config.py index 60b1956..ce999f1 100644 --- a/src/logstashui/LogstashUI/config.py +++ b/src/logstashui/LogstashUI/config.py @@ -11,6 +11,38 @@ _FALSE = ("0", "false", "no", "off") +def _split_csv_hosts(raw: str) -> list[str]: + return [part.strip() for part in (raw or "").split(",") if part.strip()] + + +def merge_allowed_hosts( + allowed: str | None = None, + host_ips: str | None = None, + pod_ip: str | None = None, +) -> list[str]: + """Django ALLOWED_HOSTS, plus pod/host IPs used as kube-probe Host headers. + + ``ALLOWED_HOSTS=*`` stays a single wildcard. Otherwise ``LOGSTASHUI_HOST_IPS`` + and ``POD_IP`` are appended (Kubernetes Downward API / compose host IPs). + """ + if allowed is None: + allowed = os.environ.get("ALLOWED_HOSTS", "*") + hosts = _split_csv_hosts(allowed) + if not hosts: + hosts = ["*"] + if hosts == ["*"]: + return hosts + extras = _split_csv_hosts( + host_ips if host_ips is not None else os.environ.get("LOGSTASHUI_HOST_IPS", "") + ) + extra_pod = pod_ip if pod_ip is not None else os.environ.get("POD_IP", "") + extras.extend(_split_csv_hosts(extra_pod)) + for host in extras: + if host not in hosts: + hosts.append(host) + return hosts + + def env_bool(name: str, default: bool = False) -> bool: raw = os.environ.get(name) if raw is None: diff --git a/src/logstashui/LogstashUI/settings.py b/src/logstashui/LogstashUI/settings.py index bed0bb3..06334cc 100644 --- a/src/logstashui/LogstashUI/settings.py +++ b/src/logstashui/LogstashUI/settings.py @@ -18,7 +18,7 @@ import os, platform from importlib.metadata import version, PackageNotFoundError from Common.encryption import get_django_secret_key -from .config import CONFIG +from .config import CONFIG, merge_allowed_hosts from .database import build_databases from .logging_config import resolve_django_log_levels, resolve_log_level from .paths import resolve_data_dir, resolve_docs_dir, resolve_logs_dir @@ -52,7 +52,7 @@ # SECURITY WARNING: Set ALLOWED_HOSTS to your domain(s) in production # Example: ALLOWED_HOSTS=yourdomain.com,www.yourdomain.com -ALLOWED_HOSTS = os.environ.get('ALLOWED_HOSTS', '*').split(',') +ALLOWED_HOSTS = merge_allowed_hosts() def _get_version(): """Get version from installed package metadata or pyproject.toml""" diff --git a/src/logstashui/LogstashUI/tests/test_config.py b/src/logstashui/LogstashUI/tests/test_config.py index 2889fcc..3c8e862 100644 --- a/src/logstashui/LogstashUI/tests/test_config.py +++ b/src/logstashui/LogstashUI/tests/test_config.py @@ -2,7 +2,27 @@ #or more contributor license agreements. Licensed under the Elastic License; #you may not use this file except in compliance with the Elastic License. -from LogstashUI.config import load_config +from LogstashUI.config import load_config, merge_allowed_hosts + + +def test_merge_allowed_hosts_wildcard_unchanged(): + assert merge_allowed_hosts(allowed="*", host_ips="10.11.3.107") == ["*"] + + +def test_merge_allowed_hosts_appends_pod_ip(): + hosts = merge_allowed_hosts( + allowed="logstashui.example.com,logstashui", + host_ips="10.11.3.107", + pod_ip="", + ) + assert hosts == ["logstashui.example.com", "logstashui", "10.11.3.107"] + + +def test_merge_allowed_hosts_pod_ip_env_and_no_dupes(monkeypatch): + monkeypatch.setenv("ALLOWED_HOSTS", "logstashui") + monkeypatch.setenv("LOGSTASHUI_HOST_IPS", "10.11.3.107") + monkeypatch.setenv("POD_IP", "10.11.3.107") + assert merge_allowed_hosts() == ["logstashui", "10.11.3.107"] def test_load_config_defaults(monkeypatch): From 0ef9af9e38b8e5fdc60e1a9aa96807ae23366abb Mon Sep 17 00:00:00 2001 From: Aaron Mildenstein Date: Wed, 2 Sep 2026 16:15:55 -0600 Subject: [PATCH 22/62] docs: air-gapped freeze design spec Optional freeze script (wheels + docker zip + experimental PyInstaller). Default uv build unchanged. CPython 3.12 linux-x86_64, [databases] included. --- .../specs/2026-09-02-airgap-freeze-design.md | 182 ++++++++++++++++++ 1 file changed, 182 insertions(+) create mode 100644 docs/superpowers/specs/2026-09-02-airgap-freeze-design.md diff --git a/docs/superpowers/specs/2026-09-02-airgap-freeze-design.md b/docs/superpowers/specs/2026-09-02-airgap-freeze-design.md new file mode 100644 index 0000000..955d751 --- /dev/null +++ b/docs/superpowers/specs/2026-09-02-airgap-freeze-design.md @@ -0,0 +1,182 @@ +# 2026-09-02 — Air-gapped freeze (optional offline zip) + +**LogstashUI version:** 0.5.2 (packaging feature; does not bump by itself) +**Builder:** Linux x86_64, CPython **3.12**, uv, Docker (for the image tarball) +**Isolated host (wheels):** Linux x86_64, CPython **3.12** (venv module / distro `python3.12-venv`) +**Isolated host (docker):** Docker Engine +**Isolated host (standalone):** glibc-compatible Linux x86_64, no Python +**Chosen approach:** Optional freeze **script** that emits up to three artifacts. Default `uv build` / hatchling / published image **unchanged**. + +## Goal + +A maintainer with internet builds zip files that an air-gapped operator can copy, unpack, and run **without PyPI or a container registry**. This is not the default packaging path. + +## Non-goals (v1) + +- Changing hatchling includes, extras, or `uv build` output. +- A `logstashui freeze` CLI subcommand. +- Bundling LogstashAgent (separate product; compose `embedded` profile stays online-registry). +- Bundling CPython into the wheelhouse (host must already have 3.12). +- arm64 or Windows artifacts (names keep platform/ABI so those can be added later). +- Multiple CPython ABIs in one zip (cp312 only). +- Shipping uv, Node, or a compiler to the isolated host. +- Helm, K8s operator, or air-gap registry mirroring as this feature. +- Making freeze a required PR CI gate. +- Auto-update / delta patches of a freeze. + +## Decisions (locked) + +| Topic | Decision | +|---|---| +| Builder entry | `bin/freeze_logstashui.sh` (bash). Not default packaging. | +| Artifacts | Wheels zip **and** Docker tarball **and** standalone zip. Operator tests all three. | +| Default `uv build` | Unchanged (sdist + py3-none-any wheel only). | +| Python ABI | CPython **3.12** x86_64 only. Documented pin. | +| Platform | Linux x86_64 only. Later: arm64 / Windows as extra tags, not a rename. | +| Extras | Always include `LogstashUI[databases]` (psycopg binary + PyMySQL). | +| Agent | Not in any freeze zip. | +| Isolated run | Each zip has `README.md` + helper (`install.sh` / `load.sh` / `run.sh`). | +| Config | Same env as today. No YAML. `LOGSTASHUI_DATA_DIR` default `$(pwd)/logstashui_data`. | +| Standalone tool | PyInstaller. **Experimental** until smoke covers migrate + SNMP sync + TLS serve. | +| Wheel policy | **Binary wheels only** (`pip download --only-binary :all:`). Sdist-only dep → freeze **fails**. | +| Pins | Resolve from `uv.lock` (`uv export --frozen`). | +| PyInstaller in deps | **No.** Freeze script uses `uvx pyinstaller` (or a throwaway venv). Do not add to `[project]` or default `dev`. | +| CI | Optional `workflow_dispatch` only. Not on every PR. | + +## Architecture + +``` +connected builder (linux x86_64, python3.12, uv, docker) + | + +-- bin/freeze_logstashui.sh [--wheels] [--docker] [--standalone] [--all] + | + +-- dist/offline/logstashui-{ver}-offline-wheels-linux-x86_64-cp312.zip + +-- dist/offline/logstashui-{ver}-offline-docker-linux-x86_64.zip + +-- dist/offline/logstashui-{ver}-offline-standalone-linux-x86_64.zip + +isolated host copies zip → unzip → helper → logstashui serve +``` + +`--all` if no artifact flags. `--output DIR` default `dist/offline`. `--image NAME` for docker-save (skip build). `--version` taken from `pyproject.toml`. + +Output stays under gitignored `/dist/`. + +## Artifact 1 — Wheelhouse + +**Builder steps** + +1. Ensure Tailwind CSS exists (`src/logstashui/theme/static/css/dist/styles.css`); build it the same way `uv build` / Dockerfile does if missing. +2. `uv build` (reuse the normal wheel; do not invent a second hatch target). +3. `uv export --frozen --no-dev --extra databases --no-emit-project -o dist/offline/requirements-offline.txt` (keep hashes). +4. Copy the local LogstashUI wheel into `wheels/`. Download deps with **pip** (uv 0.12 has no `uv pip download`): + `uv run --python 3.12 --with pip python -m pip download -r requirements-offline.txt -d wheels/ --python-version 3.12 --platform manylinux2014_x86_64 --implementation cp --abi cp312 --only-binary :all:` +5. Copy `LICENSE.txt`, `NOTICE.txt`, generated `install.sh`, `README.md`, `MANIFEST.txt`, `SHA256SUMS.txt`. +6. Zip. Fail if any `.tar.gz` sdist landed in `wheels/`. + +**MANIFEST.txt** must list: LogstashUI version, git sha (or `unknown` if not a git checkout), CPython 3.12, platform `linux-x86_64`, extra `databases`, package list with versions. + +**Isolated `install.sh`** + +- `PYTHON=${PYTHON:-python3.12}` +- Refuse unless `sys.version_info[:2] == (3, 12)` and the interpreter is 64-bit. +- `$PYTHON -m venv .venv` (do **not** `pip install --upgrade pip` — that hits PyPI). +- `.venv/bin/python -m pip install --no-index --no-cache-dir --find-links ./wheels 'LogstashUI[databases]'` +- Print: activate / `.venv/bin/logstashui serve`, data-dir default, env pointer to the configuration docs (copied as a short README section, not a second docs site). + +`install.sh` must not reach the network. `pip` `--no-index` is required. Prefer `--disable-pip-version-check`. + +Isolated host packages: `python3.12` and the distro venv module (Debian/Ubuntu: `python3.12-venv`). **uv is not required** on the isolated host. + +## Artifact 2 — Docker tarball + +**Builder** + +- If `--image` is set: `docker save` that **local** image. **Never** `docker pull`. Fail if the name is missing. +- Else: `docker build -f docker/Dockerfile -t logstashui:offline-{version} .` from repo root (same file as K8s/compose), then save that tag. +- Write `image.tar.gz` (`docker save | gzip`). Zip it with `load.sh`, UI-only `compose.offline.yml`, `README.md`, `SHA256SUMS.txt`, license files. Same zip story as the other two artifacts. + +**`compose.offline.yml`** + +- **One** service: LogstashUI. **No** Agent, **no** `embedded` profile. +- Image name matches what `docker load` will register. +- Port **8443**, `LOGSTASHUI_TLS` on, `LOGSTASHUI_DATA_DIR=/var/lib/logstashui`, named or bind volume for data. +- Same env-first model as Option 1. Operator fills `LOGSTASHUI_ALLOWED_HOSTS` / `LOGSTASHUI_DB_*` as needed. + +**Isolated `load.sh`:** `docker load -i `, print the image name, print `docker compose -f compose.offline.yml up -d`. + +## Artifact 3 — Standalone (experimental) + +PyInstaller **onedir** (not onefile — Django data files + gunicorn + gevent extract less painfully). Entry: `LogstashUI.cli:main`. + +Spec + helper templates live in repo `packaging/offline/` (not inside the Django wheel). Freeze script invokes `uvx pyinstaller` with that spec. + +Must collect: + +- All Django apps that the hatch wheel force-includes (`LogstashUI`, `PipelineManager`, `Management`, `Utilities`, `SNMP`, `Monitoring`, `Site`, `Documentation`, `AI`, `theme`, `Common`). +- Templates, static (including built Tailwind CSS), SNMP official data, packaged docs. +- Hidden imports: Django, gunicorn, gevent, greenlet, cryptography, pysnmp, lark, pygrok, whitenoise, psycopg, pymysql, yaml, htmx/tailwind Django apps. + +**Why experimental:** Django loads apps/commands by name; gunicorn+gevent fork and monkey-patch; native wheels (`cryptography`, `psycopg`, `gevent`) need the same glibc. A missed hidden import or data file often boots then dies on `migrate`, SNMP, or the first TLS handshake. The freeze **must** document that. + +**Pass criteria to drop “experimental”:** isolated-like run (no network) of `./logstashui serve` completes migrate + SNMP sync + collectstatic, binds **8443** HTTPS, `GET /` is not 500. Until then README says experimental; `run.sh` is `./logstashui serve`. + +If gevent+PyInstaller cannot pass that smoke, do **not** silently switch the product default worker. Document the failure and stop; a sync/gthread workaround is a spec amendment, not a surprise. + +## Isolated runtime (all three) + +Unchanged product behavior after install: + +- `logstashui serve` (migrate, SNMP official sync, collectstatic, product CA, gunicorn HTTPS :8443). +- Env: `LOGSTASHUI_*` / `LOGSTASHUI_DB_*` as today. SQLite default; Postgres/MySQL if they have a server. +- Do not rotate the product CA. Do not relocate `DATA_DIR` into the zip. +- systemd: after wheelhouse install, `logstashui systemd` still works if they have root (helper does **not** enable the unit). Standalone: document `ExecStart=` path to the unpacked binary; do not auto-write units in v1. + +## Error handling + +| Condition | Result | +|---|---| +| Builder not CPython 3.12 | Exit non-zero, print required ABI | +| Missing uv | Exit, how to install uv | +| `--docker` and Docker missing / image missing | Exit | +| `--standalone` and `uvx pyinstaller` fails | Exit | +| Dep has no manylinux cp312 binary wheel | Exit, name the package (no sdist fallback) | +| Tailwind CSS missing and npm build fails | Exit | +| Isolated `install.sh` wrong Python | Exit, “need CPython 3.12 x86_64” | +| Isolated pip would use PyPI | `--no-index` makes it fail; do not catch and retry online | + +Never log DB passwords. MANIFEST may list package names/versions. + +## Testing + +Not a default pytest matrix job. + +1. **Wheels (required for merge of the script):** run freeze `--wheels`; assert zip contains only `.whl` + scripts/docs; `docker run --network=none -v $PWD/zipdir:/offline python:3.12-slim` runs `install.sh` and `logstashui --help` plus `logstashui manage check`. +2. **Docker:** freeze `--docker`; `docker load`; `docker image inspect` the tag. Full compose up is manual. +3. **Standalone:** freeze `--standalone`; binary `--help`. Full `serve` smoke is the experiment gate (can be `--network=none` on a throwaway dir). + +Optional GitHub Action: `workflow_dispatch`, linux runner, `--wheels` only (fastest, no PyInstaller cache pain). `--docker` / `--standalone` stay maintainer commands. + +## Docs + +New page: `docs/docs/logstashui/general/offline.md` (air-gap freeze). Link from `deploy.md` as an extra option **after** Docker / pip / systemd / K8s. State clearly: not the recommended connected-network install. + +Page covers: builder prerequisites, the three zips, CPython 3.12 pin, `[databases]` included, no Agent, how to set env, DATA_DIR, and “later: arm64 / Windows”. + +CHANGELOG under 0.5.2: optional freeze script; default wheel unchanged. + +## Layout (repo) + +``` +bin/freeze_logstashui.sh # builder +packaging/offline/ # PyInstaller spec, README/install/load/run templates +docs/docs/logstashui/general/offline.md +``` + +Do not put freeze helpers inside `src/logstashui/LogstashUI/packaging/` (that tree is systemd templates shipped in the wheel). + +## Later (not v1) + +- `linux/arm64` and `win_amd64` as additional freeze invocations (`--python-platform` / Windows builder). +- LogstashAgent sibling freeze script. +- Promoting standalone from experimental after the serve smoke exists. +- Bundling a CPython embed for hosts with no 3.12 (different product). From 90b3351bead956770ae4d4af9ce9b3fd81c90a72 Mon Sep 17 00:00:00 2001 From: Aaron Mildenstein Date: Wed, 2 Sep 2026 16:30:29 -0600 Subject: [PATCH 23/62] feat: optional air-gapped freeze script Add bin/freeze_logstashui.sh for wheels, docker save, and experimental PyInstaller zips. Default uv build is unchanged. Isolated wheels install is pip --no-index on CPython 3.12 x86_64. --- .github/workflows/offline-freeze.yml | 20 ++ CHANGELOG.md | 1 + bin/freeze_logstashui.sh | 280 ++++++++++++++++++ bin/test_freeze_wheels.sh | 58 ++++ docs/docs/logstashui/general/build.md | 2 + docs/docs/logstashui/general/deploy.md | 10 + docs/docs/logstashui/general/index.md | 9 + docs/docs/logstashui/general/offline.md | 57 ++++ .../plans/2026-09-02-airgap-freeze.md | 34 +++ .../specs/2026-09-02-airgap-freeze-design.md | 7 +- packaging/offline/README.md | 5 + packaging/offline/compose.offline.yml | 34 +++ packaging/offline/docker-README.md | 16 + packaging/offline/docker-load.sh | 27 ++ packaging/offline/download_wheels.py | 176 +++++++++++ packaging/offline/entry.py | 10 + packaging/offline/logstashui.spec | 99 +++++++ packaging/offline/standalone-README.md | 19 ++ packaging/offline/standalone-run.sh | 16 + packaging/offline/wheels-README.md | 35 +++ packaging/offline/wheels-install.sh | 56 ++++ src/logstashui/LogstashUI/cli.py | 19 +- src/logstashui/LogstashUI/tests/test_cli.py | 21 +- 23 files changed, 1004 insertions(+), 7 deletions(-) create mode 100644 .github/workflows/offline-freeze.yml create mode 100755 bin/freeze_logstashui.sh create mode 100755 bin/test_freeze_wheels.sh create mode 100644 docs/docs/logstashui/general/offline.md create mode 100644 docs/superpowers/plans/2026-09-02-airgap-freeze.md create mode 100644 packaging/offline/README.md create mode 100644 packaging/offline/compose.offline.yml create mode 100644 packaging/offline/docker-README.md create mode 100755 packaging/offline/docker-load.sh create mode 100644 packaging/offline/download_wheels.py create mode 100644 packaging/offline/entry.py create mode 100644 packaging/offline/logstashui.spec create mode 100644 packaging/offline/standalone-README.md create mode 100755 packaging/offline/standalone-run.sh create mode 100644 packaging/offline/wheels-README.md create mode 100755 packaging/offline/wheels-install.sh diff --git a/.github/workflows/offline-freeze.yml b/.github/workflows/offline-freeze.yml new file mode 100644 index 0000000..4c397fb --- /dev/null +++ b/.github/workflows/offline-freeze.yml @@ -0,0 +1,20 @@ +name: Offline wheels freeze + +on: + workflow_dispatch: + +jobs: + wheels: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: astral-sh/setup-uv@v4 + with: + python-version: "3.12" + - uses: actions/setup-python@v5 + with: + python-version: "3.12" + - name: Freeze wheelhouse + run: bash bin/freeze_logstashui.sh --wheels + - name: Network-none install smoke + run: bash bin/test_freeze_wheels.sh diff --git a/CHANGELOG.md b/CHANGELOG.md index 9881c1b..3baa720 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,7 @@ SQLite does not scale under gunicorn/gevent. Operators can now run LogstashUI on ### Packaging and Docker - Native extras: `LogstashUI[postgres]`, `LogstashUI[mysql]`, `LogstashUI[databases]`. Default wheel stays SQLite-only. +- Optional air-gapped freeze: `bin/freeze_logstashui.sh` (`--wheels` / `--docker` / `--standalone`). Default `uv build` unchanged. Linux x86_64, CPython 3.12, `[databases]` included, no Agent. Wheelhouse prefers manylinux2014 then manylinux_2_28; pure-Python sdists are wheeled on the builder. Standalone PyInstaller is experimental. See [Air-gapped freeze](docs/docs/logstashui/general/offline.md). - Container image installs `LogstashUI[databases]` so Kubernetes only sets env. - systemd generator prompts for engine/host/port/name/user; sample `/etc/default/logstashui` documents all `LOGSTASHUI_DB_*` keys. Set the password in the EnvironmentFile or a Secret (`chmod 640`). - gunicorn writes `$LOGSTASHUI_DATA_DIR/gunicorn.pid`. diff --git a/bin/freeze_logstashui.sh b/bin/freeze_logstashui.sh new file mode 100755 index 0000000..5667589 --- /dev/null +++ b/bin/freeze_logstashui.sh @@ -0,0 +1,280 @@ +#!/bin/bash +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +# Optional air-gapped freeze. Default `uv build` is unchanged. +# Usage: +# ./bin/freeze_logstashui.sh [--wheels] [--docker] [--standalone] [--all] +# [--output DIR] [--image NAME] +# No artifact flags → --all. + +set -euo pipefail + +ROOT=$(CDPATH= cd -- "$(dirname "$0")/.." && pwd) +TEMPLATES="$ROOT/packaging/offline" +DO_WHEELS=0 +DO_DOCKER=0 +DO_STANDALONE=0 +EXPLICIT_STANDALONE=0 +OUT="" +IMAGE_OVERRIDE="" + +usage() { + cat <<'EOF' +Usage: freeze_logstashui.sh [--wheels] [--docker] [--standalone] [--all] + [--output DIR] [--image NAME] + +Connected-builder freeze for air-gapped hosts. Default uv build is unchanged. + + --wheels CPython 3.12 manylinux x86_64 wheelhouse zip + --docker docker save of a local image (never docker pull) + --standalone experimental PyInstaller onedir (Linux x86_64 only) + --all all three (default if no artifact flags) + --output DIR default: /dist/offline + --image NAME docker save this local tag instead of building + +Isolated wheels host: CPython 3.12 x86_64 + python3.12-venv. No uv. +EOF +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --wheels) DO_WHEELS=1 ;; + --docker) DO_DOCKER=1 ;; + --standalone) DO_STANDALONE=1; EXPLICIT_STANDALONE=1 ;; + --all) DO_WHEELS=1; DO_DOCKER=1; DO_STANDALONE=1 ;; + --output) + OUT="${2:?--output requires a directory}" + shift + ;; + --image) + IMAGE_OVERRIDE="${2:?--image requires a name}" + shift + ;; + -h|--help) + usage + exit 0 + ;; + *) + echo "Unknown option: $1" >&2 + usage >&2 + exit 1 + ;; + esac + shift +done + +if [[ $DO_WHEELS -eq 0 && $DO_DOCKER -eq 0 && $DO_STANDALONE -eq 0 ]]; then + DO_WHEELS=1 + DO_DOCKER=1 + DO_STANDALONE=1 +fi + +OUT="${OUT:-$ROOT/dist/offline}" +mkdir -p "$OUT" + +die() { echo "ERROR: $*" >&2; exit 1; } + +version_from_pyproject() { + sed -n 's/^version = "\([^"]*\)"/\1/p' "$ROOT/pyproject.toml" | head -n 1 +} + +subst() { + # $1 src $2 dest + local git_sha="$GIT_SHA" + local version="$VERSION" + local image="$IMAGE_NAME" + sed \ + -e "s|__VERSION__|${version}|g" \ + -e "s|__GIT_SHA__|${git_sha}|g" \ + -e "s|__IMAGE_NAME__|${image}|g" \ + "$1" > "$2" +} + +sha256_tree() { + local dir="$1" + local out="$2" + ( + cd "$dir" + if command -v sha256sum >/dev/null 2>&1; then + find . -type f ! -name SHA256SUMS.txt -print0 | sort -z | xargs -0 sha256sum + else + find . -type f ! -name SHA256SUMS.txt -print0 | sort -z | xargs -0 shasum -a 256 + fi + ) > "$out" +} + +zip_dir() { + local src="$1" + local dest="$2" + rm -f "$dest" + ( + cd "$(dirname "$src")" + zip -r -q "$dest" "$(basename "$src")" + ) +} + +require_uv() { + command -v uv >/dev/null 2>&1 || die "uv is required. Install: https://docs.astral.sh/uv/getting-started/installation/" + uv python find 3.12 >/dev/null 2>&1 || die "Need CPython 3.12 (uv python install 3.12)" + command -v zip >/dev/null 2>&1 || die "zip is required to pack freeze artifacts" +} + +ensure_tailwind() { + local css="$ROOT/src/logstashui/theme/static/css/dist/styles.css" + if [[ -s "$css" ]]; then + return 0 + fi + command -v npm >/dev/null 2>&1 || die "Tailwind CSS missing and npm is not installed" + (cd "$ROOT/src/logstashui/theme/static_src" && npm install && npm run build) + [[ -s "$css" ]] || die "Tailwind CSS build did not produce $css" +} + +linux_x86_64() { + [[ "$(uname -s)" == Linux && "$(uname -m)" == x86_64 ]] +} + +freeze_wheels() { + local stage="$OUT/logstashui-${VERSION}-offline-wheels-linux-x86_64-cp312" + local wheels="$stage/wheels" + local req="$OUT/requirements-offline.txt" + local zip="$OUT/logstashui-${VERSION}-offline-wheels-linux-x86_64-cp312.zip" + + rm -rf "$stage" + mkdir -p "$wheels" + + echo "==> uv build (normal wheel)" + (cd "$ROOT" && uv build) + + local whl + whl=$(ls -1 "$ROOT/dist"/logstashui-"${VERSION}"-*.whl 2>/dev/null | head -n 1 || true) + [[ -n "$whl" && -f "$whl" ]] || die "uv build did not produce dist/logstashui-${VERSION}-*.whl" + cp "$whl" "$wheels/" + + echo "==> uv export --frozen --extra databases" + (cd "$ROOT" && uv export --frozen --no-dev --extra databases --no-emit-project \ + -o "$req" >/dev/null) + local req_plain="$OUT/requirements-offline.nohash.txt" + (cd "$ROOT" && uv export --frozen --no-dev --extra databases --no-emit-project --no-hashes \ + -o "$req_plain" >/dev/null) + + echo "==> download manylinux cp312 wheels (pure-python sdists → wheel on builder)" + (cd "$ROOT" && uv run --python 3.12 --with pip python \ + "$TEMPLATES/download_wheels.py" "$req_plain" "$wheels") + + subst "$TEMPLATES/wheels-install.sh" "$stage/install.sh" + subst "$TEMPLATES/wheels-README.md" "$stage/README.md" + chmod +x "$stage/install.sh" + cp "$ROOT/LICENSE.txt" "$stage/LICENSE.txt" + cp "$ROOT/NOTICE.txt" "$stage/NOTICE.txt" + cp "$req" "$stage/requirements-offline.txt" + + { + echo "LogstashUI ${VERSION}" + echo "git ${GIT_SHA}" + echo "python CPython 3.12" + echo "platform linux-x86_64" + echo "extra databases" + echo + echo "wheels:" + (cd "$wheels" && ls -1 *.whl | sort) + } > "$stage/MANIFEST.txt" + + sha256_tree "$stage" "$stage/SHA256SUMS.txt" + zip_dir "$stage" "$zip" + echo "Wrote $zip" +} + +freeze_docker() { + local tag="${IMAGE_OVERRIDE:-logstashui:offline-${VERSION}}" + IMAGE_NAME="$tag" + local stage="$OUT/logstashui-${VERSION}-offline-docker-linux-x86_64" + local zip="$OUT/logstashui-${VERSION}-offline-docker-linux-x86_64.zip" + + command -v docker >/dev/null 2>&1 || die "docker is required for --docker" + + if [[ -n "$IMAGE_OVERRIDE" ]]; then + docker image inspect "$IMAGE_OVERRIDE" >/dev/null 2>&1 \ + || die "image ${IMAGE_OVERRIDE} is not local (never docker pull). Build it or omit --image." + else + echo "==> docker build --platform linux/amd64 ${tag}" + docker build --platform linux/amd64 -f "$ROOT/docker/Dockerfile" -t "$tag" "$ROOT" + fi + + rm -rf "$stage" + mkdir -p "$stage" + echo "==> docker save ${tag}" + docker save "$tag" | gzip -c > "$stage/image.tar.gz" + + subst "$TEMPLATES/docker-load.sh" "$stage/load.sh" + subst "$TEMPLATES/docker-README.md" "$stage/README.md" + subst "$TEMPLATES/compose.offline.yml" "$stage/compose.offline.yml" + chmod +x "$stage/load.sh" + cp "$ROOT/LICENSE.txt" "$stage/LICENSE.txt" + cp "$ROOT/NOTICE.txt" "$stage/NOTICE.txt" + sha256_tree "$stage" "$stage/SHA256SUMS.txt" + zip_dir "$stage" "$zip" + echo "Wrote $zip" +} + +freeze_standalone() { + if ! linux_x86_64; then + if [[ "$EXPLICIT_STANDALONE" -eq 1 ]]; then + die "standalone freeze requires Linux x86_64 (PyInstaller binary is per-OS)" + fi + echo "WARNING: skipping --standalone (not Linux x86_64)" >&2 + return 0 + fi + + local venv="$OUT/.standalone-venv" + local work="$OUT/pyinstaller-work" + local dist="$OUT/pyinstaller-dist" + local stage="$OUT/logstashui-${VERSION}-offline-standalone-linux-x86_64" + local zip="$OUT/logstashui-${VERSION}-offline-standalone-linux-x86_64.zip" + echo "==> throwaway venv + PyInstaller (not a project dependency)" + rm -rf "$venv" "$work" "$dist" + uv venv --python 3.12 "$venv" + (cd "$ROOT" && uv pip install --python "$venv" ".[databases]") + uv pip install --python "$venv" pyinstaller + "$venv/bin/pyinstaller" \ + --noconfirm \ + --clean \ + --workpath "$work" \ + --distpath "$dist" \ + "$TEMPLATES/logstashui.spec" + + [[ -x "$dist/logstashui/logstashui" ]] || die "PyInstaller did not produce $dist/logstashui/logstashui" + + rm -rf "$stage" + mkdir -p "$stage" + cp -a "$dist/logstashui" "$stage/logstashui" + subst "$TEMPLATES/standalone-run.sh" "$stage/run.sh" + subst "$TEMPLATES/standalone-README.md" "$stage/README.md" + chmod +x "$stage/run.sh" + cp "$ROOT/LICENSE.txt" "$stage/LICENSE.txt" + cp "$ROOT/NOTICE.txt" "$stage/NOTICE.txt" + sha256_tree "$stage" "$stage/SHA256SUMS.txt" + zip_dir "$stage" "$zip" + echo "Wrote $zip" +} + +VERSION=$(version_from_pyproject) +[[ -n "$VERSION" ]] || die "could not read version from pyproject.toml" +GIT_SHA=$(git -C "$ROOT" rev-parse --short HEAD 2>/dev/null || echo unknown) +IMAGE_NAME="${IMAGE_OVERRIDE:-logstashui:offline-${VERSION}}" + +require_uv +ensure_tailwind + +echo "LogstashUI ${VERSION} git ${GIT_SHA} output ${OUT}" + +if [[ $DO_WHEELS -eq 1 ]]; then + freeze_wheels +fi +if [[ $DO_DOCKER -eq 1 ]]; then + freeze_docker +fi +if [[ $DO_STANDALONE -eq 1 ]]; then + freeze_standalone +fi diff --git a/bin/test_freeze_wheels.sh b/bin/test_freeze_wheels.sh new file mode 100755 index 0000000..3b4f841 --- /dev/null +++ b/bin/test_freeze_wheels.sh @@ -0,0 +1,58 @@ +#!/bin/bash +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +# Smoke the air-gapped wheelhouse: unzip, pip --no-index in a network-none +# linux/amd64 CPython 3.12 container. Pull python:3.12-slim first (needs net). + +set -euo pipefail + +ROOT=$(CDPATH= cd -- "$(dirname "$0")/.." && pwd) +OUT="${1:-$ROOT/dist/offline}" + +ZIP=$(ls -1 "$OUT"/logstashui-*-offline-wheels-linux-x86_64-cp312.zip 2>/dev/null | tail -n 1 || true) +[[ -n "$ZIP" && -f "$ZIP" ]] || { + echo "ERROR: no wheels zip in $OUT. Run: bin/freeze_logstashui.sh --wheels" >&2 + exit 1 +} + +WORKDIR=$(mktemp -d) +trap 'rm -rf "$WORKDIR"' EXIT +unzip -q "$ZIP" -d "$WORKDIR" +STAGE=$(find "$WORKDIR" -maxdepth 1 -type d -name 'logstashui-*-offline-wheels-*' | head -n 1) +[[ -n "$STAGE" ]] || STAGE=$WORKDIR + +shopt -s nullglob +sdists=("$STAGE"/wheels/*.tar.gz) +if (( ${#sdists[@]} )); then + echo "ERROR: sdist in wheelhouse" >&2 + printf '%s\n' "${sdists[@]}" + exit 1 +fi + +command -v docker >/dev/null 2>&1 || { + echo "ERROR: docker required for network-none install smoke" >&2 + exit 1 +} + +echo "==> docker pull python:3.12-slim (linux/amd64)" +docker pull --platform linux/amd64 python:3.12-slim + +echo "==> install.sh + logstashui --help / manage check (--network=none)" +docker run --rm \ + --platform linux/amd64 \ + --network=none \ + -v "$STAGE:/offline:ro" \ + -w /tmp \ + python:3.12-slim \ + bash -lc ' + set -euo pipefail + cp -a /offline /tmp/pkg + cd /tmp/pkg + PYTHON=python3 sh install.sh + .venv/bin/logstashui --help + .venv/bin/logstashui manage check + ' + +echo "Wheels freeze smoke passed: $ZIP" diff --git a/docs/docs/logstashui/general/build.md b/docs/docs/logstashui/general/build.md index ee40814..43f6285 100644 --- a/docs/docs/logstashui/general/build.md +++ b/docs/docs/logstashui/general/build.md @@ -84,6 +84,8 @@ uv build The wheel includes systemd templates (`LogstashUI/packaging/`) and the `logstashui` console script. +Air-gapped hosts that cannot reach PyPI or a registry: optional `bin/freeze_logstashui.sh` (not default packaging). See [Air-gapped freeze](/docs/docs/logstashui/general/offline.md). + --- ## Building the Docker Image Locally diff --git a/docs/docs/logstashui/general/deploy.md b/docs/docs/logstashui/general/deploy.md index 2b4798d..c77524b 100644 --- a/docs/docs/logstashui/general/deploy.md +++ b/docs/docs/logstashui/general/deploy.md @@ -10,6 +10,7 @@ The ways to deploy LogstashUI, from the standard Docker install to running from - [Option 3: pip / uv + systemd](#option-3-pip--uv--systemd) - [Option 4: Source Development Setup](#option-4-source-development-setup) - [Option 5: Kubernetes](#option-5-kubernetes) +- [Option 6: Air-gapped freeze (optional)](#option-6-air-gapped-freeze-optional) --- @@ -99,11 +100,20 @@ One-replica StatefulSet, PVC at `/var/lib/logstashui`, image `codyjackson032/log --- +## Option 6: Air-gapped freeze (optional) + +Not the default packaging path. When the install host has **no PyPI and no registry**, a connected maintainer runs `bin/freeze_logstashui.sh` and copies a zip. Isolated host: CPython **3.12** x86_64 (wheels), Docker (image zip), or experimental PyInstaller (Linux x86_64). `[databases]` is included; LogstashAgent is not. + +**📖 Full instructions: [Air-gapped freeze](/docs/docs/logstashui/general/offline.md)** + +--- + ## Related Documentation - **[Getting Started](/docs/docs/getting_started.md)** - Step-by-step standard install - **[Kubernetes](/docs/docs/logstashui/kubernetes/index.md)** - StatefulSet, PVC, Ingress, Envoy Gateway, CNPG - **[Database](/docs/docs/logstashui/database/index.md)** - SQLite, PostgreSQL, MySQL/MariaDB +- **[Air-gapped freeze](/docs/docs/logstashui/general/offline.md)** - Optional offline zip (wheels / Docker / experimental standalone) - **[Building LogstashUI from Source](/docs/docs/logstashui/general/build.md)** - Source builds and local development - **[Host Mode Setup](/docs/docs/logstashui/configuration/host_mode.md)** - High-performance simulation setup - **[Updating LogstashUI](/docs/docs/logstashui/general/updating.md)** - Keeping your deployment current diff --git a/docs/docs/logstashui/general/index.md b/docs/docs/logstashui/general/index.md index 11227c6..3e49f16 100644 --- a/docs/docs/logstashui/general/index.md +++ b/docs/docs/logstashui/general/index.md @@ -13,12 +13,21 @@ All the ways to deploy LogstashUI. - Host-backed simulation - pip / uv + systemd - Kubernetes (see also the [Kubernetes](/docs/docs/logstashui/kubernetes/index.md) subsection) +- Air-gapped freeze (optional offline zips) - Source development setup **📖 [View deployment guide →](/docs/docs/logstashui/general/deploy.md)** --- +## **[Air-gapped freeze](/docs/docs/logstashui/general/offline.md)** + +Optional connected-builder script that freezes LogstashUI plus `[databases]` into zips for hosts with no PyPI or registry. Default `uv build` is unchanged. + +**📖 [View air-gapped freeze →](/docs/docs/logstashui/general/offline.md)** + +--- + ## **[Building from Source](/docs/docs/logstashui/general/build.md)** Instructions for building and running LogstashUI from source. diff --git a/docs/docs/logstashui/general/offline.md b/docs/docs/logstashui/general/offline.md new file mode 100644 index 0000000..d7895c4 --- /dev/null +++ b/docs/docs/logstashui/general/offline.md @@ -0,0 +1,57 @@ +# Air-gapped freeze (optional) + +Build zip files on a **connected** Linux x86_64 machine, copy them to a host with **no PyPI and no container registry**, and run LogstashUI. + +This is **not** the default packaging path and **not** the recommended install when the network can reach GitHub or a registry. Prefer [Option 1: Docker](deploy.md#option-1-standard-docker-deployment-recommended) or [Option 3: pip / uv](deploy.md#option-3-pip--uv--systemd). Default `uv build` (sdist + `py3-none-any` wheel) is unchanged. + +## What you get + +`bin/freeze_logstashui.sh` emits up to three zips under `dist/offline/` (gitignored): + +| Zip | Isolated host needs | Run | +|---|---|---| +| `logstashui-*-offline-wheels-linux-x86_64-cp312.zip` | CPython **3.12** x86_64 + venv | `./install.sh` then `.venv/bin/logstashui serve` | +| `logstashui-*-offline-docker-linux-x86_64.zip` | Docker Engine | `./load.sh` then `docker compose -f compose.offline.yml up -d` | +| `logstashui-*-offline-standalone-linux-x86_64.zip` | glibc Linux x86_64 | `./run.sh` (**experimental**) | + +All three include `LogstashUI[databases]` (psycopg + PyMySQL). SQLite remains the runtime default. **LogstashAgent is not bundled.** arm64 and Windows are later freeze invocations, not this zip. + +## Builder (connected) + +Requirements: Linux x86_64 (wheels can also be downloaded from another OS via pip's `--platform`), CPython 3.12 (`uv python install 3.12`), [uv](https://docs.astral.sh/uv/), Docker for `--docker`, and (on Linux x86_64 only) a throwaway venv for PyInstaller. + +```bash +./bin/freeze_logstashui.sh --wheels +./bin/freeze_logstashui.sh --docker +./bin/freeze_logstashui.sh --standalone # Linux x86_64 only +./bin/freeze_logstashui.sh --all # default if you pass no artifact flags +./bin/freeze_logstashui.sh --docker --image logstashui:offline-0.5.1 +``` + +`--image` saves a **local** tag. The script never `docker pull`. `--standalone` on macOS/Windows/ARM **fails** if you passed that flag; `--all` **skips** it with a warning. + +Wheel policy: **zip contains only `.whl` files**. The builder prefers `manylinux2014` then `manylinux_2_28` cp312 wheels (isolated host needs glibc **2.28+**, e.g. RHEL 8 / Ubuntu 20.04). Pure-Python sdists (no manylinux wheel) are converted to `py3-none-any` on the **connected** builder. A native package with no manylinux wheel fails the freeze — the isolated host has no compiler. Pins come from `uv.lock`. + +Smoke the wheelhouse (pulls `python:3.12-slim` first, then installs with `--network=none`): + +```bash +./bin/test_freeze_wheels.sh +``` + +Optional CI: `.github/workflows/offline-freeze.yml` is `workflow_dispatch` only (not a required PR check). + +## Isolated host + +Same env as a normal install: `LOGSTASHUI_DATA_DIR` (default `$(pwd)/logstashui_data`), `LOGSTASHUI_*`, `LOGSTASHUI_DB_*`. HTTPS on **:8443**. Product CA is created on first start under the data dir — do not expect CA files inside the zip. + +**Wheels:** Debian/Ubuntu need `python3.12` and `python3.12-venv`. `install.sh` uses `pip install --no-index --find-links ./wheels 'LogstashUI[databases]'`. It does **not** upgrade pip (that would hit PyPI). uv is not required. + +**Docker:** UI-only compose (no Agent, no `embedded` profile). Set `ALLOWED_HOSTS` / `LOGSTASHUI_HOST_*` / `LOGSTASHUI_DB_*` as needed. + +**Standalone:** experimental PyInstaller onedir. Treat as a trial until `serve` completes migrate, SNMP official sync, collectstatic, and HTTPS :8443 with no network. Gunicorn stays gevent; do not switch workers as a workaround. + +After a wheelhouse install, `logstashui systemd` still writes the unit and `/etc/default/logstashui` and does **not** enable it. + +## Later + +linux/arm64 and Windows x86_64 as extra freeze tags; a sibling LogstashAgent freeze; promoting standalone off experimental after the serve smoke exists. diff --git a/docs/superpowers/plans/2026-09-02-airgap-freeze.md b/docs/superpowers/plans/2026-09-02-airgap-freeze.md new file mode 100644 index 0000000..247b778 --- /dev/null +++ b/docs/superpowers/plans/2026-09-02-airgap-freeze.md @@ -0,0 +1,34 @@ +# Air-gapped freeze Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Optional `bin/freeze_logstashui.sh` emits Linux x86_64 wheelhouse, Docker, and experimental PyInstaller zips so an air-gapped host can run LogstashUI with no PyPI or registry. + +**Architecture:** Connected builder runs a bash script (not a CLI subcommand). Default `uv build` is unchanged. Isolated helpers (`install.sh` / `load.sh` / `run.sh`) live in each zip. Templates and the PyInstaller spec live in `packaging/offline/`. `os.execvp("gunicorn")` cannot work inside PyInstaller; `cli.py` gets a frozen in-process gunicorn path without changing `--worker-class gevent`. + +**Tech Stack:** bash, uv, pip download (manylinux2014_x86_64 / cp312), Docker save, PyInstaller onedir, MkDocs-style docs. + +**Spec:** `docs/superpowers/specs/2026-09-02-airgap-freeze-design.md` + +--- + +## File map + +| Path | Role | +|---|---| +| `bin/freeze_logstashui.sh` | Builder | +| `bin/test_freeze_wheels.sh` | Unzip + `docker run --platform linux/amd64 --network=none` install smoke | +| `packaging/offline/*` | Templates + PyInstaller spec/entry | +| `src/logstashui/LogstashUI/cli.py` | Frozen gunicorn exec | +| `docs/docs/logstashui/general/offline.md` | Operator + maintainer docs | +| `docs/docs/logstashui/general/deploy.md` | Option 6 | +| `CHANGELOG.md` | 0.5.2 packaging note | +| `.github/workflows/offline-freeze.yml` | `workflow_dispatch` `--wheels` only | + +### Task 1: Templates + freeze script + docs + frozen hook + +Implement per spec. `--all` on non-Linux-x86_64 skips standalone with a warning; explicit `--standalone` fails. Never `docker pull`. `pip download` without `--abi` (so `py3-none-any` wheels are kept). Builder requires `uv python find 3.12`. + +### Task 2: Wheels smoke + +`bin/freeze_logstashui.sh --wheels` then `bin/test_freeze_wheels.sh`. diff --git a/docs/superpowers/specs/2026-09-02-airgap-freeze-design.md b/docs/superpowers/specs/2026-09-02-airgap-freeze-design.md index 955d751..eeea23e 100644 --- a/docs/superpowers/specs/2026-09-02-airgap-freeze-design.md +++ b/docs/superpowers/specs/2026-09-02-airgap-freeze-design.md @@ -38,7 +38,7 @@ A maintainer with internet builds zip files that an air-gapped operator can copy | Isolated run | Each zip has `README.md` + helper (`install.sh` / `load.sh` / `run.sh`). | | Config | Same env as today. No YAML. `LOGSTASHUI_DATA_DIR` default `$(pwd)/logstashui_data`. | | Standalone tool | PyInstaller. **Experimental** until smoke covers migrate + SNMP sync + TLS serve. | -| Wheel policy | **Binary wheels only** (`pip download --only-binary :all:`). Sdist-only dep → freeze **fails**. | +| Wheel policy | Zip contains **only `.whl`**. Prefer manylinux2014 then manylinux_2_28. Pure-Python sdists are wheeled on the builder; native sdist-only → freeze **fails**. | | Pins | Resolve from `uv.lock` (`uv export --frozen`). | | PyInstaller in deps | **No.** Freeze script uses `uvx pyinstaller` (or a throwaway venv). Do not add to `[project]` or default `dev`. | | CI | Optional `workflow_dispatch` only. Not on every PR. | @@ -68,10 +68,9 @@ Output stays under gitignored `/dist/`. 1. Ensure Tailwind CSS exists (`src/logstashui/theme/static/css/dist/styles.css`); build it the same way `uv build` / Dockerfile does if missing. 2. `uv build` (reuse the normal wheel; do not invent a second hatch target). 3. `uv export --frozen --no-dev --extra databases --no-emit-project -o dist/offline/requirements-offline.txt` (keep hashes). -4. Copy the local LogstashUI wheel into `wheels/`. Download deps with **pip** (uv 0.12 has no `uv pip download`): - `uv run --python 3.12 --with pip python -m pip download -r requirements-offline.txt -d wheels/ --python-version 3.12 --platform manylinux2014_x86_64 --implementation cp --abi cp312 --only-binary :all:` +4. Copy the local LogstashUI wheel into `wheels/`. Download deps with `packaging/offline/download_wheels.py` (uv 0.12 has no `uv pip download`): try `manylinux2014_x86_64` then `manylinux_2_28_x86_64` `--only-binary :all:`. Pure-Python sdists (e.g. `django-login-required-middleware==0.9.0`) are wheeled on the **builder** to `py3-none-any`. Native packages with no manylinux wheel fail the freeze. 5. Copy `LICENSE.txt`, `NOTICE.txt`, generated `install.sh`, `README.md`, `MANIFEST.txt`, `SHA256SUMS.txt`. -6. Zip. Fail if any `.tar.gz` sdist landed in `wheels/`. +6. Zip. Fail if any `.tar.gz` sdist landed in `wheels/`. Isolated host needs glibc 2.28+. **MANIFEST.txt** must list: LogstashUI version, git sha (or `unknown` if not a git checkout), CPython 3.12, platform `linux-x86_64`, extra `databases`, package list with versions. diff --git a/packaging/offline/README.md b/packaging/offline/README.md new file mode 100644 index 0000000..82c439e --- /dev/null +++ b/packaging/offline/README.md @@ -0,0 +1,5 @@ +# Offline freeze templates + +Used by `bin/freeze_logstashui.sh`. Placeholders `__VERSION__`, `__GIT_SHA__`, `__IMAGE_NAME__` are substituted at freeze time. + +These files are **not** shipped in the default hatchling wheel (systemd templates stay in `src/logstashui/LogstashUI/packaging/`). diff --git a/packaging/offline/compose.offline.yml b/packaging/offline/compose.offline.yml new file mode 100644 index 0000000..7529312 --- /dev/null +++ b/packaging/offline/compose.offline.yml @@ -0,0 +1,34 @@ +# Air-gapped UI-only compose. No Agent, no embedded profile, no registry pull. +# Load the image first: ./load.sh +# Image: __IMAGE_NAME__ + +name: logstashui-offline + +services: + logstashui: + image: __IMAGE_NAME__ + ports: + - "8443:8443" + volumes: + - logstashui_data:/var/lib/logstashui + environment: + DEBUG: "False" + LOGSTASHUI_DATA_DIR: /var/lib/logstashui + LOGSTASHUI_TLS: "true" + ALLOWED_HOSTS: ${ALLOWED_HOSTS:-*} + CSRF_TRUSTED_ORIGINS: ${CSRF_TRUSTED_ORIGINS:-https://localhost:8443,https://127.0.0.1:8443} + LOGSTASHUI_HOST_HOSTNAME: ${LOGSTASHUI_HOST_HOSTNAME:-} + LOGSTASHUI_HOST_IPS: ${LOGSTASHUI_HOST_IPS:-} + LOGSTASHUI_TLS_SANS: ${LOGSTASHUI_TLS_SANS:-} + LOGSTASHUI_DB_ENGINE: ${LOGSTASHUI_DB_ENGINE:-sqlite} + LOGSTASHUI_DB_HOST: ${LOGSTASHUI_DB_HOST:-} + LOGSTASHUI_DB_PORT: ${LOGSTASHUI_DB_PORT:-} + LOGSTASHUI_DB_NAME: ${LOGSTASHUI_DB_NAME:-} + LOGSTASHUI_DB_USER: ${LOGSTASHUI_DB_USER:-} + LOGSTASHUI_DB_PASSWORD: ${LOGSTASHUI_DB_PASSWORD:-} + PUID: ${PUID:-} + PGID: ${PGID:-} + restart: unless-stopped + +volumes: + logstashui_data: diff --git a/packaging/offline/docker-README.md b/packaging/offline/docker-README.md new file mode 100644 index 0000000..026408b --- /dev/null +++ b/packaging/offline/docker-README.md @@ -0,0 +1,16 @@ +# LogstashUI air-gapped Docker image (__VERSION__) + +Linux **x86_64** image tarball. No registry. The image already includes `LogstashUI[databases]`. There is **no** LogstashAgent in this zip. + +## Load and run + +```sh +./load.sh +docker compose -f compose.offline.yml up -d +``` + +`load.sh` never pulls. HTTPS UI is **:8443**. Named volume `logstashui_data` is `LOGSTASHUI_DATA_DIR=/var/lib/logstashui`. Keep `LOGSTASHUI_TLS` on. + +Set `ALLOWED_HOSTS`, `LOGSTASHUI_HOST_*`, and `LOGSTASHUI_DB_*` in the environment or a `.env` next to `compose.offline.yml` as needed. + +Git: `__GIT_SHA__` diff --git a/packaging/offline/docker-load.sh b/packaging/offline/docker-load.sh new file mode 100755 index 0000000..f9dc0a4 --- /dev/null +++ b/packaging/offline/docker-load.sh @@ -0,0 +1,27 @@ +#!/bin/sh +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +set -eu + +HERE=$(CDPATH= cd -- "$(dirname "$0")" && pwd) +TAR="$HERE/image.tar.gz" +IMAGE="__IMAGE_NAME__" + +if ! command -v docker >/dev/null 2>&1; then + echo "docker is required to load this image" >&2 + exit 1 +fi + +[ -f "$TAR" ] || { + echo "missing $TAR" >&2 + exit 1 +} + +echo "Loading $TAR ..." +docker load -i "$TAR" +echo "Loaded image: $IMAGE" +echo "Start (UI only, no Agent):" +echo " docker compose -f \"$HERE/compose.offline.yml\" up -d" +echo "Browse https://:8443 — data volume logstashui_data" diff --git a/packaging/offline/download_wheels.py b/packaging/offline/download_wheels.py new file mode 100644 index 0000000..48cea12 --- /dev/null +++ b/packaging/offline/download_wheels.py @@ -0,0 +1,176 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +"""Download cp312 manylinux wheels. Pure-Python sdists are wheeled on the builder.""" + +from __future__ import annotations + +import argparse +import re +import subprocess +import sys +import tempfile +from pathlib import Path + +REQ_LINE = re.compile( + r"^(?P[A-Za-z0-9_.-]+)==(?P[^;\\\s]+)(?:\s*;\s*(?P.+))?$" +) + + +def parse_requirements(path: Path) -> list[tuple[str, str, str | None]]: + pkgs: list[tuple[str, str, str | None]] = [] + pending = "" + for raw in path.read_text(encoding="utf-8").splitlines(): + line = raw.strip() + if not line or line.startswith("#") or line.startswith("--"): + continue + if line.endswith("\\"): + pending += line[:-1].strip() + " " + continue + line = (pending + line).strip() + pending = "" + # Drop hash fragments pip export may still leave on the same line. + line = re.sub(r"\s+--hash=\S+", "", line).strip() + m = REQ_LINE.match(line) + if not m: + continue + marker = (m.group("marker") or "").strip() or None + pkgs.append((m.group("name"), m.group("ver"), marker)) + return pkgs + + +def pip(*args: str) -> subprocess.CompletedProcess[str]: + return subprocess.run( + [sys.executable, "-m", "pip", *args], + text=True, + capture_output=True, + ) + + +def is_linux_x86_wheel(name: str) -> bool: + n = name.lower() + if n.endswith(".tar.gz") or n.endswith(".zip"): + return False + if "macosx" in n or "win_amd64" in n or "win32" in n or "aarch64" in n: + return False + return n.endswith(".whl") and ( + "manylinux" in n or "linux_x86_64" in n or "none-any" in n + ) + + +PLATFORMS = ( + "manylinux2014_x86_64", + "manylinux_2_28_x86_64", +) + + +def download_binary(name: str, ver: str, dest: Path) -> bool: + spec = f"{name}=={ver}" + common = [ + "download", + spec, + "-d", + str(dest), + "--no-deps", + "--python-version", + "3.12", + "--only-binary", + ":all:", + "--disable-pip-version-check", + ] + for platform in PLATFORMS: + proc = pip(*common, "--platform", platform) + if proc.returncode == 0: + return True + # py3-none-any (no --platform): login-required-middleware 0.8 has a wheel; 0.9 does not. + proc = pip(*common) + return proc.returncode == 0 + + +def wheel_sdist(name: str, ver: str, dest: Path) -> None: + spec = f"{name}=={ver}" + with tempfile.TemporaryDirectory() as tmp: + tmp_path = Path(tmp) + proc = pip( + "download", + spec, + "-d", + str(tmp_path), + "--no-deps", + "--no-binary", + ":all:", + "--disable-pip-version-check", + ) + if proc.returncode != 0: + sys.stderr.write(proc.stdout) + sys.stderr.write(proc.stderr) + raise SystemExit( + f"no manylinux cp312 wheel and sdist download failed for {spec}" + ) + sdists = list(tmp_path.glob("*.tar.gz")) + list(tmp_path.glob("*.zip")) + if not sdists: + raise SystemExit(f"sdist missing after download for {spec}") + proc = pip( + "wheel", + "--no-deps", + "--no-cache-dir", + "-w", + str(tmp_path), + str(sdists[0]), + "--disable-pip-version-check", + ) + if proc.returncode != 0: + sys.stderr.write(proc.stdout) + sys.stderr.write(proc.stderr) + raise SystemExit(f"could not build a wheel from sdist for {spec}") + wheels = [p for p in tmp_path.glob("*.whl") if is_linux_x86_wheel(p.name)] + if not wheels: + built = list(tmp_path.glob("*.whl")) + raise SystemExit( + f"{spec} is sdist-only and built {built or 'no wheel'} " + f"(need py3-none-any or manylinux x86_64). Freeze aborted." + ) + target = dest / wheels[0].name + target.write_bytes(wheels[0].read_bytes()) + print(f"built wheel from sdist: {wheels[0].name}", flush=True) + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("requirements") + parser.add_argument("dest") + args = parser.parse_args() + dest = Path(args.dest) + dest.mkdir(parents=True, exist_ok=True) + pkgs = parse_requirements(Path(args.requirements)) + if not pkgs: + raise SystemExit(f"no packages parsed from {args.requirements}") + for name, ver, marker in pkgs: + marker_norm = (marker or "").replace('"', "'") + if "sys_platform == 'win32'" in marker_norm or "sys_platform == 'darwin'" in marker_norm: + print(f"skip {name}=={ver} (marker {marker})", flush=True) + continue + spec = f"{name}=={ver}" + print(f"download {spec}", flush=True) + if download_binary(name, ver, dest): + continue + print(f"binary miss {spec}; trying sdist→wheel", flush=True) + wheel_sdist(name, ver, dest) + bad = [ + p.name + for p in dest.iterdir() + if p.is_file() and not is_linux_x86_wheel(p.name) and p.suffix != ".txt" + ] + # Allow the LogstashUI wheel already copied in (py3-none-any). + leftover_sdists = list(dest.glob("*.tar.gz")) + list(dest.glob("*.zip")) + if leftover_sdists: + names = ", ".join(p.name for p in leftover_sdists) + raise SystemExit(f"sdist left in wheelhouse: {names}") + if bad: + raise SystemExit(f"non-linux-x86_64 artifacts: {', '.join(bad)}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/packaging/offline/entry.py b/packaging/offline/entry.py new file mode 100644 index 0000000..c75fff8 --- /dev/null +++ b/packaging/offline/entry.py @@ -0,0 +1,10 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +"""PyInstaller entry for the experimental standalone freeze.""" + +from LogstashUI.cli import main + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/packaging/offline/logstashui.spec b/packaging/offline/logstashui.spec new file mode 100644 index 0000000..e432a1a --- /dev/null +++ b/packaging/offline/logstashui.spec @@ -0,0 +1,99 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +# PyInstaller onedir spec for LogstashUI. Experimental: gunicorn+gevent+Django +# hidden imports. Do not change product --worker-class gevent from here. +# -*- mode: python ; coding: utf-8 -*- + +from __future__ import annotations + +import os + +from PyInstaller.building.api import COLLECT, EXE, PYZ +from PyInstaller.building.build_main import Analysis +from PyInstaller.utils.hooks import collect_all + +SPECDIR = os.path.dirname(os.path.abspath(SPEC)) +ROOT = os.path.abspath(os.path.join(SPECDIR, "..", "..")) +SRC = os.path.join(ROOT, "src", "logstashui") +ENTRY = os.path.join(SPECDIR, "entry.py") + +PACKAGES = [ + "LogstashUI", + "PipelineManager", + "Management", + "Utilities", + "SNMP", + "Monitoring", + "Site", + "Documentation", + "AI", + "theme", + "Common", + "django", + "gunicorn", + "gevent", + "greenlet", + "cryptography", + "pysnmp", + "lark", + "pygrok", + "whitenoise", + "psycopg", + "pymysql", + "yaml", + "django_htmx", + "tailwind", + "login_required", + "elasticsearch", + "requests", + "markdown", + "packaging", + "django_browser_reload", +] + +datas: list = [] +binaries: list = [] +hiddenimports: list = [] +for pkg in PACKAGES: + try: + d, b, h = collect_all(pkg) + except Exception: + continue + datas += d + binaries += b + hiddenimports += h + +a = Analysis( + [ENTRY], + pathex=[SRC], + binaries=binaries, + datas=datas, + hiddenimports=hiddenimports, + hookspath=[], + runtime_hooks=[], + excludes=[], + noarchive=False, +) +pyz = PYZ(a.pure) +exe = EXE( + pyz, + a.scripts, + [], + exclude_binaries=True, + name="logstashui", + debug=False, + bootloader_ignore_signals=False, + strip=False, + upx=False, + console=True, +) +coll = COLLECT( + exe, + a.binaries, + a.datas, + strip=False, + upx=False, + name="logstashui", +) diff --git a/packaging/offline/standalone-README.md b/packaging/offline/standalone-README.md new file mode 100644 index 0000000..2c21d2f --- /dev/null +++ b/packaging/offline/standalone-README.md @@ -0,0 +1,19 @@ +# LogstashUI experimental standalone (__VERSION__) + +**Experimental.** PyInstaller onedir for Linux **x86_64**. No Python on the host. Not production-supported until `./logstashui serve` completes migrate, SNMP sync, collectstatic, and HTTPS :8443 without network. + +Gunicorn stays `--worker-class gevent`. If this bundle cannot serve, that is a freeze bug — do not switch workers as a workaround without amending the design spec. + +## Run + +```sh +./run.sh +# or: +./logstashui/logstashui serve +``` + +Data dir default: `$(pwd)/logstashui_data`. Same `LOGSTASHUI_*` env as a wheel/Docker install. + +systemd: set `ExecStart=` to the unpacked `logstashui/logstashui serve`. This zip does not write units. + +Git: `__GIT_SHA__` diff --git a/packaging/offline/standalone-run.sh b/packaging/offline/standalone-run.sh new file mode 100755 index 0000000..955b841 --- /dev/null +++ b/packaging/offline/standalone-run.sh @@ -0,0 +1,16 @@ +#!/bin/sh +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +set -eu + +HERE=$(CDPATH= cd -- "$(dirname "$0")" && pwd) +BIN="$HERE/logstashui/logstashui" + +if [ ! -x "$BIN" ]; then + echo "missing $BIN" >&2 + exit 1 +fi + +exec "$BIN" serve "$@" diff --git a/packaging/offline/wheels-README.md b/packaging/offline/wheels-README.md new file mode 100644 index 0000000..161f987 --- /dev/null +++ b/packaging/offline/wheels-README.md @@ -0,0 +1,35 @@ +# LogstashUI air-gapped wheelhouse (__VERSION__) + +Linux **x86_64**, CPython **3.12** only. No PyPI. `[databases]` extras (psycopg + PyMySQL) are in `wheels/`. SQLite is still the runtime default. + +This zip is **not** the recommended install when the host can reach GitHub or a container registry. Prefer Docker Compose (connected) or `pip install` of the normal wheel. + +## Host packages + +- CPython 3.12 (64-bit x86_64) +- Distro venv module (Debian/Ubuntu: `python3.12-venv`) + +uv is **not** required. + +## Install and run + +```sh +./install.sh +.venv/bin/logstashui serve +``` + +Override the interpreter with `PYTHON=/path/to/python3.12 ./install.sh`. + +HTTPS UI is **:8443**. Data dir default is `$(pwd)/logstashui_data` (`LOGSTASHUI_DATA_DIR`). Same env vars as a normal install (`LOGSTASHUI_*`, `LOGSTASHUI_DB_*`). + +systemd (does **not** enable the unit): + +```sh +sudo .venv/bin/logstashui systemd +``` + +## Not included + +LogstashAgent. Enroll agents separately. arm64 / Windows freezes are not this zip. + +Git: `__GIT_SHA__` diff --git a/packaging/offline/wheels-install.sh b/packaging/offline/wheels-install.sh new file mode 100755 index 0000000..61ca066 --- /dev/null +++ b/packaging/offline/wheels-install.sh @@ -0,0 +1,56 @@ +#!/bin/sh +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +# Air-gapped install: CPython 3.12 x86_64 venv + pip --no-index. +# Do not upgrade pip (that hits PyPI). + +set -eu + +HERE=$(CDPATH= cd -- "$(dirname "$0")" && pwd) +WHEELS="$HERE/wheels" +PYTHON="${PYTHON:-python3.12}" + +die() { + printf '%s\n' "$*" >&2 + exit 1 +} + +if ! command -v "$PYTHON" >/dev/null 2>&1; then + if command -v python3 >/dev/null 2>&1; then + PYTHON=python3 + else + die "need CPython 3.12 x86_64 (python3.12 not found). On Debian/Ubuntu: apt install python3.12 python3.12-venv" + fi +fi + +"$PYTHON" - <<'PY' || die "need CPython 3.12 x86_64" +import platform +import sys + +if sys.version_info[:2] != (3, 12): + sys.exit(1) +if sys.maxsize <= 2**32: + sys.exit(1) +mach = platform.machine().lower() +if mach not in ("x86_64", "amd64"): + sys.exit(1) +PY + +[ -d "$WHEELS" ] || die "missing $WHEELS" + +VENV="$HERE/.venv" +"$PYTHON" -m venv "$VENV" +"$VENV/bin/python" -m pip install \ + --disable-pip-version-check \ + --no-index \ + --no-cache-dir \ + --find-links "$WHEELS" \ + 'LogstashUI[databases]' + +printf '\nInstalled into %s\n' "$VENV" +printf 'Start:\n' +printf ' %s/bin/logstashui serve\n' "$VENV" +printf 'Data directory default: $(pwd)/logstashui_data (override LOGSTASHUI_DATA_DIR)\n' +printf 'Configuration is environment variables only. See README.md.\n' diff --git a/src/logstashui/LogstashUI/cli.py b/src/logstashui/LogstashUI/cli.py index 212ca96..c0414d1 100644 --- a/src/logstashui/LogstashUI/cli.py +++ b/src/logstashui/LogstashUI/cli.py @@ -380,6 +380,22 @@ def _check_db_floor() -> None: check_server_version(connection) +def _exec_gunicorn(gunicorn_cmd: list[str]) -> int: + """Replace this process with gunicorn, or run it in-process when frozen. + + PyInstaller onedir has no ``gunicorn`` console script on PATH. Calling + gunicorn's WSGI app in-process keeps ``--worker-class gevent``. + """ + if getattr(sys, "frozen", False): + from gunicorn.app.wsgiapp import run as gunicorn_run + + sys.argv = list(gunicorn_cmd) + result = gunicorn_run() + return int(result or 0) + os.execvp("gunicorn", gunicorn_cmd) + return 1 + + def cmd_serve(args: argparse.Namespace) -> int: from LogstashUI.database import canonical_engine from LogstashUI.paths import resolve_data_dir @@ -447,8 +463,7 @@ def cmd_serve(args: argparse.Namespace) -> int: fullchain.write_bytes(cert.read_bytes()) gunicorn_cmd += ["--certfile", str(fullchain), "--keyfile", str(key)] - os.execvp("gunicorn", gunicorn_cmd) - return 1 + return _exec_gunicorn(gunicorn_cmd) def cmd_systemd(args: argparse.Namespace) -> int: diff --git a/src/logstashui/LogstashUI/tests/test_cli.py b/src/logstashui/LogstashUI/tests/test_cli.py index b7c24c3..ba63103 100644 --- a/src/logstashui/LogstashUI/tests/test_cli.py +++ b/src/logstashui/LogstashUI/tests/test_cli.py @@ -6,7 +6,7 @@ from django.core.management.base import CommandError -from LogstashUI.cli import build_parser, cmd_serve, install_systemd +from LogstashUI.cli import _exec_gunicorn, build_parser, cmd_serve, install_systemd def test_parser_defaults_to_serve(): @@ -209,3 +209,22 @@ def fake_execvp(file, args): err = capsys.readouterr().err assert "SQLite is the small-install default" in err + +def test_exec_gunicorn_frozen_runs_in_process(monkeypatch): + import sys + + monkeypatch.setattr(sys, "frozen", True, raising=False) + seen = {} + + def fake_run(): + seen["argv"] = list(sys.argv) + return 0 + + monkeypatch.setattr("gunicorn.app.wsgiapp.run", fake_run) + rc = _exec_gunicorn( + ["gunicorn", "LogstashUI.wsgi:application", "--bind", "0.0.0.0:8443"] + ) + assert rc == 0 + assert seen["argv"][0] == "gunicorn" + assert "LogstashUI.wsgi:application" in seen["argv"] + From ea656757e3f2dc8ae50db0036cfdd5d42026a563 Mon Sep 17 00:00:00 2001 From: Aaron Mildenstein Date: Wed, 2 Sep 2026 17:29:18 -0600 Subject: [PATCH 24/62] Moved these files to be not tracked --- .../plans/2026-08-21-multi-database.md | 1731 ----------------- .../plans/2026-09-02-airgap-freeze.md | 34 - .../specs/2026-08-21-multi-database-design.md | 266 --- .../specs/2026-09-02-airgap-freeze-design.md | 181 -- 4 files changed, 2212 deletions(-) delete mode 100644 docs/superpowers/plans/2026-08-21-multi-database.md delete mode 100644 docs/superpowers/plans/2026-09-02-airgap-freeze.md delete mode 100644 docs/superpowers/specs/2026-08-21-multi-database-design.md delete mode 100644 docs/superpowers/specs/2026-09-02-airgap-freeze-design.md diff --git a/docs/superpowers/plans/2026-08-21-multi-database.md b/docs/superpowers/plans/2026-08-21-multi-database.md deleted file mode 100644 index 8b78d94..0000000 --- a/docs/superpowers/plans/2026-08-21-multi-database.md +++ /dev/null @@ -1,1731 +0,0 @@ -# Multi-database (SQLite | PostgreSQL | MariaDB/MySQL) Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Let LogstashUI 0.5.2 run on SQLite (default), PostgreSQL, or MariaDB/MySQL via discrete `LOGSTASHUI_DB_*` env vars, with a BETA sqlite→server migrator and a local Docker test matrix. - -**Architecture:** Do not add a CRUD/repository layer. Django ORM already abstracts queries. The only switch is `LogstashUI/database.py` → `DATABASES['default']`. Drivers are install extras; the Docker image installs `[databases]`. Gunicorn stays gevent. `psycopg[binary]>=3.2` cooperates with gunicorn’s gevent `patch_select` (psycopg ≥ 3.1.14); PyMySQL is the MySQL driver so the C `mysqlclient` does not block the hub. Migration is `dumpdata` / `migrate` / `loaddata` in **child processes** so Django settings never have to switch engines in-process. - -**Tech Stack:** Django 6, Python 3.12–3.14, `psycopg[binary]`, PyMySQL, gunicorn/gevent, Docker Compose for Postgres 16 + MariaDB 11 + MySQL 8.0. - -**Spec:** `docs/superpowers/specs/2026-08-21-multi-database-design.md` - -**Context:** Work on the current branch (`feat/sqleng` or whatever the operator is on). Do not rotate the product CA. Do not put Postgres/MariaDB/MySQL in the smoke compose. Do not “fix” pre-existing pytest failures (`test_update_policy_default_policy_forbidden`, `test_delete_policy_default_policy_forbidden`, `test_clone_policy_success`) unless they fail **because of** engine differences. - -Every new/edited `.py` file must keep the Elastic license header already used in this repo. - ---- - -## File map - -| File | Responsibility | -|---|---| -| `pyproject.toml` | Optional extras `[postgres]`, `[mysql]`, `[databases]` | -| `src/logstashui/LogstashUI/database.py` | Engine aliases, `DATABASES` dict, fail-fast, SSL/CONN, driver import, server version check | -| `src/logstashui/LogstashUI/migrate_engine.py` | BETA migrator: pidfile SIGTERM, WAL checkpoint, dump/load subprocesses, `--write-env` | -| `src/logstashui/LogstashUI/cli.py` | `migrate-engine` subcommand, gunicorn `--pid`, SQLite scale warning, systemd DB prompts | -| `src/logstashui/LogstashUI/wsgi.py` | Comment only: gevent+psycopg is automatic at ≥ 3.1.14 | -| `src/logstashui/LogstashUI/packaging/logstashui.default` | Documented `LOGSTASHUI_DB_*` keys | -| `src/logstashui/LogstashUI/tests/test_database.py` | Unit tests for `build_databases` / version check (no live server) | -| `src/logstashui/LogstashUI/tests/test_migrate_engine.py` | Unit tests for migrator (mocked subprocess / pid) | -| `src/logstashui/LogstashUI/tests/test_cli.py` | Parser + serve pid + warning | -| `src/logstashui/LogstashUI/tests/test_migrate_live.py` | Live dump/load; skipped unless `LOGSTASHUI_LIVE_DB=1` | -| `docker/docker-compose.db.yml` | Postgres, MariaDB, MySQL for local/CI | -| `bin/test_databases.sh` / `bin/test_databases.bat` | SQLite pytest + three-engine pytest + live migrator | -| `docker/Dockerfile` | `uv pip install '/app[databases]'` | -| `docs/docs/logstashui/configuration/environment.md` | Env table + extras + scale warning + optional PgBouncer | -| `docs/docs/logstashui/general/deploy.md` | External DB + PVC still required + offline migrate | -| `CHANGELOG.md` | 0.5.2 section | -| `.github/workflows/test-databases.yml` | CI runs `bin/test_databases.sh` | -| `scripts/generate_notice.py` | Map psycopg / PyMySQL licenses | - -No new DAO modules. No YAML. No `DATABASE_URL`. - ---- - -### Task 1: Install extras in pyproject.toml - -**Files:** -- Modify: `pyproject.toml` -- Modify: `uv.lock` (via `uv lock`) -- Modify: `scripts/generate_notice.py` (repository mappings only) - -- [ ] **Step 1: Add optional-dependencies after `[project.urls]`** - -In `pyproject.toml`, immediately after the `[project.urls]` block, insert: - -```toml -[project.optional-dependencies] -postgres = [ - "psycopg[binary]>=3.2.0", -] -mysql = [ - "PyMySQL>=1.1.1", -] -databases = [ - "psycopg[binary]>=3.2.0", - "PyMySQL>=1.1.1", -] -``` - -Do **not** add these to the default `[project].dependencies` list. Default `pip install LogstashUI` must stay SQLite-only. - -- [ ] **Step 2: Map licenses so NOTICE can mention extras** - -In `scripts/generate_notice.py`, add to `REPOSITORY_MAPPINGS`: - -```python - "psycopg": "https://github.com/psycopg/psycopg/blob/master/LICENSE.txt", - "psycopg-binary": "https://github.com/psycopg/psycopg/blob/master/LICENSE.txt", - "PyMySQL": "https://github.com/PyMySQL/PyMySQL/blob/main/LICENSE", -``` - -And to `CUSTOM_DEPENDENCIES` so extras are documented even when not in the default wheel: - -```python - "psycopg": "https://github.com/psycopg/psycopg/blob/master/LICENSE.txt", - "PyMySQL": "https://github.com/PyMySQL/PyMySQL/blob/main/LICENSE", -``` - -- [ ] **Step 3: Lock** - -Run: - -```bash -uv lock -``` - -Expected: `uv.lock` updates; no change to the default resolved production set beyond optional extra packages. - -- [ ] **Step 4: Commit** - -```bash -git add pyproject.toml uv.lock scripts/generate_notice.py -git commit -m "build: add postgres/mysql/databases install extras" -``` - ---- - -### Task 2: Failing unit tests for `build_databases` - -**Files:** -- Modify: `src/logstashui/LogstashUI/tests/test_database.py` - -- [ ] **Step 1: Replace `test_database.py` with the suite below** - -Overwrite `src/logstashui/LogstashUI/tests/test_database.py` with: - -```python -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. - -from pathlib import Path - -import pytest - -from LogstashUI.database import ( - build_databases, - canonical_engine, - check_server_version, -) - - -def _clear_db_env(monkeypatch): - for name in ( - "LOGSTASHUI_DB_ENGINE", - "LOGSTASHUI_DB_NAME", - "LOGSTASHUI_DB_HOST", - "LOGSTASHUI_DB_PORT", - "LOGSTASHUI_DB_USER", - "LOGSTASHUI_DB_PASSWORD", - "LOGSTASHUI_DB_SSLMODE", - "LOGSTASHUI_DB_SSL_CA", - "LOGSTASHUI_DB_CONN_MAX_AGE", - "LOGSTASHUI_DB_CONN_HEALTH_CHECKS", - ): - monkeypatch.delenv(name, raising=False) - - -def test_build_databases_sqlite_default(tmp_path, monkeypatch): - _clear_db_env(monkeypatch) - db = build_databases(tmp_path) - assert db["default"]["ENGINE"] == "django.db.backends.sqlite3" - assert db["default"]["NAME"] == tmp_path / "db.sqlite3" - assert db["default"]["OPTIONS"]["timeout"] == 20 - assert "PRAGMA journal_mode=WAL" in db["default"]["OPTIONS"]["init_command"] - - -@pytest.mark.parametrize( - "raw,expected", - [ - ("", "sqlite"), - ("sqlite", "sqlite"), - ("sqlite3", "sqlite"), - ("postgres", "postgresql"), - ("postgresql", "postgresql"), - ("mysql", "mysql"), - ("mariadb", "mysql"), - ("my", "mysql"), - ("POSTGRESQL", "postgresql"), - ], -) -def test_canonical_engine_aliases(raw, expected): - assert canonical_engine(raw) == expected - - -def test_unknown_engine_fails(tmp_path, monkeypatch): - _clear_db_env(monkeypatch) - monkeypatch.setenv("LOGSTASHUI_DB_ENGINE", "oracle") - with pytest.raises(RuntimeError, match="Unknown LOGSTASHUI_DB_ENGINE"): - build_databases(tmp_path) - - -def test_postgresql_requires_host_user(tmp_path, monkeypatch): - _clear_db_env(monkeypatch) - monkeypatch.setenv("LOGSTASHUI_DB_ENGINE", "postgresql") - monkeypatch.setattr("LogstashUI.database._import_or_raise", lambda *a, **k: None) - with pytest.raises(RuntimeError, match="LOGSTASHUI_DB_HOST"): - build_databases(tmp_path) - monkeypatch.setenv("LOGSTASHUI_DB_HOST", "db.example") - with pytest.raises(RuntimeError, match="LOGSTASHUI_DB_USER"): - build_databases(tmp_path) - - -def test_build_databases_postgresql(tmp_path, monkeypatch): - _clear_db_env(monkeypatch) - monkeypatch.setenv("LOGSTASHUI_DB_ENGINE", "postgres") - monkeypatch.setenv("LOGSTASHUI_DB_HOST", "db.example") - monkeypatch.setenv("LOGSTASHUI_DB_USER", "lsui") - monkeypatch.setenv("LOGSTASHUI_DB_PASSWORD", "s3cret") - monkeypatch.setenv("LOGSTASHUI_DB_SSLMODE", "require") - monkeypatch.setenv("LOGSTASHUI_DB_SSL_CA", "/etc/ssl/db-ca.pem") - monkeypatch.setattr("LogstashUI.database._import_or_raise", lambda *a, **k: None) - db = build_databases(tmp_path)["default"] - assert db["ENGINE"] == "django.db.backends.postgresql" - assert db["NAME"] == "logstashui" - assert db["HOST"] == "db.example" - assert db["PORT"] == "5432" - assert db["USER"] == "lsui" - assert db["PASSWORD"] == "s3cret" - assert db["CONN_MAX_AGE"] == 60 - assert db["CONN_HEALTH_CHECKS"] is True - assert db["OPTIONS"]["sslmode"] == "require" - assert db["OPTIONS"]["sslrootcert"] == "/etc/ssl/db-ca.pem" - - -def test_build_databases_mysql_mariadb_alias(tmp_path, monkeypatch): - _clear_db_env(monkeypatch) - monkeypatch.setenv("LOGSTASHUI_DB_ENGINE", "mariadb") - monkeypatch.setenv("LOGSTASHUI_DB_HOST", "127.0.0.1") - monkeypatch.setenv("LOGSTASHUI_DB_USER", "lsui") - monkeypatch.setenv("LOGSTASHUI_DB_PORT", "3307") - monkeypatch.setenv("LOGSTASHUI_DB_CONN_MAX_AGE", "0") - monkeypatch.setenv("LOGSTASHUI_DB_CONN_HEALTH_CHECKS", "false") - monkeypatch.setattr("LogstashUI.database._import_or_raise", lambda *a, **k: None) - db = build_databases(tmp_path)["default"] - assert db["ENGINE"] == "django.db.backends.mysql" - assert db["PORT"] == "3307" - assert db["CONN_MAX_AGE"] == 0 - assert db["CONN_HEALTH_CHECKS"] is False - assert db["OPTIONS"]["charset"] == "utf8mb4" - assert "utf8mb4_bin" in db["OPTIONS"]["init_command"] - assert db["TEST"]["CHARSET"] == "utf8mb4" - assert db["TEST"]["COLLATION"] == "utf8mb4_bin" - - -def test_postgresql_missing_driver(tmp_path, monkeypatch): - _clear_db_env(monkeypatch) - monkeypatch.setenv("LOGSTASHUI_DB_ENGINE", "postgresql") - monkeypatch.setenv("LOGSTASHUI_DB_HOST", "localhost") - monkeypatch.setenv("LOGSTASHUI_DB_USER", "lsui") - - def boom(module, extra): - raise RuntimeError( - f"{module} is not installed. Install with: uv pip install 'LogstashUI[{extra}]'" - ) - - monkeypatch.setattr("LogstashUI.database._import_or_raise", boom) - with pytest.raises(RuntimeError, match=r"LogstashUI\[postgres\]"): - build_databases(tmp_path) - - -def test_check_server_version_sqlite_noop(): - class Conn: - vendor = "sqlite" - - check_server_version(Conn()) - - -def test_check_server_version_postgres_too_old(): - class Conn: - vendor = "postgresql" - pg_version = 130000 - - with pytest.raises(RuntimeError, match="PostgreSQL 14"): - check_server_version(Conn()) - - -def test_check_server_version_mysql_and_mariadb(): - class Mysql: - vendor = "mysql" - mysql_is_mariadb = False - mysql_server_info = "8.0.36" - - def get_database_version(self): - return (8, 0, 36) - - check_server_version(Mysql()) - - class OldMysql: - vendor = "mysql" - mysql_is_mariadb = False - mysql_server_info = "5.7.44" - - def get_database_version(self): - return (5, 7, 44) - - with pytest.raises(RuntimeError, match="MySQL 8.0"): - check_server_version(OldMysql()) - - class Maria: - vendor = "mysql" - mysql_is_mariadb = True - mysql_server_info = "10.5.22-MariaDB" - - def get_database_version(self): - return (10, 5, 22) - - with pytest.raises(RuntimeError, match="MariaDB 10.6"): - check_server_version(Maria()) -``` - -- [ ] **Step 2: Run tests to verify they fail** - -```bash -cd /Users/buh/WORK/LogstashUI -uv run pytest src/logstashui/LogstashUI/tests/test_database.py -v --no-cov -``` - -Expected: FAIL — `canonical_engine`, `_import_or_raise`, and `check_server_version` are not defined; postgresql/mysql still raise “not implemented”. - -- [ ] **Step 3: Commit tests** - -```bash -git add src/logstashui/LogstashUI/tests/test_database.py -git commit -m "test: specify multi-engine build_databases behavior" -``` - ---- - -### Task 3: Implement `build_databases` and version check - -**Files:** -- Modify: `src/logstashui/LogstashUI/database.py` - -- [ ] **Step 1: Replace `database.py`** - -Overwrite `src/logstashui/LogstashUI/database.py` with: - -```python -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. - -"""Build Django DATABASES from discrete LOGSTASHUI_DB_* environment variables.""" - -from __future__ import annotations - -import os -from pathlib import Path - -from .config import env_bool - -_ENGINE_ALIASES = { - "": "sqlite", - "sqlite": "sqlite", - "sqlite3": "sqlite", - "postgres": "postgresql", - "postgresql": "postgresql", - "mysql": "mysql", - "mariadb": "mysql", - "my": "mysql", -} - -_CANONICAL = ("sqlite", "postgresql", "mysql") - - -def canonical_engine(raw: str | None) -> str: - key = (raw or "").strip().lower() - if key not in _ENGINE_ALIASES: - supported = "sqlite, postgresql, mysql (aliases: sqlite3, postgres, mariadb, my)" - raise RuntimeError( - f"Unknown LOGSTASHUI_DB_ENGINE={raw!r}. Supported: {supported}." - ) - return _ENGINE_ALIASES[key] - - -def _import_or_raise(module: str, extra: str) -> None: - try: - __import__(module) - except ImportError as exc: - raise RuntimeError( - f"{module} is not installed. Install with: uv pip install 'LogstashUI[{extra}]' " - f"(Docker/K8s image already includes LogstashUI[databases])." - ) from exc - - -def _env(name: str, default: str = "") -> str: - raw = os.environ.get(name) - if raw is None: - return default - return raw.strip() - - -def _env_int(name: str, default: int) -> int: - raw = _env(name, "") - if raw == "": - return default - return int(raw) - - -def _require(names: list[str]) -> None: - missing = [n for n in names if not _env(n)] - if missing: - raise RuntimeError( - "Missing required database settings: " + ", ".join(missing) - ) - - -def build_databases(data_dir: Path) -> dict: - engine = canonical_engine(os.environ.get("LOGSTASHUI_DB_ENGINE")) - conn_max_age = _env_int("LOGSTASHUI_DB_CONN_MAX_AGE", 60) - health = env_bool("LOGSTASHUI_DB_CONN_HEALTH_CHECKS", True) - - if engine == "sqlite": - name = _env("LOGSTASHUI_DB_NAME") - db_name = Path(name) if name else Path(data_dir) / "db.sqlite3" - return { - "default": { - "ENGINE": "django.db.backends.sqlite3", - "NAME": db_name, - "CONN_MAX_AGE": conn_max_age, - "CONN_HEALTH_CHECKS": health, - "OPTIONS": { - "init_command": ( - "PRAGMA busy_timeout=20000;" - "PRAGMA journal_mode=WAL;" - ), - "timeout": 20, - }, - } - } - - if engine == "postgresql": - _import_or_raise("psycopg", "postgres") - _require(["LOGSTASHUI_DB_HOST", "LOGSTASHUI_DB_USER"]) - sslmode = _env("LOGSTASHUI_DB_SSLMODE", "prefer") or "prefer" - options: dict = {"sslmode": sslmode} - ca = _env("LOGSTASHUI_DB_SSL_CA") - if ca: - options["sslrootcert"] = ca - return { - "default": { - "ENGINE": "django.db.backends.postgresql", - "NAME": _env("LOGSTASHUI_DB_NAME", "logstashui") or "logstashui", - "USER": _env("LOGSTASHUI_DB_USER"), - "PASSWORD": os.environ.get("LOGSTASHUI_DB_PASSWORD") or "", - "HOST": _env("LOGSTASHUI_DB_HOST"), - "PORT": _env("LOGSTASHUI_DB_PORT", "5432") or "5432", - "CONN_MAX_AGE": conn_max_age, - "CONN_HEALTH_CHECKS": health, - "OPTIONS": options, - } - } - - _import_or_raise("pymysql", "mysql") - import pymysql - - pymysql.install_as_MySQLdb() - _require(["LOGSTASHUI_DB_HOST", "LOGSTASHUI_DB_USER"]) - options = { - "charset": "utf8mb4", - "init_command": ( - "SET sql_mode='STRICT_TRANS_TABLES', " - "NAMES utf8mb4 COLLATE utf8mb4_bin" - ), - } - ca = _env("LOGSTASHUI_DB_SSL_CA") - if ca: - options["ssl"] = {"ca": ca} - return { - "default": { - "ENGINE": "django.db.backends.mysql", - "NAME": _env("LOGSTASHUI_DB_NAME", "logstashui") or "logstashui", - "USER": _env("LOGSTASHUI_DB_USER"), - "PASSWORD": os.environ.get("LOGSTASHUI_DB_PASSWORD") or "", - "HOST": _env("LOGSTASHUI_DB_HOST"), - "PORT": _env("LOGSTASHUI_DB_PORT", "3306") or "3306", - "CONN_MAX_AGE": conn_max_age, - "CONN_HEALTH_CHECKS": health, - "OPTIONS": options, - "TEST": { - "CHARSET": "utf8mb4", - "COLLATION": "utf8mb4_bin", - }, - } - } - - -def check_server_version(connection) -> None: - """Fail-fast if the server is below documented floors. No-op for SQLite.""" - vendor = getattr(connection, "vendor", "") - if vendor == "postgresql": - pg_version = int(getattr(connection, "pg_version", 0) or 0) - if pg_version and pg_version < 140000: - raise RuntimeError( - f"PostgreSQL 14+ is required (server_version_num={pg_version})." - ) - return - if vendor != "mysql": - return - is_mariadb = bool(getattr(connection, "mysql_is_mariadb", False)) - info = (getattr(connection, "mysql_server_info", "") or "").lower() - if "mariadb" in info: - is_mariadb = True - if hasattr(connection, "get_database_version"): - tup = connection.get_database_version() - else: - tup = (0, 0, 0) - major_minor = (int(tup[0]), int(tup[1])) - if is_mariadb and major_minor < (10, 6): - raise RuntimeError( - f"MariaDB 10.6+ is required (server={getattr(connection, 'mysql_server_info', tup)})." - ) - if not is_mariadb and major_minor < (8, 0): - raise RuntimeError( - f"MySQL 8.0+ is required (server={getattr(connection, 'mysql_server_info', tup)})." - ) -``` - -- [ ] **Step 2: Run unit tests** - -```bash -uv run pytest src/logstashui/LogstashUI/tests/test_database.py -v --no-cov -``` - -Expected: all PASS. - -- [ ] **Step 3: Confirm default pytest still uses sqlite** - -```bash -uv run pytest src/logstashui/LogstashUI/tests/test_database.py src/logstashui/LogstashUI/tests/test_paths.py -v --no-cov -``` - -Expected: PASS (paths tests still copy `db.sqlite3`). - -- [ ] **Step 4: Commit** - -```bash -git add src/logstashui/LogstashUI/database.py -git commit -m "feat: wire PostgreSQL and MySQL Django backends from env" -``` - ---- - -### Task 4: SQLite scale warning, gunicorn pidfile, version check on serve - -**Files:** -- Modify: `src/logstashui/LogstashUI/cli.py` -- Modify: `src/logstashui/LogstashUI/tests/test_cli.py` -- Modify: `src/logstashui/LogstashUI/wsgi.py` (comment only) - -- [ ] **Step 1: Add failing CLI tests** - -Append to `src/logstashui/LogstashUI/tests/test_cli.py`: - -```python -def test_parser_migrate_engine_requires_backup_flag(): - parser = build_parser() - ns = parser.parse_args(["migrate-engine", "--to", "postgresql"]) - assert ns.command == "migrate-engine" - assert ns.to == "postgresql" - assert ns.i_have_a_backup is False - - -def test_serve_adds_pidfile_and_warns_sqlite(monkeypatch, tmp_path, capsys): - from LogstashUI import cli - - monkeypatch.setattr(cli, "_manage", lambda argv: None) - monkeypatch.setenv("LOGSTASHUI_TLS", "false") - monkeypatch.setenv("LOGSTASHUI_DATA_DIR", str(tmp_path)) - monkeypatch.delenv("LOGSTASHUI_DB_ENGINE", raising=False) - - captured = {} - - def fake_execvp(file, args): - captured["file"] = file - captured["args"] = list(args) - raise SystemExit(0) - - monkeypatch.setattr(cli.os, "execvp", fake_execvp) - ns = Namespace(skip_migrate=True, no_tls=True, bind="127.0.0.1:8443", workers=2) - try: - cmd_serve(ns) - except SystemExit: - pass - assert "--pid" in captured["args"] - pid_idx = captured["args"].index("--pid") - assert captured["args"][pid_idx + 1].endswith("gunicorn.pid") - err = capsys.readouterr().err - assert "SQLite is the small-install default" in err -``` - -- [ ] **Step 2: Run the new tests — expect FAIL** - -```bash -uv run pytest src/logstashui/LogstashUI/tests/test_cli.py -v --no-cov -``` - -Expected: FAIL on unknown `migrate-engine` subparser and missing `--pid`. - -- [ ] **Step 3: Implement CLI pieces** - -In `build_parser()`, after the `manage` subparser, add: - -```python - migrate = sub.add_parser( - "migrate-engine", - help="BETA: copy SQLite data to PostgreSQL or MySQL (stops gunicorn)", - ) - migrate.add_argument("--to", required=True, choices=("postgresql", "mysql")) - migrate.add_argument( - "--i-have-a-backup", - dest="i_have_a_backup", - action="store_true", - help="Required. Confirms db.sqlite3 was copied aside.", - ) - migrate.add_argument("--pid", type=Path, default=None, help="gunicorn pidfile to signal") - migrate.add_argument( - "--write-env", - type=Path, - default=None, - help="Append LOGSTASHUI_DB_* to this EnvironmentFile", - ) -``` - -In `cmd_serve`, **before** building `gunicorn_cmd`: - -```python - import logging - from .database import canonical_engine, check_server_version - from .paths import resolve_data_dir - - engine = canonical_engine(os.environ.get("LOGSTASHUI_DB_ENGINE")) - if engine == "sqlite" and int(args.workers) > 1: - msg = ( - "SQLite is the small-install default; use PostgreSQL or MySQL/MariaDB " - "for concurrent agents (LOGSTASHUI_WORKERS>1)." - ) - logging.getLogger("LogstashUI").warning(msg) - print(f"WARNING: {msg}", file=sys.stderr) -``` - -After migrate (still inside `if not args.skip_migrate`), after `_manage(["migrate", "--noinput"])`: - -```python - _django_setup() - from django.db import connection - - check_server_version(connection) -``` - -When skip_migrate is true, still run version check only if Django can connect — skip if skip_migrate to keep tests simple. Version check stays tied to migrate path. - -Add pidfile to `gunicorn_cmd` after `--error-logfile`: - -```python - data_dir = resolve_data_dir() - data_dir.mkdir(parents=True, exist_ok=True) - gunicorn_cmd += ["--pid", str(data_dir / "gunicorn.pid")] -``` - -`resolve_data_dir` lives in `LogstashUI.paths` and does not import Django. Import it at the top of `cli.py`: - -```python -from .paths import resolve_data_dir -``` - -`cli.py` currently has no relative imports; it uses `from LogstashUI...` nowhere. Keep consistency with the rest of the file: use - -```python -from LogstashUI.paths import resolve_data_dir -from LogstashUI.database import canonical_engine, check_server_version -``` - -Wire `main()`: - -```python - if command == "migrate-engine": - from LogstashUI.migrate_engine import cmd_migrate_engine - - return cmd_migrate_engine(args) -``` - -For this task, `migrate_engine.py` does not exist yet. **Do not add the main() branch until Task 6.** Only add the argparse subparser so `test_parser_migrate_engine_requires_backup_flag` passes. Leave `main()` unchanged except serve. - -In `wsgi.py`, above `application = get_wsgi_application()`, add this comment (no code): - -```python -# gunicorn --worker-class gevent monkey-patches select before this module loads. -# psycopg 3.1.14+ detects that and waits cooperatively; do not use psycogreen. -``` - -- [ ] **Step 4: Re-run CLI tests** - -```bash -uv run pytest src/logstashui/LogstashUI/tests/test_cli.py -v --no-cov -``` - -Expected: PASS (migrate-engine parser exists; serve pid + warning). `main()` still errors if someone runs migrate-engine — that is Task 6. - -- [ ] **Step 5: Commit** - -```bash -git add src/logstashui/LogstashUI/cli.py src/logstashui/LogstashUI/tests/test_cli.py src/logstashui/LogstashUI/wsgi.py -git commit -m "feat: gunicorn pidfile and SQLite scale warning" -``` - ---- - -### Task 5: Docker Compose DB matrix and `bin/test_databases.sh` - -**Files:** -- Create: `docker/docker-compose.db.yml` -- Create: `bin/test_databases.sh` -- Create: `bin/test_databases.bat` -- Create: `src/logstashui/LogstashUI/tests/test_migrate_live.py` (skip unless env set — empty skip is enough this task) - -- [ ] **Step 1: Write compose file** - -Create `docker/docker-compose.db.yml`: - -```yaml -# Local/CI database matrix for LogstashUI. Not used by smoke compose. -# Host ports avoid clashing with a developer’s own 5432/3306. -name: logstashui-db-test - -services: - postgres: - image: postgres:16 - environment: - POSTGRES_USER: logstashui - POSTGRES_PASSWORD: logstashui - POSTGRES_DB: logstashui - ports: - - "55432:5432" - healthcheck: - test: ["CMD-SHELL", "pg_isready -U logstashui -d logstashui"] - interval: 2s - timeout: 5s - retries: 30 - - mariadb: - image: mariadb:11 - environment: - MARIADB_ROOT_PASSWORD: logstashui - MARIADB_DATABASE: logstashui - MARIADB_USER: logstashui - MARIADB_PASSWORD: logstashui - command: --character-set-server=utf8mb4 --collation-server=utf8mb4_bin - ports: - - "53306:3306" - healthcheck: - test: ["CMD", "healthcheck.sh", "--connect", "--innodb_initialized"] - interval: 3s - timeout: 5s - retries: 40 - - mysql: - image: mysql:8.0 - environment: - MYSQL_ROOT_PASSWORD: logstashui - MYSQL_DATABASE: logstashui - MYSQL_USER: logstashui - MYSQL_PASSWORD: logstashui - command: --character-set-server=utf8mb4 --collation-server=utf8mb4_bin - ports: - - "53307:3306" - healthcheck: - test: ["CMD", "mysqladmin", "ping", "-h", "127.0.0.1", "-ulogstashui", "-plogstashui"] - interval: 3s - timeout: 5s - retries: 40 -``` - -pytest-django must CREATE DATABASE. Postgres `POSTGRES_USER` is superuser. MariaDB/MySQL app users are not. **For the mysql/mariadb pytest runs, use root / logstashui** so Django can create `test_logstashui`. - -- [ ] **Step 2: Write `bin/test_databases.sh`** - -```bash -#!/usr/bin/env bash -# Run SQLite pytest, then the same suite against Postgres, MariaDB, and MySQL, -# then live dump/load tests. Requires Docker. Default `pytest` does not. -set -euo pipefail - -ROOT="$(cd "$(dirname "$0")/.." && pwd)" -cd "$ROOT" -COMPOSE=(docker compose -f docker/docker-compose.db.yml) -KEEP=0 -if [[ "${1:-}" == "--keep" ]]; then - KEEP=1 -fi - -if ! command -v docker >/dev/null; then - echo "ERROR: Docker is required for bin/test_databases.sh" >&2 - echo "Default pytest (SQLite) still works: uv run pytest" >&2 - exit 1 -fi - -uv sync --extra databases --group dev - -echo "==> SQLite pytest (no compose)" -uv run pytest src/logstashui --no-cov - -echo "==> Starting Postgres / MariaDB / MySQL" -"${COMPOSE[@]}" up -d --wait - -run_engine () { - local name="$1" - shift - echo "==> pytest on ${name}" - env "$@" uv run pytest src/logstashui --no-cov -} - -run_engine postgresql \ - LOGSTASHUI_DB_ENGINE=postgresql \ - LOGSTASHUI_DB_HOST=127.0.0.1 \ - LOGSTASHUI_DB_PORT=55432 \ - LOGSTASHUI_DB_NAME=logstashui \ - LOGSTASHUI_DB_USER=logstashui \ - LOGSTASHUI_DB_PASSWORD=logstashui - -run_engine mariadb \ - LOGSTASHUI_DB_ENGINE=mysql \ - LOGSTASHUI_DB_HOST=127.0.0.1 \ - LOGSTASHUI_DB_PORT=53306 \ - LOGSTASHUI_DB_NAME=logstashui \ - LOGSTASHUI_DB_USER=root \ - LOGSTASHUI_DB_PASSWORD=logstashui - -run_engine mysql \ - LOGSTASHUI_DB_ENGINE=mysql \ - LOGSTASHUI_DB_HOST=127.0.0.1 \ - LOGSTASHUI_DB_PORT=53307 \ - LOGSTASHUI_DB_NAME=logstashui \ - LOGSTASHUI_DB_USER=root \ - LOGSTASHUI_DB_PASSWORD=logstashui - -echo "==> Live migrator tests" -env LOGSTASHUI_LIVE_DB=1 \ - LOGSTASHUI_LIVE_PG_PORT=55432 \ - LOGSTASHUI_LIVE_MARIA_PORT=53306 \ - LOGSTASHUI_LIVE_MYSQL_PORT=53307 \ - LOGSTASHUI_LIVE_DB_USER=root \ - LOGSTASHUI_LIVE_DB_PASSWORD=logstashui \ - LOGSTASHUI_LIVE_PG_USER=logstashui \ - uv run pytest src/logstashui/LogstashUI/tests/test_migrate_live.py -v --no-cov - -if [[ "$KEEP" -eq 0 ]]; then - "${COMPOSE[@]}" down -v -fi -``` - -`chmod +x bin/test_databases.sh` - -Until Task 7, `test_migrate_live.py` should skip (no `LOGSTASHUI_LIVE_DB` assertions yet — see Step 3). Running the live file with the env set will collect 0 tests or skip. Create a placeholder that skips: - -```python -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. - -import os - -import pytest - -pytestmark = pytest.mark.skipif( - os.environ.get("LOGSTASHUI_LIVE_DB") != "1", - reason="set LOGSTASHUI_LIVE_DB=1 (bin/test_databases.sh)", -) - - -def test_live_placeholder(): - pytest.skip("migrator live tests land in Task 7") -``` - -- [ ] **Step 3: Write `bin/test_databases.bat`** - -```bat -@echo off -setlocal -cd /d "%~dp0\.." -where docker >nul 2>&1 -if errorlevel 1 ( - echo ERROR: Docker is required for bin\test_databases.bat - exit /b 1 -) -uv sync --extra databases --group dev -uv run pytest src\logstashui --no-cov -if errorlevel 1 exit /b 1 -docker compose -f docker\docker-compose.db.yml up -d --wait -if errorlevel 1 exit /b 1 - -set LOGSTASHUI_DB_ENGINE=postgresql -set LOGSTASHUI_DB_HOST=127.0.0.1 -set LOGSTASHUI_DB_PORT=55432 -set LOGSTASHUI_DB_NAME=logstashui -set LOGSTASHUI_DB_USER=logstashui -set LOGSTASHUI_DB_PASSWORD=logstashui -uv run pytest src\logstashui --no-cov -if errorlevel 1 goto :down - -set LOGSTASHUI_DB_ENGINE=mysql -set LOGSTASHUI_DB_PORT=53306 -set LOGSTASHUI_DB_USER=root -uv run pytest src\logstashui --no-cov -if errorlevel 1 goto :down - -set LOGSTASHUI_DB_PORT=53307 -uv run pytest src\logstashui --no-cov -if errorlevel 1 goto :down - -set LOGSTASHUI_LIVE_DB=1 -set LOGSTASHUI_LIVE_PG_PORT=55432 -set LOGSTASHUI_LIVE_MARIA_PORT=53306 -set LOGSTASHUI_LIVE_MYSQL_PORT=53307 -uv run pytest src\logstashui\LogstashUI\tests\test_migrate_live.py -v --no-cov - -:down -if /I not "%1"=="--keep" docker compose -f docker\docker-compose.db.yml down -v -``` - -- [ ] **Step 4: Smoke the compose file (not the full suite if too long — at least `up --wait`)** - -```bash -docker compose -f docker/docker-compose.db.yml up -d --wait -docker compose -f docker/docker-compose.db.yml ps -docker compose -f docker/docker-compose.db.yml down -v -``` - -Expected: three services healthy, then removed. - -Then run **one** engine pytest if time allows: - -```bash -uv sync --extra databases --group dev -LOGSTASHUI_DB_ENGINE=postgresql LOGSTASHUI_DB_HOST=127.0.0.1 LOGSTASHUI_DB_PORT=55432 \ - LOGSTASHUI_DB_NAME=logstashui LOGSTASHUI_DB_USER=logstashui LOGSTASHUI_DB_PASSWORD=logstashui \ - uv run pytest src/logstashui/LogstashUI/tests/test_database.py src/logstashui/Site/tests/test_views.py::test_health_check_returns_200 --no-cov -``` - -(Bring compose back up first.) Expected: PASS. If `test_health_check` fails on migrate, fix charset/user and re-run. Known CRUD failures on sqlite may also appear on postgres — do not xfail them in this task. - -- [ ] **Step 5: Commit** - -```bash -git add docker/docker-compose.db.yml bin/test_databases.sh bin/test_databases.bat \ - src/logstashui/LogstashUI/tests/test_migrate_live.py -git commit -m "test: Docker Postgres/MariaDB/MySQL matrix script" -``` - ---- - -### Task 6: BETA `migrate-engine` (unit-tested, no live servers) - -**Files:** -- Create: `src/logstashui/LogstashUI/migrate_engine.py` -- Create: `src/logstashui/LogstashUI/tests/test_migrate_engine.py` -- Modify: `src/logstashui/LogstashUI/cli.py` (`main()` branch) - -- [ ] **Step 1: Write failing unit tests** - -Create `src/logstashui/LogstashUI/tests/test_migrate_engine.py`: - -```python -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. - -from argparse import Namespace -from pathlib import Path - -import pytest - -from LogstashUI import migrate_engine as me - - -def test_refuses_without_backup_flag(capsys): - ns = Namespace(to="postgresql", i_have_a_backup=False, pid=None, write_env=None) - with pytest.raises(SystemExit) as exc: - me.cmd_migrate_engine(ns) - assert exc.value.code == 2 - assert "back up" in capsys.readouterr().err.lower() - - -def test_refuses_sqlite_target(tmp_path, monkeypatch, capsys): - monkeypatch.setenv("LOGSTASHUI_DATA_DIR", str(tmp_path)) - ns = Namespace(to="sqlite", i_have_a_backup=True, pid=None, write_env=None) - with pytest.raises(SystemExit): - me.cmd_migrate_engine(ns) - - -def test_refuses_missing_sqlite_file(tmp_path, monkeypatch, capsys): - monkeypatch.setenv("LOGSTASHUI_DATA_DIR", str(tmp_path)) - ns = Namespace(to="postgresql", i_have_a_backup=True, pid=None, write_env=None) - with pytest.raises(SystemExit) as exc: - me.cmd_migrate_engine(ns) - assert exc.value.code == 1 - assert "db.sqlite3" in capsys.readouterr().err - - -def test_stop_pid_sends_sigterm(tmp_path, monkeypatch): - pidfile = tmp_path / "gunicorn.pid" - pidfile.write_text("4242\n") - sent = {} - calls = {"n": 0} - - def kill_then_gone(pid, sig): - calls["n"] += 1 - if calls["n"] == 1: - sent["pid"] = pid - sent["sig"] = sig - return - raise ProcessLookupError() - - monkeypatch.setattr(me.os, "kill", kill_then_gone) - monkeypatch.setattr(me.time, "sleep", lambda s: None) - me.stop_gunicorn(pidfile) - assert sent["pid"] == 4242 - assert sent["sig"] == me.signal.SIGTERM - - -def test_write_env_appends(tmp_path, monkeypatch): - envf = tmp_path / "logstashui.default" - envf.write_text("LOGSTASHUI_DATA_DIR=/var/lib/logstashui\n") - monkeypatch.setenv("LOGSTASHUI_DB_HOST", "db.example") - monkeypatch.setenv("LOGSTASHUI_DB_USER", "lsui") - monkeypatch.setenv("LOGSTASHUI_DB_NAME", "logstashui") - me.write_env_file(envf, "postgresql") - text = envf.read_text() - assert "LOGSTASHUI_DB_ENGINE=postgresql" in text - assert "LOGSTASHUI_DB_HOST=db.example" in text - assert "PASSWORD" not in text -``` - -- [ ] **Step 2: Run — expect FAIL (module missing)** - -```bash -uv run pytest src/logstashui/LogstashUI/tests/test_migrate_engine.py -v --no-cov -``` - -- [ ] **Step 3: Implement `migrate_engine.py`** - -```python -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. - -"""BETA sqlite → PostgreSQL/MySQL copy via dumpdata/loaddata in child processes.""" - -from __future__ import annotations - -import os -import signal -import sqlite3 -import subprocess -import sys -import time -from pathlib import Path - -from LogstashUI.database import canonical_engine -from LogstashUI.paths import resolve_data_dir - -_DUMP_EXCLUDE = ["contenttypes", "auth.permission", "sessions"] -_SEQUENCE_APPS = [ - "admin", - "auth", - "PipelineManager", - "Management", - "SNMP", - "AI", - "Monitoring", - "Site", -] - - -def cmd_migrate_engine(args) -> int: - if not getattr(args, "i_have_a_backup", False): - print( - "BETA migrate-engine: copy DATA_DIR/db.sqlite3 to a backup first, " - "then re-run with --i-have-a-backup.", - file=sys.stderr, - ) - raise SystemExit(2) - - target = canonical_engine(getattr(args, "to", "")) - if target not in ("postgresql", "mysql"): - print(" --to must be postgresql or mysql", file=sys.stderr) - raise SystemExit(2) - - data_dir = resolve_data_dir(migrate_legacy=False) - sqlite_path = data_dir / "db.sqlite3" - if not sqlite_path.is_file(): - print(f"No SQLite database at {sqlite_path}", file=sys.stderr) - raise SystemExit(1) - - print( - "BETA: migrate-engine stops gunicorn (UI port down), dumps SQLite, " - "loads the target, and does not restart serve.", - file=sys.stderr, - ) - - pidfile = args.pid or (data_dir / "gunicorn.pid") - if Path(pidfile).is_file(): - stop_gunicorn(Path(pidfile)) - - wal_checkpoint(sqlite_path) - dump_path = data_dir / "migrate-engine-dump.json" - run_manage( - [ - "dumpdata", - "--natural-foreign", - "--natural-primary", - "--output", - str(dump_path), - *[f"-e={item}" for item in _DUMP_EXCLUDE], - ], - extra_env={"LOGSTASHUI_DB_ENGINE": "sqlite"}, - ) - run_manage(["migrate", "--noinput"], extra_env={"LOGSTASHUI_DB_ENGINE": target}) - run_manage(["loaddata", str(dump_path)], extra_env={"LOGSTASHUI_DB_ENGINE": target}) - if target == "postgresql": - reset_postgres_sequences(extra_env={"LOGSTASHUI_DB_ENGINE": target}) - - if args.write_env: - write_env_file(Path(args.write_env), target) - - print( - "Done. Keep LOGSTASHUI_DB_ENGINE=" - f"{target} and start LogstashUI (systemctl start logstashui). " - "Do not auto-restart: systemd Restart= would race." - ) - return 0 - - -def wal_checkpoint(sqlite_path: Path) -> None: - conn = sqlite3.connect(str(sqlite_path)) - try: - conn.execute("PRAGMA wal_checkpoint(TRUNCATE)") - finally: - conn.close() - - -def stop_gunicorn(pidfile: Path) -> None: - raw = pidfile.read_text(encoding="utf-8").strip() - if not raw.isdigit(): - print(f"Invalid pidfile {pidfile}", file=sys.stderr) - raise SystemExit(1) - pid = int(raw) - try: - os.kill(pid, signal.SIGTERM) - except ProcessLookupError: - pidfile.unlink(missing_ok=True) - return - deadline = time.time() + 30 - while time.time() < deadline: - try: - os.kill(pid, 0) - except ProcessLookupError: - pidfile.unlink(missing_ok=True) - return - time.sleep(0.2) - print(f"gunicorn pid {pid} did not exit after SIGTERM", file=sys.stderr) - raise SystemExit(1) - - -def run_manage(argv: list[str], extra_env: dict[str, str]) -> None: - env = os.environ.copy() - env.update(extra_env) - env.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") - code = ( - "import sys; from django.core.management import execute_from_command_line; " - "execute_from_command_line(['logstashui'] + sys.argv[1:])" - ) - cmd = [sys.executable, "-c", code, *argv] - proc = subprocess.run(cmd, env=env, check=False) - if proc.returncode != 0: - raise SystemExit(proc.returncode) - - -def reset_postgres_sequences(extra_env: dict[str, str]) -> None: - env = os.environ.copy() - env.update(extra_env) - env.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") - code = ( - "import sys; from django.core.management import execute_from_command_line; " - "execute_from_command_line(['logstashui', 'sqlsequencereset'] + sys.argv[1:])" - ) - sql = subprocess.run( - [sys.executable, "-c", code, *_SEQUENCE_APPS], - env=env, - check=False, - capture_output=True, - text=True, - ) - if sql.returncode != 0: - print(sql.stderr, file=sys.stderr) - raise SystemExit(sql.returncode) - if not sql.stdout.strip(): - return - dbshell = subprocess.run( - [ - sys.executable, - "-c", - "import sys; from django.core.management import execute_from_command_line; " - "execute_from_command_line(['logstashui', 'dbshell'])", - ], - env=env, - input=sql.stdout, - text=True, - check=False, - ) - if dbshell.returncode != 0: - raise SystemExit(dbshell.returncode) - - -def write_env_file(path: Path, engine: str) -> None: - lines = [ - f"LOGSTASHUI_DB_ENGINE={engine}", - f"LOGSTASHUI_DB_NAME={os.environ.get('LOGSTASHUI_DB_NAME', 'logstashui')}", - f"LOGSTASHUI_DB_HOST={os.environ.get('LOGSTASHUI_DB_HOST', '')}", - f"LOGSTASHUI_DB_PORT={os.environ.get('LOGSTASHUI_DB_PORT', '')}", - f"LOGSTASHUI_DB_USER={os.environ.get('LOGSTASHUI_DB_USER', '')}", - ] - existing = path.read_text(encoding="utf-8") if path.is_file() else "" - path.write_text(existing.rstrip() + "\n\n# migrate-engine\n" + "\n".join(lines) + "\n") -``` - -`test_refuses_sqlite_target`: `canonical_engine("sqlite")` returns sqlite, then `target not in (...)` exits 2. argparse `choices` already blocks sqlite in CLI; the function still guards. - -`test_stop_pid_sends_sigterm` uses `ProcessLookupError` on the wait loop — `os.kill(pid, 0)` raises, success. - -Fix the test’s double-setattr: keep only `kill_then_gone`. - -- [ ] **Step 4: Wire `main()` in `cli.py`** - -```python - if command == "migrate-engine": - from LogstashUI.migrate_engine import cmd_migrate_engine - - return cmd_migrate_engine(args) -``` - -- [ ] **Step 5: Run unit tests** - -```bash -uv run pytest src/logstashui/LogstashUI/tests/test_migrate_engine.py src/logstashui/LogstashUI/tests/test_cli.py -v --no-cov -``` - -Expected: PASS. If `test_refuses_sqlite_target` never hits `cmd_migrate_engine` because argparse isn’t used (Namespace to=sqlite), the guard in `cmd_migrate_engine` handles it. - -- [ ] **Step 6: Commit** - -```bash -git add src/logstashui/LogstashUI/migrate_engine.py \ - src/logstashui/LogstashUI/tests/test_migrate_engine.py \ - src/logstashui/LogstashUI/cli.py -git commit -m "feat: BETA migrate-engine sqlite to postgres/mysql" -``` - ---- - -### Task 7: Live migration tests - -**Files:** -- Modify: `src/logstashui/LogstashUI/tests/test_migrate_live.py` - -- [ ] **Step 1: Replace the placeholder with subprocess live tests** - -Live tests must not use `@pytest.mark.django_db` and then switch `LOGSTASHUI_DB_ENGINE` in-process (Django settings are frozen). Use isolated `DATA_DIR` + child processes only. - -Overwrite `src/logstashui/LogstashUI/tests/test_migrate_live.py` with: - -```python -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. - -from __future__ import annotations - -import os -import subprocess -import sys -from argparse import Namespace -from pathlib import Path - -import pytest - -from LogstashUI.migrate_engine import cmd_migrate_engine, run_manage - -pytestmark = pytest.mark.skipif( - os.environ.get("LOGSTASHUI_LIVE_DB") != "1", - reason="set LOGSTASHUI_LIVE_DB=1 (bin/test_databases.sh)", -) - - -def _python_manage(args: list[str], env: dict[str, str]) -> None: - run_manage(args, extra_env=env) - - -def _seed(data_dir: Path) -> None: - env = { - "LOGSTASHUI_DATA_DIR": str(data_dir), - "LOGSTASHUI_DB_ENGINE": "sqlite", - "DJANGO_SETTINGS_MODULE": "LogstashUI.settings", - } - _python_manage(["migrate", "--noinput"], env) - code = ( - "import os, django; os.environ.setdefault('DJANGO_SETTINGS_MODULE','LogstashUI.settings'); " - "django.setup(); " - "from django.contrib.auth.models import User; " - "from PipelineManager.models import Policy; " - "User.objects.create_user('migrate-user', password='x'); " - "Policy.objects.create(name='Migrate Policy', logstash_yml='node.name: t', " - "jvm_options='#', log4j2_properties='#')" - ) - env_full = os.environ.copy() - env_full.update(env) - subprocess.run([sys.executable, "-c", code], env=env_full, check=True) - - -def _count(env: dict[str, str]) -> tuple[int, int]: - env_full = os.environ.copy() - env_full.update(env) - env_full.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") - code = ( - "import os, django; os.environ.setdefault('DJANGO_SETTINGS_MODULE','LogstashUI.settings'); " - "django.setup(); " - "from django.contrib.auth.models import User; " - "from PipelineManager.models import Policy; " - "print(User.objects.filter(username='migrate-user').count()); " - "print(Policy.objects.filter(name='Migrate Policy').count())" - ) - out = subprocess.run( - [sys.executable, "-c", code], env=env_full, check=True, capture_output=True, text=True - ) - lines = [ln for ln in out.stdout.splitlines() if ln.strip().isdigit()] - return int(lines[-2]), int(lines[-1]) - - -def _run_to(tmp_path: Path, engine: str, port: str, user: str) -> None: - data_dir = tmp_path / "data" - data_dir.mkdir() - os.environ["LOGSTASHUI_DATA_DIR"] = str(data_dir) - _seed(data_dir) - target = { - "LOGSTASHUI_DB_ENGINE": engine, - "LOGSTASHUI_DB_HOST": "127.0.0.1", - "LOGSTASHUI_DB_PORT": port, - "LOGSTASHUI_DB_NAME": "logstashui_migrate", - "LOGSTASHUI_DB_USER": user, - "LOGSTASHUI_DB_PASSWORD": os.environ.get("LOGSTASHUI_LIVE_DB_PASSWORD", "logstashui"), - } - for k, v in target.items(): - os.environ[k] = v - ns = Namespace(to="postgresql" if engine == "postgresql" else "mysql", i_have_a_backup=True, pid=None, write_env=None) - cmd_migrate_engine(ns) - users, policies = _count(target | {"LOGSTASHUI_DATA_DIR": str(data_dir)}) - assert users == 1 - assert policies == 1 - - -def test_live_postgres(tmp_path, monkeypatch): - monkeypatch.setenv("LOGSTASHUI_DATA_DIR", str(tmp_path / "data")) - _run_to( - tmp_path, - "postgresql", - os.environ.get("LOGSTASHUI_LIVE_PG_PORT", "55432"), - os.environ.get("LOGSTASHUI_LIVE_PG_USER", "logstashui"), - ) - - -def test_live_mariadb(tmp_path, monkeypatch): - _run_to( - tmp_path, - "mysql", - os.environ.get("LOGSTASHUI_LIVE_MARIA_PORT", "53306"), - os.environ.get("LOGSTASHUI_LIVE_DB_USER", "root"), - ) - - -def test_live_mysql(tmp_path, monkeypatch): - _run_to( - tmp_path, - "mysql", - os.environ.get("LOGSTASHUI_LIVE_MYSQL_PORT", "53307"), - os.environ.get("LOGSTASHUI_LIVE_DB_USER", "root"), - ) -``` - -Create the target database `logstashui_migrate` in each engine before load. Add a helper at the top of `_run_to` that uses `run_manage(["migrate", "--noinput"], target)` which `cmd_migrate_engine` already does. Postgres cannot connect if the DB name does not exist. - -**Create `logstashui_migrate` in compose** by adding a second database via init SQL. - -Add `docker/db-init/postgres-extra.sql`: - -```sql -CREATE DATABASE logstashui_migrate OWNER logstashui; -``` - -Mount it in compose under postgres: - -```yaml - volumes: - - ./db-init/postgres-extra.sql:/docker-entrypoint-initdb.d/02-migrate.sql:ro -``` - -For MariaDB/MySQL, `docker/db-init/mysql-extra.sql`: - -```sql -CREATE DATABASE IF NOT EXISTS logstashui_migrate CHARACTER SET utf8mb4 COLLATE utf8mb4_bin; -GRANT ALL ON logstashui_migrate.* TO 'logstashui'@'%'; -GRANT ALL ON logstashui_migrate.* TO 'root'@'%'; -``` - -Mount on **both** mariadb and mysql services as `/docker-entrypoint-initdb.d/02-migrate.sql`. - -If compose was already created without the volume, `down -v` so init runs again. - -- [ ] **Step 2: Run live tests** - -```bash -./bin/test_databases.sh -``` - -Expected: SQLite pytest completes; three engine pytest runs; three live migrator tests PASS. Pre-existing CRUD failures may still fail the **full** suite — if they fail on sqlite they fail on all engines. Do not change those tests. If the script must be CI-green despite them, do **not** hide them with xfail; report in the PR. Optionally restrict engine runs to `LogstashUI/tests` + `Site/tests/test_views.py::test_health_check_returns_200` **only if** the full suite is blocked by those known failures **and** they fail the same way on sqlite. Prefer full suite. - -- [ ] **Step 3: Commit** - -```bash -git add src/logstashui/LogstashUI/tests/test_migrate_live.py docker/docker-compose.db.yml docker/db-init -git commit -m "test: live sqlite dump/load onto Postgres MariaDB MySQL" -``` - ---- - -### Task 8: systemd prompts, sample env, operator docs, CHANGELOG, Docker image, CI - -**Files:** -- Modify: `src/logstashui/LogstashUI/packaging/logstashui.default` -- Modify: `src/logstashui/LogstashUI/cli.py` (`install_systemd`, `render_default_env`) -- Modify: `src/logstashui/LogstashUI/tests/test_cli.py` (`test_systemd_dry_run_writes_unit_and_default`) -- Modify: `docs/docs/logstashui/configuration/environment.md` -- Modify: `docs/docs/logstashui/general/deploy.md` -- Modify: `CHANGELOG.md` -- Modify: `docker/Dockerfile` -- Create: `.github/workflows/test-databases.yml` - -- [ ] **Step 1: Sample env** - -Replace the “Future database backends” block in `logstashui.default` with: - -``` -# Database. Unset engine = sqlite at $LOGSTASHUI_DATA_DIR/db.sqlite3. -# DATA_DIR is still required for TLS, secrets, logs, staticfiles. -# LOGSTASHUI_DB_ENGINE=sqlite -# LOGSTASHUI_DB_NAME=logstashui -# LOGSTASHUI_DB_HOST= -# LOGSTASHUI_DB_PORT= -# LOGSTASHUI_DB_USER= -# LOGSTASHUI_DB_PASSWORD= -# LOGSTASHUI_DB_SSLMODE=prefer -# LOGSTASHUI_DB_SSL_CA= -# LOGSTASHUI_DB_CONN_MAX_AGE=60 -# LOGSTASHUI_DB_CONN_HEALTH_CHECKS=true -# -# Native extras: uv pip install 'LogstashUI[postgres]' or 'LogstashUI[mysql]' -# or 'LogstashUI[databases]'. The container image already includes both drivers. -# -# Create MySQL/MariaDB with utf8mb4_bin: -# CREATE DATABASE logstashui CHARACTER SET utf8mb4 COLLATE utf8mb4_bin; -``` - -- [ ] **Step 2: systemd generator** - -Add optional kwargs to `install_systemd` and `render_default_env`: `db_engine=""`, `db_host=""`, `db_name=""`, `db_user=""`, `db_port=""`. - -In the interactive block, after `no_auth` prompt: - -```python - db_engine = _prompt("LOGSTASHUI_DB_ENGINE (sqlite/postgresql/mysql)", db_engine or "sqlite") - if canonical_engine(db_engine) != "sqlite": - db_host = _prompt("LOGSTASHUI_DB_HOST", db_host) - db_port = _prompt("LOGSTASHUI_DB_PORT", db_port) - db_name = _prompt("LOGSTASHUI_DB_NAME", db_name or "logstashui") - db_user = _prompt("LOGSTASHUI_DB_USER", db_user) -``` - -Do not prompt for password (operator edits the EnvironmentFile / Secret). - -In `render_default_env`, if `canonical_engine(db_engine) != "sqlite"` append extras like the other optional keys: - -```python - if db_engine and canonical_engine(db_engine) != "sqlite": - extras.append(f"LOGSTASHUI_DB_ENGINE={canonical_engine(db_engine)}") - if db_host: - extras.append(f"LOGSTASHUI_DB_HOST={db_host}") - if db_port: - extras.append(f"LOGSTASHUI_DB_PORT={db_port}") - if db_name: - extras.append(f"LOGSTASHUI_DB_NAME={db_name}") - if db_user: - extras.append(f"LOGSTASHUI_DB_USER={db_user}") -``` - -Pass the new kwargs from `cmd_systemd` / `install_systemd` into `render_default_env`. Keep `test_systemd_dry_run_writes_unit_and_default` working (defaults empty → sqlite comments only). - -- [ ] **Step 3: Docs** - -In `docs/docs/logstashui/configuration/environment.md`, replace the “Database (sqlite only)” section with: - -```markdown -## Database - -| Variable | Default | Purpose | -|---|---|---| -| `LOGSTASHUI_DB_ENGINE` | `sqlite` | `sqlite`, `postgresql`, or `mysql` (MariaDB uses `mysql`). Aliases: `sqlite3`, `postgres`, `mariadb`, `my` | -| `LOGSTASHUI_DB_NAME` | sqlite: `$LOGSTASHUI_DATA_DIR/db.sqlite3`; else `logstashui` | Database name / sqlite path | -| `LOGSTASHUI_DB_HOST` | empty | **Required** for postgresql/mysql | -| `LOGSTASHUI_DB_PORT` | `5432` / `3306` | | -| `LOGSTASHUI_DB_USER` | empty | **Required** for postgresql/mysql | -| `LOGSTASHUI_DB_PASSWORD` | empty | Put in a Secret / `chmod 640` EnvironmentFile | -| `LOGSTASHUI_DB_SSLMODE` | postgres: `prefer` | `disable` `allow` `prefer` `require` `verify-ca` `verify-full` | -| `LOGSTASHUI_DB_SSL_CA` | empty | CA file for mysql TLS and postgres `verify-*` | -| `LOGSTASHUI_DB_CONN_MAX_AGE` | `60` | Persistent connections (seconds); `0` closes per request | -| `LOGSTASHUI_DB_CONN_HEALTH_CHECKS` | `true` | Django `CONN_HEALTH_CHECKS` | - -Floors: PostgreSQL 14+, MariaDB 10.6+, MySQL 8.0+. Create MySQL/MariaDB as `utf8mb4` / `utf8mb4_bin` so unique names match SQLite/Postgres case-sensitivity. - -**Install extras (native pip/uv):** `uv pip install 'LogstashUI[postgres]'`, `'LogstashUI[mysql]'`, or `'LogstashUI[databases]'`. The Docker/K8s image already installs `[databases]`. Missing driver fails at startup with that extra name. - -`LOGSTASHUI_DATA_DIR` is still required when the database is remote (TLS, `.django_secret_key`, logs, staticfiles). - -**SQLite scale:** `logstashui serve` logs a warning when engine is sqlite and `LOGSTASHUI_WORKERS` > 1. Use PostgreSQL or MySQL/MariaDB for concurrent agents. Startup still succeeds. - -**Connections:** gunicorn remains gevent (`--worker-connections 1000`). Keep `LOGSTASHUI_WORKERS` × in-flight requests under the server `max_connections`. PgBouncer (or equivalent) is optional, not required. - -No `DATABASE_URL`. No YAML. - -### Offline migration (supported) - -1. `systemctl stop logstashui` (or stop the container). -2. Copy `$LOGSTASHUI_DATA_DIR/db.sqlite3` somewhere safe. Keep the rest of `DATA_DIR` (same Django secret key). -3. Create the server database (`utf8mb4_bin` on MySQL/MariaDB). -4. Set `LOGSTASHUI_DB_*` for the target. Native installs need the matching extra. -5. `logstashui manage dumpdata --natural-foreign --natural-primary -e contenttypes -e auth.permission -e sessions -o dump.json` while still on sqlite, **or** use the BETA CLI below. -6. `logstashui manage migrate --noinput && logstashui manage loaddata dump.json` -7. Postgres: `logstashui manage sqlsequencereset PipelineManager Management SNMP AI auth admin | logstashui manage dbshell` -8. Start LogstashUI. Log in again (sessions were not copied). - -### BETA CLI - -```bash -# env already points at the empty target server; sqlite file still in DATA_DIR -sudo systemctl stop logstashui # avoid Restart= racing SIGTERM -logstashui migrate-engine --to postgresql --i-have-a-backup -# optional: --write-env /etc/default/logstashui -sudo systemctl start logstashui -``` - -`--to mysql` covers MariaDB and MySQL. The command SIGTERMs gunicorn if `$LOGSTASHUI_DATA_DIR/gunicorn.pid` is live, checkpoints WAL, dump/load, and **does not** restart serve. -``` - -In `deploy.md` Data directory paragraph, after the sqlite sentence, add: the database may be external Postgres/MySQL; the PVC/bind-mount is still required for TLS and secrets. In Kubernetes minimum list, add ConfigMap `LOGSTASHUI_DB_ENGINE/HOST/NAME/USER` + Secret `LOGSTASHUI_DB_PASSWORD`. - -- [ ] **Step 4: Dockerfile** - -Change: - -``` -RUN uv pip install --system --no-cache /app -``` - -to: - -``` -RUN uv pip install --system --no-cache "/app[databases]" -``` - -Leave `sqlite3` apt package (debug/sqlite CLI in the image is fine). - -- [ ] **Step 5: CHANGELOG** - -Insert at the top of `CHANGELOG.md`: - -```markdown -## [0.5.2] - Multi-database - -Package version remains **0.5.1** until release tagging; this documents the 0.5.2 database work. - -- `LOGSTASHUI_DB_ENGINE=sqlite|postgresql|mysql` (MariaDB uses `mysql`). Discrete `LOGSTASHUI_DB_HOST/PORT/NAME/USER/PASSWORD` plus SSL and `CONN_MAX_AGE`. No YAML, no `DATABASE_URL`. -- Default is still SQLite. `logstashui serve` warns when `LOGSTASHUI_WORKERS>1` on SQLite. -- Native extras: `LogstashUI[postgres]`, `[mysql]`, `[databases]`. Container image installs `[databases]`. -- BETA `logstashui migrate-engine --to postgresql|mysql --i-have-a-backup` copies SQLite → server (stops gunicorn; does not restart). -- `bin/test_databases.sh` runs the suite and dump/load against local Docker Postgres, MariaDB, and MySQL. -``` - -Do not bump `pyproject.toml` version in this plan unless the operator asks; the spec is “0.5.2 work”. - -- [ ] **Step 6: CI workflow** - -Create `.github/workflows/test-databases.yml`: - -```yaml -name: Database matrix - -on: - pull_request: - push: - branches: [main, master] - -jobs: - db: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - - uses: astral-sh/setup-uv@v4 - with: - python-version: "3.12" - - uses: actions/setup-python@v5 - with: - python-version: "3.12" - - name: Run database matrix - run: bash bin/test_databases.sh -``` - -- [ ] **Step 7: Tests for systemd extras** - -Extend `test_systemd_dry_run_writes_unit_and_default` **or** add: - -```python -def test_systemd_env_includes_postgres_when_passed(tmp_path): - result = install_systemd( - output_dir=tmp_path, - exec_start="/usr/bin/logstashui serve", - user="logstashui", - group="logstashui", - data_dir="/var/lib/logstashui", - bind="0.0.0.0:8443", - workers=2, - allowed_hosts="*", - csrf_trusted_origins="", - tls="true", - host_hostname="", - host_ips="", - tls_sans="", - agent_ui_url="", - no_auth="false", - dry_run=True, - db_engine="postgresql", - db_host="db.example", - db_port="5432", - db_name="logstashui", - db_user="lsui", - ) - text = (tmp_path / "logstashui.default").read_text() - assert "LOGSTASHUI_DB_ENGINE=postgresql" in text - assert "LOGSTASHUI_DB_HOST=db.example" in text -``` - -Update `install_systemd` signature with those kwargs defaulting to `""`. - -- [ ] **Step 8: Run focused tests + default sqlite pytest slice** - -```bash -uv run pytest src/logstashui/LogstashUI/tests/test_cli.py src/logstashui/LogstashUI/tests/test_database.py src/logstashui/LogstashUI/tests/test_migrate_engine.py -v --no-cov -``` - -Expected: PASS. - -- [ ] **Step 9: Commit** - -```bash -git add src/logstashui/LogstashUI/packaging/logstashui.default \ - src/logstashui/LogstashUI/cli.py \ - src/logstashui/LogstashUI/tests/test_cli.py \ - docs/docs/logstashui/configuration/environment.md \ - docs/docs/logstashui/general/deploy.md \ - CHANGELOG.md docker/Dockerfile .github/workflows/test-databases.yml -git commit -m "docs: multi-engine env, systemd, Docker extras, CI matrix" -``` - ---- - -### Task 9: Verification gate - -- [ ] **Step 1: Default inner loop (no Docker extras required)** - -```bash -uv run pytest src/logstashui --no-cov -``` - -Expected: no **new** failures vs sqlite baseline. Pre-existing three CRUD tests may still fail. - -- [ ] **Step 2: Full matrix** - -```bash -./bin/test_databases.sh -``` - -Expected: sqlite + postgresql + mariadb + mysql pytest runs; live migrator PASS. - -- [ ] **Step 3: Image install extras** - -```bash -grep -n '\[databases\]' docker/Dockerfile -``` - -Expected: `uv pip install --system --no-cache "/app[databases]"`. - -- [ ] **Step 4: Do not start smoke compose unless the operator asks.** Product CA must remain untouched. - ---- - -## Spec coverage (self-review) - -| Spec item | Task | -|---|---| -| ORM only / `build_databases` | 2–3 | -| Discrete env, aliases, SSL, CONN_* | 2–3, 8 | -| Extras + Docker `[databases]` | 1, 8 | -| psycopg gevent (version pin, no psycogreen) | 1, 4 (wsgi comment) | -| PyMySQL `install_as_MySQLdb` | 3 | -| Fail-fast unknown/missing driver/host | 2–3 | -| Server version floors | 2–3, 4 (serve after migrate) | -| SQLite default + scale warning | 3–4 | -| gunicorn pidfile + SIGTERM | 4, 6 | -| Offline dump/load docs | 8 | -| BETA CLI `--i-have-a-backup` | 6–7 | -| `--write-env`, no auto-restart | 6, 8 | -| Local Docker three engines + migrator | 5, 7, 9 | -| CI uses the same script | 8 | -| systemd prompts / sample env | 8 | -| MySQL `utf8mb4_bin` | 3, 5 (compose command + CREATE DATABASE) | -| DATA_DIR still required | 8 docs | -| No YAML / DATABASE_URL / DAO / smoke-stack DB | throughout | -| Pre-existing pytest failures left alone | 5, 9 | - -**Placeholders:** none remaining. Live tests use subprocesses so Django settings never switch in-process (matches architecture). - -**Type names:** `canonical_engine`, `_import_or_raise`, `check_server_version`, `cmd_migrate_engine`, `stop_gunicorn`, `write_env_file`, `run_manage` — used consistently across tasks. diff --git a/docs/superpowers/plans/2026-09-02-airgap-freeze.md b/docs/superpowers/plans/2026-09-02-airgap-freeze.md deleted file mode 100644 index 247b778..0000000 --- a/docs/superpowers/plans/2026-09-02-airgap-freeze.md +++ /dev/null @@ -1,34 +0,0 @@ -# Air-gapped freeze Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Optional `bin/freeze_logstashui.sh` emits Linux x86_64 wheelhouse, Docker, and experimental PyInstaller zips so an air-gapped host can run LogstashUI with no PyPI or registry. - -**Architecture:** Connected builder runs a bash script (not a CLI subcommand). Default `uv build` is unchanged. Isolated helpers (`install.sh` / `load.sh` / `run.sh`) live in each zip. Templates and the PyInstaller spec live in `packaging/offline/`. `os.execvp("gunicorn")` cannot work inside PyInstaller; `cli.py` gets a frozen in-process gunicorn path without changing `--worker-class gevent`. - -**Tech Stack:** bash, uv, pip download (manylinux2014_x86_64 / cp312), Docker save, PyInstaller onedir, MkDocs-style docs. - -**Spec:** `docs/superpowers/specs/2026-09-02-airgap-freeze-design.md` - ---- - -## File map - -| Path | Role | -|---|---| -| `bin/freeze_logstashui.sh` | Builder | -| `bin/test_freeze_wheels.sh` | Unzip + `docker run --platform linux/amd64 --network=none` install smoke | -| `packaging/offline/*` | Templates + PyInstaller spec/entry | -| `src/logstashui/LogstashUI/cli.py` | Frozen gunicorn exec | -| `docs/docs/logstashui/general/offline.md` | Operator + maintainer docs | -| `docs/docs/logstashui/general/deploy.md` | Option 6 | -| `CHANGELOG.md` | 0.5.2 packaging note | -| `.github/workflows/offline-freeze.yml` | `workflow_dispatch` `--wheels` only | - -### Task 1: Templates + freeze script + docs + frozen hook - -Implement per spec. `--all` on non-Linux-x86_64 skips standalone with a warning; explicit `--standalone` fails. Never `docker pull`. `pip download` without `--abi` (so `py3-none-any` wheels are kept). Builder requires `uv python find 3.12`. - -### Task 2: Wheels smoke - -`bin/freeze_logstashui.sh --wheels` then `bin/test_freeze_wheels.sh`. diff --git a/docs/superpowers/specs/2026-08-21-multi-database-design.md b/docs/superpowers/specs/2026-08-21-multi-database-design.md deleted file mode 100644 index ce75e33..0000000 --- a/docs/superpowers/specs/2026-08-21-multi-database-design.md +++ /dev/null @@ -1,266 +0,0 @@ -# 2026-08-21 — Multi-database design (SQLite | PostgreSQL | MariaDB/MySQL) - -**LogstashUI version:** 0.5.2 -**Python:** 3.12–3.14 -**Django:** 6.x (existing) -**Chosen approach:** Django backends + keep gevent. No new CRUD/repository layer. - -## Goal - -Operators can run LogstashUI on **SQLite** (default), **PostgreSQL**, or **MariaDB/MySQL** using discrete environment variables. Default `pytest` on SQLite stays green. Local Docker can exercise each server engine for CRUD **and** SQLite→server migration. - -SQLite does not scale under gunicorn/gevent (2 workers × 1000 greenlets). Server engines address that without changing the product data model. - -## Non-goals (0.5.2) - -- A new DAO/repository over `Model.objects` (Django ORM already abstracts CRUD). -- YAML, `logstashui.yml`, or `DATABASE_URL`. -- Oracle, SQL Server, or other engines. -- Requiring PgBouncer/ProxySQL, or changing gunicorn off gevent. -- Live dual-write, reverse migration (server→SQLite), or pgloader as the supported path. -- An in-app migration wizard (stopping :8443 removes the UI). -- A 503 maintenance page on :8443 during copy. -- Helm charts; shipping Postgres/MariaDB/MySQL in the default smoke compose. -- Product CA rotation; relocating `DATA_DIR` (TLS, secrets, logs, staticfiles stay on disk). - -## Decisions (locked) - -| Topic | Decision | -|---|---| -| Abstraction | Django ORM only; the switch is `build_databases()` | -| Default engine | SQLite; warn at scale; do not refuse to start | -| Config | Env vars only, discrete `LOGSTASHUI_DB_*` | -| Engine names | Canonical: `sqlite`, `postgresql`, `mysql` | -| MariaDB vs MySQL | One engine `mysql`; both documented | -| Drivers | Optional extras; Docker image installs `[databases]` | -| Postgres driver | `psycopg[binary]` (v3) + gevent wait callback | -| MySQL driver | `PyMySQL` + `pymysql.install_as_MySQLdb()` before Django setup | -| Gunicorn | gevent for all engines | -| Pooler | Not required; `CONN_MAX_AGE` + health checks | -| Existing SQLite deploys | Stay on SQLite until the operator migrates | -| Migration | Documented offline dump/load **and** BETA CLI that stops gunicorn | -| Tests | Default pytest = SQLite; local Docker compose for Postgres + MariaDB + MySQL (functional **and** migrator); CI uses the same compose | - -## Architecture - -``` -LOGSTASHUI_DB_* → build_databases(DATA_DIR) → Django DATABASES['default'] - ├ sqlite3 DATA_DIR/db.sqlite3 - ├ postgresql psycopg 3 - └ mysql PyMySQL (MariaDB or MySQL) -``` - -All apps keep `Model.objects`, `transaction.atomic`, `select_for_update`, and existing migrations. No `raw()` / `cursor()` / `PRAGMA` outside `LogstashUI/database.py` (SQLite PRAGMAs stay there). - -`DATA_DIR` remains mandatory: TLS, `.django_secret_key`, logs, staticfiles. A remote database does not remove the PVC or bind-mount. - -`pymysql.install_as_MySQLdb()` runs once at process start, before `django.setup()`, when the MySQL extra is installed (safe no-op if engine is not mysql). - -## Configuration - -No YAML. No URL DSN. - -| Variable | Default | Rules | -|---|---|---| -| `LOGSTASHUI_DB_ENGINE` | `sqlite` | Aliases below | -| `LOGSTASHUI_DB_NAME` | sqlite: `DATA_DIR/db.sqlite3`; else `logstashui` | | -| `LOGSTASHUI_DB_HOST` | empty | **Required** for postgresql/mysql | -| `LOGSTASHUI_DB_PORT` | 5432 / 3306 | Engine default if unset | -| `LOGSTASHUI_DB_USER` | empty | **Required** for postgresql/mysql | -| `LOGSTASHUI_DB_PASSWORD` | empty | Secret / EnvironmentFile | -| `LOGSTASHUI_DB_SSLMODE` | postgres: `prefer`; mysql: unset | postgres: `disable`, `allow`, `prefer`, `require`, `verify-ca`, `verify-full` | -| `LOGSTASHUI_DB_SSL_CA` | empty | Path; mysql TLS and postgres `verify-*` | -| `LOGSTASHUI_DB_CONN_MAX_AGE` | `60` | `0` = close per request | -| `LOGSTASHUI_DB_CONN_HEALTH_CHECKS` | `true` | Django `CONN_HEALTH_CHECKS` | - -**Aliases** (normalize to canonical): - -- `sqlite3` → `sqlite` -- `postgres`, `postgresql` → `postgresql` -- `mysql`, `mariadb`, `my` → `mysql` - -**SQLite OPTIONS (unchanged):** `timeout=20`; `init_command` `PRAGMA busy_timeout=20000; PRAGMA journal_mode=WAL;` - -**MySQL OPTIONS:** `charset=utf8mb4`; `init_command=SET sql_mode='STRICT_TRANS_TABLES'`; collation `utf8mb4_bin` so unique `Policy.name` (and similar) match SQLite/Postgres case-sensitivity. MySQL’s default `_ci` collation would treat `Foo` and `foo` as a clash. - -**Postgres OPTIONS:** `sslmode` from env; `sslrootcert` when `LOGSTASHUI_DB_SSL_CA` is set. - -**Version floors** (docs; fail-fast if the server version is below): PostgreSQL **14+**, MariaDB **10.6+**, MySQL **8.0+**. - -Update `LogstashUI/packaging/logstashui.default`, `docs/docs/logstashui/configuration/environment.md`, the systemd generator (interactive prompt for engine + host/name/user; password from env or prompt), and CHANGELOG. - -## Packaging and Docker - -Default wheel: SQLite only (stdlib). - -``` -LogstashUI[postgres] → psycopg[binary] -LogstashUI[mysql] → PyMySQL -LogstashUI[databases] → both -``` - -Missing extra at runtime → `RuntimeError` naming the extra (`uv pip install 'LogstashUI[postgres]'`). - -**Docker/K8s image:** install `LogstashUI[databases]` (e.g. `uv pip install '/app[databases]'`). Operators set env; no extra pip in the cluster. - -Native pip/uv: extras are the supported advanced path. - -Python 3.12–3.14: `psycopg[binary]` wheels; PyMySQL is pure Python (no mysqlclient compile). Pin `psycopg[binary]` to a release that ships 3.14 wheels, or document build deps. - -## Gevent and connections - -Keep `--worker-class gevent` and `--worker-connections 1000`. - -- Register a psycopg3 **gevent wait callback** at process start so libpq does not block the hub. -- PyMySQL uses monkey-patched sockets; do not add mysqlclient. -- `CONN_MAX_AGE=60` and `CONN_HEALTH_CHECKS=true` by default. -- Do not require an external pooler. Document that `LOGSTASHUI_WORKERS` × in-flight greenlets must stay under the server’s `max_connections` (leave room for `migrate` and the test matrix). Optional PgBouncer is documented, not required. -- In-process `psycopg_pool` only if it is gevent-safe; otherwise skip for 0.5.2. - -**SQLite scale warning:** one WARNING at `logstashui serve` when the engine is sqlite and `LOGSTASHUI_WORKERS` > 1 (gunicorn default is 2). Message: SQLite is the small-install default; use PostgreSQL or MySQL/MariaDB for concurrent agents. Do not refuse to start. No UI modal. - -## Fail-fast (before gunicorn bind) - -| Condition | Result | -|---|---| -| Unknown engine | `RuntimeError`, list canonical names + aliases | -| postgresql/mysql, driver not installed | `RuntimeError` + extra name | -| postgresql/mysql, HOST/NAME/USER empty | `RuntimeError` naming the empty vars | -| Cannot connect during `migrate` | Django error; serve exits non-zero | -| SSL verify fail | Driver error; do not swallow | - -Never log passwords. INFO may log engine, host, and name. - -## Migration off SQLite - -One mechanism: Django `dumpdata` → target `migrate` → `loaddata` → `sqlsequencereset`. - -`DATA_DIR` does not move. `.django_secret_key` must stay or Fernet keystore rows will not decrypt. - -### Offline (supported) - -Documented in deploy/environment: - -1. Stop LogstashUI. -2. Copy `DATA_DIR/db.sqlite3` to a backup. -3. Dump from SQLite. -4. Set `LOGSTASHUI_DB_*` for the server. -5. `logstashui manage migrate --noinput`. -6. `loaddata`. -7. `sqlsequencereset` (Postgres sequences; document as skippable/no-op guidance for MySQL). -8. Start LogstashUI. Verify login, a policy, and an agent. - -### BETA CLI - -`logstashui migrate-engine --to postgresql|mysql --i-have-a-backup` - -1. Print BETA + backup warning; refuse without `--i-have-a-backup`. -2. Source must be sqlite; target must be postgresql or mysql; extras must be installed. -3. If gunicorn is running (pidfile written by `serve` in 0.5.2, or `--pid`), **SIGTERM it**. That is maintenance: the UI port closes. No in-app wizard. -4. `PRAGMA wal_checkpoint(TRUNCATE)` on SQLite. -5. `dumpdata --natural-foreign --natural-primary`, exclude `contenttypes`, `auth.permission`, `sessions`. -6. Target `DATABASES` from env. `migrate --noinput`, `loaddata`, `sqlsequencereset`. -7. Print remaining env. Do not edit `/etc/default/logstashui` unless `--write-env PATH`. Do not auto-restart serve (systemd `Restart=` would race). Document `systemctl stop` → migrate → `systemctl start`. - -No reverse migrator. No sqlite→sqlite. If a pidfile points at a live process and SIGTERM fails, refuse. - -## Testing - -### Default inner loop (unchanged) - -`pytest` uses SQLite via settings. **All existing tests must pass** with no Docker and no extras. - -Update `test_database.py`: postgresql/mysql no longer raise “not implemented”; they return a valid `DATABASES` dict (connect may be skipped in unit tests). New unit tests: aliases, missing driver, missing HOST, sqlite default path, SSL/CONN keys. - -### Local Docker matrix (required) - -Ship `docker/docker-compose.db.yml` with **three** databases for local use and CI: - -- PostgreSQL 16 (or 14+) -- MariaDB 10.11+ (or 10.6+) -- MySQL 8.0+ - -Fixed test credentials in the compose file (not production). Healthchecks so tests wait for ready. - -Ship `bin/test_databases.sh` (and `bin/test_databases.bat` to match the existing Windows start scripts): - -1. `docker compose -f docker/docker-compose.db.yml up -d --wait` -2. Run the **existing pytest suite** against each server engine (`LOGSTASHUI_DB_ENGINE` plus host/port/user/password/name pointing at the container). A SQLite run remains the first step (no compose required for that step). -3. Run **migration tests**: a fixture SQLite database → dump/load (the same helpers as `migrate-engine`) into Postgres, MariaDB, and MySQL; assert a `Policy`, `User`, and JSONField row survive. -4. Tear down compose unless `--keep`. - -This is how a developer proves basic functionality and migration without waiting on CI. CI calls the same script. - -If Docker is unavailable, `bin/test_databases.sh` fails clearly; default `pytest` still works. - -Do **not** put Postgres/MariaDB/MySQL in `docker-compose.yml` or the smoke stack by default. Smoke stays SQLite so product CA and PUID behavior stay unchanged. - -### Pre-existing pytest failures - -`test_update_policy_default_policy_forbidden`, `test_delete_policy_default_policy_forbidden`, and `test_clone_policy_success` were already failing before this work. 0.5.2 database work must not add new failures. Do not silently “fix” those unless they fail **because of** engine differences. - -## Dialect traps (must handle) - -- **Unique case-sensitivity:** MySQL `utf8mb4_bin` (or equivalent) so `Policy.name` unique matches SQLite/Postgres. -- **JSONField:** Connection `status_blob`, Revision `snapshot_json`, SNMP metadata/templates. Native JSON on all three version floors. -- **`select_for_update`:** real row locks on Postgres/MySQL; first-user creation in `Management/views.py`. Tests that mock it stay. Add one real concurrency test on a server engine if cheap. -- **`db_table = 'settings'`:** must migrate cleanly on MySQL (quoted identifier). -- **Index/constraint name length:** MySQL 64 characters; existing `UniqueConstraint` names are short enough — verify `migrate` on MySQL. -- **Timezones:** `USE_TZ=True`; Django maps Postgres timestamptz vs MySQL datetime if USE_TZ stays. -- **BooleanField / BigAutoField:** ORM. -- **RunPython migrations:** ORM-based; portable. No new SQLite-only `RunSQL`. -- **loaddata PKs:** `sqlsequencereset` after load on Postgres. -- **Encrypted CharFields:** copied as Fernet strings; secret key stays in `DATA_DIR`. - -## Docs and operator surfaces - -- `docs/docs/logstashui/configuration/environment.md` — replace “sqlite only”. -- `docs/docs/logstashui/general/deploy.md` — PVC still required; the database may be external. -- Offline migration procedure + BETA CLI warnings. -- SQLite scale warning explained. -- Extras vs Docker `[databases]`. -- Optional PgBouncer note. -- `CHANGELOG.md` for 0.5.2. -- systemd sample env keys (already stubbed in `logstashui.default`). -- Kubernetes: ConfigMap for engine/host/name/user + Secret for password; PVC still for `DATA_DIR`. - -## Obstacles - -1. **Gevent × `max_connections`:** default 2 workers is acceptable for small Postgres; document; do not open thousands of sessions. -2. **psycopg3 + gevent:** the wait callback is mandatory; missing it looks like random hangs. -3. **mysqlclient vs PyMySQL:** the C client blocks the hub; PyMySQL is the gevent choice; Django sees it via `install_as_MySQLdb`. -4. **MySQL unique collation** silently changes uniqueness vs SQLite if left at `_ci`. -5. **dumpdata/loaddata** is not a perfect replica (sessions dropped, contenttypes excluded); operators re-login. -6. **Stopping gunicorn** for BETA migrate races with systemd `Restart=`; do not auto-restart; document stop → migrate → start. -7. **Driver wheels on 3.14:** PyMySQL is safe; pin `psycopg[binary]` to a release with 3.14 wheels, or document build deps. -8. **Hatch extras** must not pull mysqlclient into the default wheel. -9. **Tests that assume sqlite paths** (`db.sqlite3` in `paths.py` legacy migrate) stay sqlite-only. -10. **Smoke stack** stays sqlite so CA/PUID work is unchanged. - -## Success criteria - -- Unset engine → SQLite, same as 0.5.1. -- `LOGSTASHUI_DB_ENGINE=postgresql` or `mysql` with valid env → migrate + serve. -- Default `pytest` green on SQLite (no new failures). -- `bin/test_databases.sh` against local Docker Postgres, MariaDB, and MySQL: existing suite + dump/load migration assertions. -- BETA `migrate-engine` refuses without `--i-have-a-backup` and stops gunicorn when a pidfile is live. -- Docker image can use all three engines via env only. -- Docs match env keys. -- Product CA and `DATA_DIR` behavior unchanged. - -## Implementation order - -1. Extras + `build_databases()` + fail-fast + unit tests (SQLite pytest). -2. PyMySQL shim + psycopg gevent wait + `CONN_*`. -3. `docker-compose.db.yml` + `bin/test_databases.sh`; existing suite passing on three servers. -4. SQLite scale warning; systemd/docs/sample env. -5. `serve` pidfile; `migrate-engine` BETA; migration tests in the same script. -6. Docker image `[databases]`. -7. CHANGELOG 0.5.2. - -## Spec self-review - -- **Placeholders:** none. Version floors, env keys, extras, compose services, and CLI flags are explicit. -- **Consistency:** architecture (ORM switch only) matches packaging, gevent, migrator, and tests. Default remains SQLite everywhere except when env selects a server engine. -- **Scope:** one release-sized feature (engine wiring + migrator + test matrix). No Helm, no worker-class split, no repository layer. -- **Ambiguity resolved:** `mysql` covers MariaDB and MySQL; migrator stops the UI port rather than serving a maintenance page; local Docker is required for the three-engine script, not for default pytest. diff --git a/docs/superpowers/specs/2026-09-02-airgap-freeze-design.md b/docs/superpowers/specs/2026-09-02-airgap-freeze-design.md deleted file mode 100644 index eeea23e..0000000 --- a/docs/superpowers/specs/2026-09-02-airgap-freeze-design.md +++ /dev/null @@ -1,181 +0,0 @@ -# 2026-09-02 — Air-gapped freeze (optional offline zip) - -**LogstashUI version:** 0.5.2 (packaging feature; does not bump by itself) -**Builder:** Linux x86_64, CPython **3.12**, uv, Docker (for the image tarball) -**Isolated host (wheels):** Linux x86_64, CPython **3.12** (venv module / distro `python3.12-venv`) -**Isolated host (docker):** Docker Engine -**Isolated host (standalone):** glibc-compatible Linux x86_64, no Python -**Chosen approach:** Optional freeze **script** that emits up to three artifacts. Default `uv build` / hatchling / published image **unchanged**. - -## Goal - -A maintainer with internet builds zip files that an air-gapped operator can copy, unpack, and run **without PyPI or a container registry**. This is not the default packaging path. - -## Non-goals (v1) - -- Changing hatchling includes, extras, or `uv build` output. -- A `logstashui freeze` CLI subcommand. -- Bundling LogstashAgent (separate product; compose `embedded` profile stays online-registry). -- Bundling CPython into the wheelhouse (host must already have 3.12). -- arm64 or Windows artifacts (names keep platform/ABI so those can be added later). -- Multiple CPython ABIs in one zip (cp312 only). -- Shipping uv, Node, or a compiler to the isolated host. -- Helm, K8s operator, or air-gap registry mirroring as this feature. -- Making freeze a required PR CI gate. -- Auto-update / delta patches of a freeze. - -## Decisions (locked) - -| Topic | Decision | -|---|---| -| Builder entry | `bin/freeze_logstashui.sh` (bash). Not default packaging. | -| Artifacts | Wheels zip **and** Docker tarball **and** standalone zip. Operator tests all three. | -| Default `uv build` | Unchanged (sdist + py3-none-any wheel only). | -| Python ABI | CPython **3.12** x86_64 only. Documented pin. | -| Platform | Linux x86_64 only. Later: arm64 / Windows as extra tags, not a rename. | -| Extras | Always include `LogstashUI[databases]` (psycopg binary + PyMySQL). | -| Agent | Not in any freeze zip. | -| Isolated run | Each zip has `README.md` + helper (`install.sh` / `load.sh` / `run.sh`). | -| Config | Same env as today. No YAML. `LOGSTASHUI_DATA_DIR` default `$(pwd)/logstashui_data`. | -| Standalone tool | PyInstaller. **Experimental** until smoke covers migrate + SNMP sync + TLS serve. | -| Wheel policy | Zip contains **only `.whl`**. Prefer manylinux2014 then manylinux_2_28. Pure-Python sdists are wheeled on the builder; native sdist-only → freeze **fails**. | -| Pins | Resolve from `uv.lock` (`uv export --frozen`). | -| PyInstaller in deps | **No.** Freeze script uses `uvx pyinstaller` (or a throwaway venv). Do not add to `[project]` or default `dev`. | -| CI | Optional `workflow_dispatch` only. Not on every PR. | - -## Architecture - -``` -connected builder (linux x86_64, python3.12, uv, docker) - | - +-- bin/freeze_logstashui.sh [--wheels] [--docker] [--standalone] [--all] - | - +-- dist/offline/logstashui-{ver}-offline-wheels-linux-x86_64-cp312.zip - +-- dist/offline/logstashui-{ver}-offline-docker-linux-x86_64.zip - +-- dist/offline/logstashui-{ver}-offline-standalone-linux-x86_64.zip - -isolated host copies zip → unzip → helper → logstashui serve -``` - -`--all` if no artifact flags. `--output DIR` default `dist/offline`. `--image NAME` for docker-save (skip build). `--version` taken from `pyproject.toml`. - -Output stays under gitignored `/dist/`. - -## Artifact 1 — Wheelhouse - -**Builder steps** - -1. Ensure Tailwind CSS exists (`src/logstashui/theme/static/css/dist/styles.css`); build it the same way `uv build` / Dockerfile does if missing. -2. `uv build` (reuse the normal wheel; do not invent a second hatch target). -3. `uv export --frozen --no-dev --extra databases --no-emit-project -o dist/offline/requirements-offline.txt` (keep hashes). -4. Copy the local LogstashUI wheel into `wheels/`. Download deps with `packaging/offline/download_wheels.py` (uv 0.12 has no `uv pip download`): try `manylinux2014_x86_64` then `manylinux_2_28_x86_64` `--only-binary :all:`. Pure-Python sdists (e.g. `django-login-required-middleware==0.9.0`) are wheeled on the **builder** to `py3-none-any`. Native packages with no manylinux wheel fail the freeze. -5. Copy `LICENSE.txt`, `NOTICE.txt`, generated `install.sh`, `README.md`, `MANIFEST.txt`, `SHA256SUMS.txt`. -6. Zip. Fail if any `.tar.gz` sdist landed in `wheels/`. Isolated host needs glibc 2.28+. - -**MANIFEST.txt** must list: LogstashUI version, git sha (or `unknown` if not a git checkout), CPython 3.12, platform `linux-x86_64`, extra `databases`, package list with versions. - -**Isolated `install.sh`** - -- `PYTHON=${PYTHON:-python3.12}` -- Refuse unless `sys.version_info[:2] == (3, 12)` and the interpreter is 64-bit. -- `$PYTHON -m venv .venv` (do **not** `pip install --upgrade pip` — that hits PyPI). -- `.venv/bin/python -m pip install --no-index --no-cache-dir --find-links ./wheels 'LogstashUI[databases]'` -- Print: activate / `.venv/bin/logstashui serve`, data-dir default, env pointer to the configuration docs (copied as a short README section, not a second docs site). - -`install.sh` must not reach the network. `pip` `--no-index` is required. Prefer `--disable-pip-version-check`. - -Isolated host packages: `python3.12` and the distro venv module (Debian/Ubuntu: `python3.12-venv`). **uv is not required** on the isolated host. - -## Artifact 2 — Docker tarball - -**Builder** - -- If `--image` is set: `docker save` that **local** image. **Never** `docker pull`. Fail if the name is missing. -- Else: `docker build -f docker/Dockerfile -t logstashui:offline-{version} .` from repo root (same file as K8s/compose), then save that tag. -- Write `image.tar.gz` (`docker save | gzip`). Zip it with `load.sh`, UI-only `compose.offline.yml`, `README.md`, `SHA256SUMS.txt`, license files. Same zip story as the other two artifacts. - -**`compose.offline.yml`** - -- **One** service: LogstashUI. **No** Agent, **no** `embedded` profile. -- Image name matches what `docker load` will register. -- Port **8443**, `LOGSTASHUI_TLS` on, `LOGSTASHUI_DATA_DIR=/var/lib/logstashui`, named or bind volume for data. -- Same env-first model as Option 1. Operator fills `LOGSTASHUI_ALLOWED_HOSTS` / `LOGSTASHUI_DB_*` as needed. - -**Isolated `load.sh`:** `docker load -i `, print the image name, print `docker compose -f compose.offline.yml up -d`. - -## Artifact 3 — Standalone (experimental) - -PyInstaller **onedir** (not onefile — Django data files + gunicorn + gevent extract less painfully). Entry: `LogstashUI.cli:main`. - -Spec + helper templates live in repo `packaging/offline/` (not inside the Django wheel). Freeze script invokes `uvx pyinstaller` with that spec. - -Must collect: - -- All Django apps that the hatch wheel force-includes (`LogstashUI`, `PipelineManager`, `Management`, `Utilities`, `SNMP`, `Monitoring`, `Site`, `Documentation`, `AI`, `theme`, `Common`). -- Templates, static (including built Tailwind CSS), SNMP official data, packaged docs. -- Hidden imports: Django, gunicorn, gevent, greenlet, cryptography, pysnmp, lark, pygrok, whitenoise, psycopg, pymysql, yaml, htmx/tailwind Django apps. - -**Why experimental:** Django loads apps/commands by name; gunicorn+gevent fork and monkey-patch; native wheels (`cryptography`, `psycopg`, `gevent`) need the same glibc. A missed hidden import or data file often boots then dies on `migrate`, SNMP, or the first TLS handshake. The freeze **must** document that. - -**Pass criteria to drop “experimental”:** isolated-like run (no network) of `./logstashui serve` completes migrate + SNMP sync + collectstatic, binds **8443** HTTPS, `GET /` is not 500. Until then README says experimental; `run.sh` is `./logstashui serve`. - -If gevent+PyInstaller cannot pass that smoke, do **not** silently switch the product default worker. Document the failure and stop; a sync/gthread workaround is a spec amendment, not a surprise. - -## Isolated runtime (all three) - -Unchanged product behavior after install: - -- `logstashui serve` (migrate, SNMP official sync, collectstatic, product CA, gunicorn HTTPS :8443). -- Env: `LOGSTASHUI_*` / `LOGSTASHUI_DB_*` as today. SQLite default; Postgres/MySQL if they have a server. -- Do not rotate the product CA. Do not relocate `DATA_DIR` into the zip. -- systemd: after wheelhouse install, `logstashui systemd` still works if they have root (helper does **not** enable the unit). Standalone: document `ExecStart=` path to the unpacked binary; do not auto-write units in v1. - -## Error handling - -| Condition | Result | -|---|---| -| Builder not CPython 3.12 | Exit non-zero, print required ABI | -| Missing uv | Exit, how to install uv | -| `--docker` and Docker missing / image missing | Exit | -| `--standalone` and `uvx pyinstaller` fails | Exit | -| Dep has no manylinux cp312 binary wheel | Exit, name the package (no sdist fallback) | -| Tailwind CSS missing and npm build fails | Exit | -| Isolated `install.sh` wrong Python | Exit, “need CPython 3.12 x86_64” | -| Isolated pip would use PyPI | `--no-index` makes it fail; do not catch and retry online | - -Never log DB passwords. MANIFEST may list package names/versions. - -## Testing - -Not a default pytest matrix job. - -1. **Wheels (required for merge of the script):** run freeze `--wheels`; assert zip contains only `.whl` + scripts/docs; `docker run --network=none -v $PWD/zipdir:/offline python:3.12-slim` runs `install.sh` and `logstashui --help` plus `logstashui manage check`. -2. **Docker:** freeze `--docker`; `docker load`; `docker image inspect` the tag. Full compose up is manual. -3. **Standalone:** freeze `--standalone`; binary `--help`. Full `serve` smoke is the experiment gate (can be `--network=none` on a throwaway dir). - -Optional GitHub Action: `workflow_dispatch`, linux runner, `--wheels` only (fastest, no PyInstaller cache pain). `--docker` / `--standalone` stay maintainer commands. - -## Docs - -New page: `docs/docs/logstashui/general/offline.md` (air-gap freeze). Link from `deploy.md` as an extra option **after** Docker / pip / systemd / K8s. State clearly: not the recommended connected-network install. - -Page covers: builder prerequisites, the three zips, CPython 3.12 pin, `[databases]` included, no Agent, how to set env, DATA_DIR, and “later: arm64 / Windows”. - -CHANGELOG under 0.5.2: optional freeze script; default wheel unchanged. - -## Layout (repo) - -``` -bin/freeze_logstashui.sh # builder -packaging/offline/ # PyInstaller spec, README/install/load/run templates -docs/docs/logstashui/general/offline.md -``` - -Do not put freeze helpers inside `src/logstashui/LogstashUI/packaging/` (that tree is systemd templates shipped in the wheel). - -## Later (not v1) - -- `linux/arm64` and `win_amd64` as additional freeze invocations (`--python-platform` / Windows builder). -- LogstashAgent sibling freeze script. -- Promoting standalone from experimental after the serve smoke exists. -- Bundling a CPython embed for hosts with no 3.12 (different product). From 638acfbc0db84b377e3d46792e42fd31b7bda3ec Mon Sep 17 00:00:00 2001 From: Aaron Mildenstein Date: Wed, 2 Sep 2026 18:10:50 -0600 Subject: [PATCH 25/62] feat: LS version pill, VERSION binary path, unreleased agent Show a cyan LS X.Y.Z oval for the running Logstash version on ConnectionManager and the Policy editor Agents tab. Persist VERSION pins as {download_dir}/logstash-X.Y.Z/bin. Agents newer than preferred show "unreleased version" instead of a backwards Upgrade. --- CHANGELOG.md | 6 + .../PipelineManager/agent_policies.py | 5 + .../PipelineManager/agent_versions.py | 91 +++++++++++++ .../PipelineManager/manager_views.py | 15 ++ .../PipelineManager/policies_crud.py | 21 +++ .../static/js/agent_policies.js | 61 +++++++++ .../templates/pipeline_manager.html | 7 +- .../tests/test_agent_policies.py | 22 +++ .../tests/test_agent_versions.py | 128 ++++++++++++++++++ .../tests/test_manager_views.py | 57 ++++++++ .../tests/test_policies_crud.py | 72 ++++++++++ 11 files changed, 484 insertions(+), 1 deletion(-) create mode 100644 src/logstashui/PipelineManager/agent_versions.py create mode 100644 src/logstashui/PipelineManager/tests/test_agent_versions.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 3baa720..45cac5e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -43,6 +43,12 @@ SQLite does not scale under gunicorn/gevent. Operators can now run LogstashUI on - Example manifests under `docs/docs/logstashui/kubernetes/examples/{sqlite,postgresql,mysql}/`. - Database subsection: engines, every `LOGSTASHUI_DB_*` default, offline dump/load, BETA `migrate-engine`, CREATE DATABASE scripts (`utf8mb4_bin` for MySQL/MariaDB), schema snapshots from 0.5.2 `migrate`. +### Agent version display + +- ConnectionManager and Policy editor Agents tab show a cyan **LS X.Y.Z** pill for the Logstash version the agent reported (`logstash_version_resolved`, else Logstash API version). Hidden until known. +- Policy Source **VERSION** live-fills and persists Binary Path as `{download_dir}/logstash-{version}/bin` (default `/opt/logstash-agent/logstash-versions/logstash-X.Y.Z/bin`). Custom paths are kept. Switching back to SYSTEM restores `/usr/share/logstash/bin` when the field still looks derived. +- Agent newer than preferred (`__PREFERRED_LS_AGENT_VERSION__`, currently 0.5.1) shows **unreleased version** instead of a backwards Upgrade button. Older agents still get Upgrade. Unparseable versions still get Upgrade. + ## [0.5.1] - Agent control plane, SNMP NMS, dual HTTPS - 08/31/2026 diff --git a/src/logstashui/PipelineManager/agent_policies.py b/src/logstashui/PipelineManager/agent_policies.py index be97166..d98e380 100644 --- a/src/logstashui/PipelineManager/agent_policies.py +++ b/src/logstashui/PipelineManager/agent_policies.py @@ -5,6 +5,7 @@ from django.http import JsonResponse from PipelineManager.models import Revision, Policy, Connection as ConnectionTable, Keystore +from PipelineManager.agent_versions import resolve_running_logstash_version from Common.decorators import require_admin_role @@ -636,6 +637,10 @@ def get_policy_nodes(request): "status_class": status_class, "last_check_in": node.last_check_in.isoformat() if node.last_check_in else None, "agent_version": node.status_blob.get('agent_version') if node.status_blob else None, + "logstash_version": resolve_running_logstash_version( + logstash_version_resolved=node.logstash_version_resolved, + status_blob=node.status_blob if isinstance(node.status_blob, dict) else None, + ), "cpm_enabled": cpm_enabled, "snmp_networks": sorted(snmp_networks_by_conn.get(node.id, [])), }) diff --git a/src/logstashui/PipelineManager/agent_versions.py b/src/logstashui/PipelineManager/agent_versions.py new file mode 100644 index 0000000..a505e44 --- /dev/null +++ b/src/logstashui/PipelineManager/agent_versions.py @@ -0,0 +1,91 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +"""Logstash / agent version display and VERSION binary-path helpers.""" + +from __future__ import annotations + +from packaging.version import InvalidVersion, Version + +SYSTEM_BINARY_PATH = "/usr/share/logstash/bin" +DEFAULT_DOWNLOAD_DIR = "/opt/logstash-agent/logstash-versions" + + +def derive_version_binary_path(download_dir: str | None, version: str | None) -> str | None: + ver = (version or "").strip() + if not ver: + return None + root = (download_dir or DEFAULT_DOWNLOAD_DIR).rstrip("/") or DEFAULT_DOWNLOAD_DIR + return f"{root}/logstash-{ver}/bin" + + +def is_derived_version_binary_path(path: str | None, download_dir: str | None) -> bool: + p = (path or "").rstrip("/") + if not p: + return False + root = (download_dir or DEFAULT_DOWNLOAD_DIR).rstrip("/") or DEFAULT_DOWNLOAD_DIR + prefix = f"{root}/logstash-" + suffix = "/bin" + if not (p.startswith(prefix) and p.endswith(suffix)): + return False + mid = p[len(prefix) : -len(suffix)] + return bool(mid) and "/" not in mid + + +def resolve_running_logstash_version( + *, + logstash_version_resolved: str | None = None, + status_blob: dict | None = None, +) -> str | None: + resolved = (logstash_version_resolved or "").strip() + if resolved: + return resolved + if not isinstance(status_blob, dict): + return None + api = status_blob.get("logstash_api") + if isinstance(api, dict): + ver = str(api.get("version") or "").strip() + if ver: + return ver + for key in ("logstash_version_resolved", "logstash_version"): + ver = str(status_blob.get(key) or "").strip() + if ver: + return ver + return None + + +def agent_version_relation(current: str | None, preferred: str | None) -> str: + try: + cur = Version(str(current or "").strip()) + pref = Version(str(preferred or "").strip()) + except InvalidVersion: + return "unknown" + if cur < pref: + return "older" + if cur > pref: + return "newer" + return "equal" + + +def resolve_persisted_binary_path( + *, + source: str | None, + version: str | None, + download_dir: str | None, + binary_path: str | None, +) -> str: + current = (binary_path or "").strip() or SYSTEM_BINARY_PATH + src = (source or "SYSTEM").upper() + derived = derive_version_binary_path(download_dir, version) + if src == "VERSION": + if derived and ( + not (binary_path or "").strip() + or current.rstrip("/") == SYSTEM_BINARY_PATH.rstrip("/") + or is_derived_version_binary_path(current, download_dir) + ): + return derived + return current + if is_derived_version_binary_path(current, download_dir): + return SYSTEM_BINARY_PATH + return current diff --git a/src/logstashui/PipelineManager/manager_views.py b/src/logstashui/PipelineManager/manager_views.py index 89802cf..2f801ad 100644 --- a/src/logstashui/PipelineManager/manager_views.py +++ b/src/logstashui/PipelineManager/manager_views.py @@ -86,6 +86,10 @@ def PipelineManager(request): pass from PipelineManager.agent_modes import is_embedded_connection + from PipelineManager.agent_versions import ( + agent_version_relation, + resolve_running_logstash_version, + ) connections = [ conn @@ -93,6 +97,7 @@ def PipelineManager(request): "connection_type", "name", "host", "cloud_id", "cloud_url", "pk", "policy__name", "policy_id", "policy__policy_type", "agent_id", "last_check_in", "status_blob", "desired_agent_version", + "logstash_version_resolved", ) if not is_embedded_connection(conn) ] @@ -109,6 +114,16 @@ def PipelineManager(request): # reports "unknown" (keeps the page-load fallback inspect card in sync # with the live AgentInspect endpoint). _normalize_status_blob_api_status(conn.get('status_blob')) + + blob = conn.get("status_blob") if isinstance(conn.get("status_blob"), dict) else None + agent_ver = blob.get("agent_version") if blob else None + conn["logstash_version"] = resolve_running_logstash_version( + logstash_version_resolved=conn.get("logstash_version_resolved"), + status_blob=blob, + ) + conn["agent_version_relation"] = agent_version_relation( + agent_ver, settings.__PREFERRED_LS_AGENT_VERSION__ + ) # Sort connections: centralized first, then by policy name # This groups agents with the same policy together diff --git a/src/logstashui/PipelineManager/policies_crud.py b/src/logstashui/PipelineManager/policies_crud.py index 65191e0..3be030e 100644 --- a/src/logstashui/PipelineManager/policies_crud.py +++ b/src/logstashui/PipelineManager/policies_crud.py @@ -160,6 +160,18 @@ def _optional_int(key, default): apply_simulate_path_bundle(policy) policy.save() + from PipelineManager.agent_versions import resolve_persisted_binary_path + + resolved_binary = resolve_persisted_binary_path( + source=policy.logstash_source, + version=policy.logstash_version, + download_dir=policy.logstash_download_dir, + binary_path=policy.binary_path, + ) + if resolved_binary != policy.binary_path: + policy.binary_path = resolved_binary + policy.save(update_fields=["binary_path"]) + # Generate enrollment token for the new policy enrollment_token = secrets.token_urlsafe(32) @@ -307,6 +319,15 @@ def update_policy(request): if 'log4j2_properties' in data: policy.log4j2_properties = data['log4j2_properties'] + from PipelineManager.agent_versions import resolve_persisted_binary_path + + policy.binary_path = resolve_persisted_binary_path( + source=policy.logstash_source, + version=policy.logstash_version, + download_dir=policy.logstash_download_dir, + binary_path=policy.binary_path, + ) + policy.save() logger.info(f"User '{request.user.username}' updated policy '{policy_name}'") diff --git a/src/logstashui/PipelineManager/static/js/agent_policies.js b/src/logstashui/PipelineManager/static/js/agent_policies.js index 46d00d8..60ed09e 100644 --- a/src/logstashui/PipelineManager/static/js/agent_policies.js +++ b/src/logstashui/PipelineManager/static/js/agent_policies.js @@ -6,6 +6,46 @@ let originalFileContents = {}; let changedFiles = new Set(); const DRAFT_POLICY_VALUE = '__draft__'; +const SYSTEM_BINARY_PATH = '/usr/share/logstash/bin'; +const DEFAULT_LS_DOWNLOAD_DIR = '/opt/logstash-agent/logstash-versions'; + +function deriveVersionBinaryPath(downloadDir, version) { + const ver = (version || '').trim(); + if (!ver) return null; + const root = (downloadDir || DEFAULT_LS_DOWNLOAD_DIR).replace(/\/+$/, '') || DEFAULT_LS_DOWNLOAD_DIR; + return `${root}/logstash-${ver}/bin`; +} + +function isDerivedVersionBinaryPath(path, downloadDir) { + const p = (path || '').replace(/\/+$/, ''); + if (!p) return false; + const root = (downloadDir || DEFAULT_LS_DOWNLOAD_DIR).replace(/\/+$/, '') || DEFAULT_LS_DOWNLOAD_DIR; + const prefix = `${root}/logstash-`; + const suffix = '/bin'; + if (!p.startsWith(prefix) || !p.endsWith(suffix)) return false; + const mid = p.slice(prefix.length, p.length - suffix.length); + return mid.length > 0 && !mid.includes('/'); +} + +function syncVersionBinaryPath() { + const sourceEl = document.getElementById('logstashSource'); + const versionEl = document.getElementById('logstashVersion'); + const downloadEl = document.getElementById('logstashDownloadDir'); + const binaryEl = document.getElementById('binaryPath'); + if (!binaryEl) return; + const source = sourceEl?.value || 'SYSTEM'; + const version = versionEl?.value || ''; + const downloadDir = downloadEl?.value || DEFAULT_LS_DOWNLOAD_DIR; + const current = binaryEl.value.trim(); + const derived = deriveVersionBinaryPath(downloadDir, version); + if (source === 'VERSION') { + if (derived && (!current || current === SYSTEM_BINARY_PATH || isDerivedVersionBinaryPath(current, downloadDir))) { + binaryEl.value = derived; + } + } else if (isDerivedVersionBinaryPath(current, downloadDir)) { + binaryEl.value = SYSTEM_BINARY_PATH; + } +} const POLICY_TYPE_INFO = { PACKAGED: { @@ -1599,6 +1639,7 @@ document.addEventListener('DOMContentLoaded', function() { if (hint) { hint.classList.toggle('hidden', !showVersion); } + syncVersionBinaryPath(); } function setPolicyFieldValue(id, value) { @@ -1791,6 +1832,22 @@ document.addEventListener('DOMContentLoaded', function() { toggleVersionFieldsVisibility(); if (typeof detectChanges === 'function') detectChanges(); }); + document.getElementById('logstashVersion')?.addEventListener('input', () => { + syncVersionBinaryPath(); + if (typeof detectChanges === 'function') detectChanges(); + }); + document.getElementById('logstashVersion')?.addEventListener('change', () => { + syncVersionBinaryPath(); + if (typeof detectChanges === 'function') detectChanges(); + }); + document.getElementById('logstashDownloadDir')?.addEventListener('input', () => { + syncVersionBinaryPath(); + if (typeof detectChanges === 'function') detectChanges(); + }); + document.getElementById('logstashDownloadDir')?.addEventListener('change', () => { + syncVersionBinaryPath(); + if (typeof detectChanges === 'function') detectChanges(); + }); // Load policies on page load, auto-selecting a policy if policy_id is in the URL const _urlParams = new URLSearchParams(window.location.search); @@ -1896,6 +1953,7 @@ async function loadPolicyData(policyValue) { if (versionEl) versionEl.value = policy.logstash_version || ''; const downloadEl = document.getElementById('logstashDownloadDir'); if (downloadEl) downloadEl.value = policy.logstash_download_dir || '/opt/logstash-agent/logstash-versions'; + syncVersionBinaryPath(); const agentPortEl = document.getElementById('agentApiPort'); if (agentPortEl) agentPortEl.value = policy.agent_api_port ?? 9500; const lsPortEl = document.getElementById('logstashApiPort'); @@ -3534,6 +3592,9 @@ function renderPolicyNodes(nodes) { const badges = [ `LogstashAgent` ]; + if (node.logstash_version) { + badges.push(`LS ${node.logstash_version}`); + } if (node.cpm_enabled) { badges.push(`CPM`); } diff --git a/src/logstashui/PipelineManager/templates/pipeline_manager.html b/src/logstashui/PipelineManager/templates/pipeline_manager.html index 7935507..0387ae8 100644 --- a/src/logstashui/PipelineManager/templates/pipeline_manager.html +++ b/src/logstashui/PipelineManager/templates/pipeline_manager.html @@ -219,7 +219,9 @@

Connection Manager

v{{ connection.status_blob.agent_version }} {% if connection.desired_agent_version and connection.status_blob.agent_version != connection.desired_agent_version %} Upgrading - {% elif connection.status_blob.agent_version != preferred_agent_version %} + {% elif connection.agent_version_relation == "newer" %} + unreleased version + {% elif connection.agent_version_relation == "older" or connection.agent_version_relation == "unknown" %} @@ -230,6 +232,9 @@

Connection Manager

{% if connection.feature_agent %} LogstashAgent {% endif %} + {% if connection.logstash_version %} + LS {{ connection.logstash_version }} + {% endif %} {% if connection.feature_cpm %} CPM {% endif %} diff --git a/src/logstashui/PipelineManager/tests/test_agent_policies.py b/src/logstashui/PipelineManager/tests/test_agent_policies.py index 40ccb4c..54827ba 100644 --- a/src/logstashui/PipelineManager/tests/test_agent_policies.py +++ b/src/logstashui/PipelineManager/tests/test_agent_policies.py @@ -980,3 +980,25 @@ def test_delete_keystore_entry_invalid_json(self, authenticated_client): data = response.json() assert data['success'] is False assert 'Invalid JSON data' in data['error'] + + +@pytest.mark.django_db +class TestGetPolicyNodes: + def test_includes_logstash_version(self, authenticated_client, test_policy, test_agent_connection): + test_agent_connection.logstash_version_resolved = '9.4.3' + test_agent_connection.status_blob = {'agent_version': '0.5.1'} + test_agent_connection.save() + response = authenticated_client.get( + f'/ConnectionManager/GetPolicyNodes/?policy_id={test_policy.id}' + ) + assert response.status_code == 200 + node = response.json()['nodes'][0] + assert node['logstash_version'] == '9.4.3' + assert node['agent_version'] == '0.5.1' + + def test_logstash_version_null_when_unknown(self, authenticated_client, test_policy, test_agent_connection): + response = authenticated_client.get( + f'/ConnectionManager/GetPolicyNodes/?policy_id={test_policy.id}' + ) + node = response.json()['nodes'][0] + assert node.get('logstash_version') in (None, '') diff --git a/src/logstashui/PipelineManager/tests/test_agent_versions.py b/src/logstashui/PipelineManager/tests/test_agent_versions.py new file mode 100644 index 0000000..c3a077f --- /dev/null +++ b/src/logstashui/PipelineManager/tests/test_agent_versions.py @@ -0,0 +1,128 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +from PipelineManager.agent_versions import ( + SYSTEM_BINARY_PATH, + agent_version_relation, + derive_version_binary_path, + is_derived_version_binary_path, + resolve_persisted_binary_path, + resolve_running_logstash_version, +) + + +class TestDeriveVersionBinaryPath: + def test_default_dir(self): + assert derive_version_binary_path(None, "9.4.3") == ( + "/opt/logstash-agent/logstash-versions/logstash-9.4.3/bin" + ) + + def test_custom_dir_strips_slash(self): + assert derive_version_binary_path("/opt/ls/vers/", "8.17.1") == ( + "/opt/ls/vers/logstash-8.17.1/bin" + ) + + def test_empty_version_returns_none(self): + assert derive_version_binary_path("/opt/ls", "") is None + assert derive_version_binary_path("/opt/ls", None) is None + assert derive_version_binary_path("/opt/ls", " ") is None + + def test_is_derived_true(self): + path = derive_version_binary_path(None, "9.4.3") + assert is_derived_version_binary_path(path, None) is True + assert is_derived_version_binary_path(path + "/", None) is True + + def test_is_derived_false_for_system(self): + assert is_derived_version_binary_path(SYSTEM_BINARY_PATH, None) is False + assert is_derived_version_binary_path("/opt/custom/bin", None) is False + + +class TestResolveRunningLogstashVersion: + def test_prefers_column(self): + assert resolve_running_logstash_version( + logstash_version_resolved="9.4.3", + status_blob={"logstash_api": {"version": "8.0.0"}}, + ) == "9.4.3" + + def test_falls_back_to_api_version(self): + assert resolve_running_logstash_version( + logstash_version_resolved="", + status_blob={"logstash_api": {"version": "9.1.0"}}, + ) == "9.1.0" + + def test_falls_back_to_blob_keys(self): + assert resolve_running_logstash_version( + status_blob={"logstash_version": "8.15.0"} + ) == "8.15.0" + + def test_missing_returns_none(self): + assert resolve_running_logstash_version() is None + assert resolve_running_logstash_version(status_blob={}) is None + + +class TestAgentVersionRelation: + def test_older_equal_newer(self): + assert agent_version_relation("0.5.0", "0.5.1") == "older" + assert agent_version_relation("0.5.1", "0.5.1") == "equal" + assert agent_version_relation("0.5.2", "0.5.1") == "newer" + + def test_prerelease_newer_than_preferred(self): + assert agent_version_relation("0.5.2.dev0", "0.5.1") == "newer" + + def test_garbage_unknown(self): + assert agent_version_relation("not-a-version", "0.5.1") == "unknown" + assert agent_version_relation("", "0.5.1") == "unknown" + assert agent_version_relation(None, "0.5.1") == "unknown" + + +class TestResolvePersistedBinaryPath: + def test_version_autofills_system_default(self): + assert resolve_persisted_binary_path( + source="VERSION", + version="9.4.3", + download_dir="/opt/logstash-agent/logstash-versions", + binary_path="/usr/share/logstash/bin", + ) == "/opt/logstash-agent/logstash-versions/logstash-9.4.3/bin" + + def test_version_updates_previous_derived_path(self): + old = "/opt/logstash-agent/logstash-versions/logstash-9.4.3/bin" + assert resolve_persisted_binary_path( + source="VERSION", + version="9.5.0", + download_dir="/opt/logstash-agent/logstash-versions", + binary_path=old, + ) == "/opt/logstash-agent/logstash-versions/logstash-9.5.0/bin" + + def test_version_keeps_custom_path(self): + assert resolve_persisted_binary_path( + source="VERSION", + version="9.4.3", + download_dir="/opt/logstash-agent/logstash-versions", + binary_path="/opt/my/logstash/bin", + ) == "/opt/my/logstash/bin" + + def test_version_empty_pin_does_not_rewrite(self): + assert resolve_persisted_binary_path( + source="VERSION", + version="", + download_dir="/opt/logstash-agent/logstash-versions", + binary_path="/usr/share/logstash/bin", + ) == "/usr/share/logstash/bin" + + def test_system_restores_when_path_was_derived(self): + derived = "/opt/logstash-agent/logstash-versions/logstash-9.4.3/bin" + assert resolve_persisted_binary_path( + source="SYSTEM", + version="9.4.3", + download_dir="/opt/logstash-agent/logstash-versions", + binary_path=derived, + ) == SYSTEM_BINARY_PATH + + def test_system_keeps_custom_path(self): + assert resolve_persisted_binary_path( + source="SYSTEM", + version="9.4.3", + download_dir="/opt/logstash-agent/logstash-versions", + binary_path="/opt/my/bin", + ) == "/opt/my/bin" diff --git a/src/logstashui/PipelineManager/tests/test_manager_views.py b/src/logstashui/PipelineManager/tests/test_manager_views.py index 3553bf0..47ee18e 100644 --- a/src/logstashui/PipelineManager/tests/test_manager_views.py +++ b/src/logstashui/PipelineManager/tests/test_manager_views.py @@ -485,6 +485,63 @@ def test_context_has_connections(self, authenticated_client, test_connection): assert 'has_connections' in response.context assert response.context['has_connections'] is True + def test_agent_row_has_ls_pill_and_unreleased(self, authenticated_client, db): + from django.conf import settings + from PipelineManager.models import Connection, Policy + + policy = Policy.objects.create( + name='PM Policy', + settings_path='/etc/logstash/', + logs_path='/var/log/logstash', + binary_path='/usr/share/logstash/bin', + logstash_yml='http.host: "0.0.0.0"', + jvm_options='-Xms1g', + log4j2_properties='', + ) + Connection.objects.create( + name='Ahead Agent', + connection_type='AGENT', + host='agent.example.com', + agent_id='ahead-001', + is_active=True, + policy=policy, + logstash_version_resolved='9.4.3', + status_blob={'agent_version': '0.5.2', 'logstash_api': {'version': '9.4.3'}}, + ) + response = authenticated_client.get('/ConnectionManager/') + html = response.content.decode() + assert 'LS 9.4.3' in html + assert 'unreleased version' in html + assert 'upgradeAgent(' not in html + assert settings.__PREFERRED_LS_AGENT_VERSION__ == '0.5.1' + + def test_older_agent_shows_upgrade_not_unreleased(self, authenticated_client, db): + from PipelineManager.models import Connection, Policy + + policy = Policy.objects.create( + name='Old Policy', + settings_path='/etc/logstash/', + logs_path='/var/log/logstash', + binary_path='/usr/share/logstash/bin', + logstash_yml='', + jvm_options='', + log4j2_properties='', + ) + Connection.objects.create( + name='Old Agent', + connection_type='AGENT', + host='agent.example.com', + agent_id='old-001', + is_active=True, + policy=policy, + status_blob={'agent_version': '0.4.0'}, + ) + html = authenticated_client.get('/ConnectionManager/').content.decode() + assert 'upgradeAgent(' in html + assert 'unreleased version' not in html + after_name = html.split('Old Agent', 1)[1][:800] + assert 'LS ' not in after_name + # ============================================================================ # GetPipeline endpoint diff --git a/src/logstashui/PipelineManager/tests/test_policies_crud.py b/src/logstashui/PipelineManager/tests/test_policies_crud.py index 9019366..bf76488 100644 --- a/src/logstashui/PipelineManager/tests/test_policies_crud.py +++ b/src/logstashui/PipelineManager/tests/test_policies_crud.py @@ -368,6 +368,25 @@ def test_add_policy_rejects_unknown_type(self, authenticated_client): assert 'Invalid policy_type' in data['error'] assert not Policy.objects.filter(name='Nope Weird').exists() + def test_add_policy_version_pin_fills_binary_path(self, authenticated_client): + response = authenticated_client.post( + '/ConnectionManager/AddPolicy/', + data=json.dumps({ + 'name': 'Version Pin Policy', + 'policy_type': 'SIMULATE', + 'logstash_source': 'VERSION', + 'logstash_version': '9.4.3', + 'binary_path': '/usr/share/logstash/bin', + }), + content_type='application/json', + ) + assert response.status_code == 200 + assert response.json()['success'] is True + policy = Policy.objects.get(name='Version Pin Policy') + assert policy.binary_path == ( + '/opt/logstash-agent/logstash-versions/logstash-9.4.3/bin' + ) + # ============================================================================ # UpdatePolicy Tests @@ -528,6 +547,59 @@ def test_update_policy_invalid_json(self, authenticated_client): assert data['success'] is False assert 'Invalid JSON data' in data['error'] + def test_update_policy_version_pin_fills_binary_path(self, authenticated_client, test_policy): + response = authenticated_client.post( + '/ConnectionManager/UpdatePolicy/', + data=json.dumps({ + 'policy_name': 'Test Policy', + 'logstash_source': 'VERSION', + 'logstash_version': '9.4.3', + 'logstash_download_dir': '/opt/logstash-agent/logstash-versions', + 'binary_path': '/usr/share/logstash/bin', + }), + content_type='application/json', + ) + assert response.status_code == 200 + test_policy.refresh_from_db() + assert test_policy.binary_path == ( + '/opt/logstash-agent/logstash-versions/logstash-9.4.3/bin' + ) + + def test_update_policy_version_keeps_custom_binary_path(self, authenticated_client, test_policy): + response = authenticated_client.post( + '/ConnectionManager/UpdatePolicy/', + data=json.dumps({ + 'policy_name': 'Test Policy', + 'logstash_source': 'VERSION', + 'logstash_version': '9.4.3', + 'binary_path': '/opt/my/logstash/bin', + }), + content_type='application/json', + ) + assert response.status_code == 200 + test_policy.refresh_from_db() + assert test_policy.binary_path == '/opt/my/logstash/bin' + + def test_update_policy_system_restores_derived_binary_path(self, authenticated_client, test_policy): + test_policy.logstash_source = 'VERSION' + test_policy.logstash_version = '9.4.3' + test_policy.binary_path = '/opt/logstash-agent/logstash-versions/logstash-9.4.3/bin' + test_policy.save() + response = authenticated_client.post( + '/ConnectionManager/UpdatePolicy/', + data=json.dumps({ + 'policy_name': 'Test Policy', + 'logstash_source': 'SYSTEM', + 'logstash_version': '9.4.3', + 'logstash_download_dir': '/opt/logstash-agent/logstash-versions', + 'binary_path': '/opt/logstash-agent/logstash-versions/logstash-9.4.3/bin', + }), + content_type='application/json', + ) + assert response.status_code == 200 + test_policy.refresh_from_db() + assert test_policy.binary_path == '/usr/share/logstash/bin' + # ============================================================================ # DeletePolicy Tests From ebc3305b4a4ca954712283a73e464babcc79d54b Mon Sep 17 00:00:00 2001 From: Aaron Mildenstein Date: Thu, 3 Sep 2026 10:07:34 -0600 Subject: [PATCH 26/62] Improve testing... Migrated all database tests to `tests/integration` or `tests/unit` Now using `testcontainers[postgres]` and `testcontainers[mysql]` for integration tests. - Docker required. Tests will be skipped if docker is not present or not working Starting the migration of tests away from `src/logstashui/*/tests` --- pyproject.toml | 7 +- .../LogstashUI/tests/test_migrate_live.py | 165 -------- tests/__init__.py | 0 tests/integration/__init__.py | 0 tests/integration/conftest.py | 244 +++++++++++ tests/integration/test_db_config.py | 166 ++++++++ tests/integration/test_migrate_engine.py | 293 +++++++++++++ tests/integration/test_migrations.py | 101 +++++ tests/integration/test_orm.py | 389 ++++++++++++++++++ tests/unit/__init__.py | 0 .../tests => tests/unit}/test_database.py | 0 .../unit}/test_migrate_engine.py | 0 uv.lock | 267 ++++++++++-- 13 files changed, 1422 insertions(+), 210 deletions(-) delete mode 100644 src/logstashui/LogstashUI/tests/test_migrate_live.py create mode 100644 tests/__init__.py create mode 100644 tests/integration/__init__.py create mode 100644 tests/integration/conftest.py create mode 100644 tests/integration/test_db_config.py create mode 100644 tests/integration/test_migrate_engine.py create mode 100644 tests/integration/test_migrations.py create mode 100644 tests/integration/test_orm.py create mode 100644 tests/unit/__init__.py rename {src/logstashui/LogstashUI/tests => tests/unit}/test_database.py (100%) rename {src/logstashui/LogstashUI/tests => tests/unit}/test_migrate_engine.py (100%) diff --git a/pyproject.toml b/pyproject.toml index 683bb2a..c54754a 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -99,13 +99,18 @@ dev = [ "pytest>=9.0.2", "pytest-cov>=7.1.0", "pytest-django>=4.10.0", + "testcontainers[postgres]>=4.8.0", + "testcontainers[mysql]>=4.8.0", ] [tool.pytest.ini_options] DJANGO_SETTINGS_MODULE = "LogstashUI.settings" python_files = ["tests.py", "test_*.py", "*_tests.py"] pythonpath = ["src/logstashui"] -testpaths = ["src/logstashui"] +testpaths = ["src/logstashui", "tests"] +markers = [ + "integration: marks tests that require running Docker containers", +] addopts = [ "--import-mode=importlib", "--cov", diff --git a/src/logstashui/LogstashUI/tests/test_migrate_live.py b/src/logstashui/LogstashUI/tests/test_migrate_live.py deleted file mode 100644 index ed4ee62..0000000 --- a/src/logstashui/LogstashUI/tests/test_migrate_live.py +++ /dev/null @@ -1,165 +0,0 @@ -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. - -from argparse import Namespace -import json -import os -import subprocess -import sys - -import pytest - -from LogstashUI import migrate_engine as me - -pytestmark = pytest.mark.skipif( - os.environ.get("LOGSTASHUI_LIVE_DB") != "1", - reason="set LOGSTASHUI_LIVE_DB=1 (bin/test_databases.sh)", -) - -_SEED = """ -import os -import django -os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") -django.setup() -from django.contrib.auth import get_user_model -from PipelineManager.models import Connection, Policy -User = get_user_model() -User.objects.create_user(username="migrate-user", password="migrate-pass") -policy = Policy.objects.create( - name="Migrate Policy", - logstash_yml="http.host: 0.0.0.0", - jvm_options="-Xms1g", - log4j2_properties="status = error", -) -Connection.objects.create( - name="Migrate Conn", - connection_type=Connection.ConnectionType.AGENT, - host="127.0.0.1", - policy=policy, - status_blob={"health": "green", "n": 1}, -) -""" - -_COUNT = """ -import json -import os -import django -os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") -django.setup() -from django.contrib.auth import get_user_model -from PipelineManager.models import Connection, Policy -User = get_user_model() -conn = Connection.objects.filter(name="Migrate Conn").first() -print(json.dumps({ - "users": User.objects.count(), - "policies": Policy.objects.count(), - "migrate_user": User.objects.filter(username="migrate-user").count(), - "migrate_policy": Policy.objects.filter(name="Migrate Policy").count(), - "status_blob": conn.status_blob if conn else None, -})) -""" - -_TARGET_KEYS = ( - "LOGSTASHUI_DATA_DIR", - "LOGSTASHUI_DB_ENGINE", - "LOGSTASHUI_DB_HOST", - "LOGSTASHUI_DB_PORT", - "LOGSTASHUI_DB_USER", - "LOGSTASHUI_DB_PASSWORD", - "LOGSTASHUI_DB_NAME", -) - - -def _run_python(code: str, extra_env: dict[str, str]) -> str: - env = os.environ.copy() - env.update(extra_env) - env.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") - me._with_package_pythonpath(env) - proc = subprocess.run( - [sys.executable, "-c", code], - env=env, - check=False, - capture_output=True, - text=True, - ) - if proc.returncode != 0: - raise AssertionError( - f"python -c exited {proc.returncode}\n" - f"stdout:\n{proc.stdout}\nstderr:\n{proc.stderr}" - ) - return proc.stdout - - -def _run_to(tmp_path, engine: str, *, port: str, user: str) -> None: - data_dir = tmp_path - sqlite_path = data_dir / "db.sqlite3" - sqlite_env = { - "LOGSTASHUI_DATA_DIR": str(data_dir), - "LOGSTASHUI_DB_ENGINE": "sqlite", - "LOGSTASHUI_DB_NAME": str(sqlite_path), - } - me.run_manage(["migrate", "--noinput"], sqlite_env) - _run_python(_SEED, sqlite_env) - - target = { - "LOGSTASHUI_DATA_DIR": str(data_dir), - "LOGSTASHUI_DB_ENGINE": engine, - "LOGSTASHUI_DB_HOST": "127.0.0.1", - "LOGSTASHUI_DB_PORT": str(port), - "LOGSTASHUI_DB_USER": user, - "LOGSTASHUI_DB_PASSWORD": os.environ.get( - "LOGSTASHUI_LIVE_DB_PASSWORD", "logstashui" - ), - "LOGSTASHUI_DB_NAME": "logstashui_migrate", - } - previous = {key: os.environ.get(key) for key in _TARGET_KEYS} - try: - os.environ.update(target) - ns = Namespace(to=engine, i_have_a_backup=True, pid=None, write_env=None) - try: - rc = me.cmd_migrate_engine(ns) - except SystemExit as exc: - raise AssertionError(f"cmd_migrate_engine SystemExit {exc.code}") from exc - assert rc == 0 - raw = _run_python(_COUNT, target) - finally: - for key, value in previous.items(): - if value is None: - os.environ.pop(key, None) - else: - os.environ[key] = value - - counts = json.loads(raw.strip().splitlines()[-1]) - assert counts["users"] >= 1 - assert counts["policies"] >= 1 - assert counts["migrate_user"] == 1 - assert counts["migrate_policy"] == 1 - assert counts["status_blob"] == {"health": "green", "n": 1} - - -def test_live_postgres(tmp_path): - _run_to( - tmp_path, - "postgresql", - port=os.environ.get("LOGSTASHUI_LIVE_PG_PORT", "55432"), - user=os.environ.get("LOGSTASHUI_LIVE_PG_USER", "logstashui"), - ) - - -def test_live_mariadb(tmp_path): - _run_to( - tmp_path, - "mysql", - port=os.environ.get("LOGSTASHUI_LIVE_MARIA_PORT", "53306"), - user=os.environ.get("LOGSTASHUI_LIVE_DB_USER", "root"), - ) - - -def test_live_mysql(tmp_path): - _run_to( - tmp_path, - "mysql", - port=os.environ.get("LOGSTASHUI_LIVE_MYSQL_PORT", "53307"), - user=os.environ.get("LOGSTASHUI_LIVE_DB_USER", "root"), - ) diff --git a/tests/__init__.py b/tests/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/integration/__init__.py b/tests/integration/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/integration/conftest.py b/tests/integration/conftest.py new file mode 100644 index 0000000..1ae0172 --- /dev/null +++ b/tests/integration/conftest.py @@ -0,0 +1,244 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +import os +import shutil +import subprocess +import sys +import uuid + +import pytest + + +# --------------------------------------------------------------------------- +# Docker availability — checked once at module import time +# --------------------------------------------------------------------------- + +def _check_docker() -> tuple[bool, str]: + if not shutil.which("docker"): + return False, "docker binary not found in PATH" + try: + r = subprocess.run( + ["docker", "info"], + capture_output=True, + timeout=10, + ) + if r.returncode != 0: + return False, ( + f"docker info returned {r.returncode}: " + f"{r.stderr.decode()[:200]}" + ) + return True, "" + except (subprocess.TimeoutExpired, OSError) as exc: + return False, str(exc) + + +_DOCKER_OK, _DOCKER_REASON = _check_docker() + + +@pytest.fixture(scope="session", autouse=True) +def skip_if_no_docker(): + """Skip every test in the integration suite when Docker is unavailable.""" + if not _DOCKER_OK: + pytest.skip(f"Docker not available: {_DOCKER_REASON}") + + +# --------------------------------------------------------------------------- +# Container fixtures (session scope — start once, reused across all tests) +# --------------------------------------------------------------------------- + +@pytest.fixture(scope="session") +def postgres_container(): + from testcontainers.postgres import PostgresContainer + + with PostgresContainer( + image="postgres:16", + username="logstashui", + password="logstashui", + dbname="logstashui_test", + ) as c: + yield c + + +@pytest.fixture(scope="session") +def mysql_container(): + from testcontainers.mysql import MySqlContainer + + c = MySqlContainer( + image="mysql:8.0", + root_password="logstashui", + dbname="logstashui_test", + ) + c.with_command( + "--character-set-server=utf8mb4 --collation-server=utf8mb4_bin" + ) + with c: + yield c + + +@pytest.fixture(scope="session") +def mariadb_container(): + from testcontainers.mysql import MySqlContainer + + c = MySqlContainer( + image="mariadb:11", + root_password="logstashui", + dbname="logstashui_test", + ) + c.with_command( + "--character-set-server=utf8mb4 --collation-server=utf8mb4_bin" + ) + with c: + yield c + + +# --------------------------------------------------------------------------- +# Env-dict helpers (module-level, not fixtures — importable by test files) +# --------------------------------------------------------------------------- + +def pg_env(container, *, dbname: str = "logstashui_test") -> dict[str, str]: + """Return LOGSTASHUI_DB_* env dict for a PostgreSQL container.""" + return { + "LOGSTASHUI_DB_ENGINE": "postgresql", + "LOGSTASHUI_DB_HOST": container.get_container_host_ip(), + "LOGSTASHUI_DB_PORT": str(container.get_exposed_port(5432)), + "LOGSTASHUI_DB_USER": "logstashui", + "LOGSTASHUI_DB_PASSWORD": "logstashui", + "LOGSTASHUI_DB_NAME": dbname, + } + + +def mysql_env(container, *, dbname: str = "logstashui_test") -> dict[str, str]: + """Return LOGSTASHUI_DB_* env dict for a MySQL/MariaDB container (root user).""" + return { + "LOGSTASHUI_DB_ENGINE": "mysql", + "LOGSTASHUI_DB_HOST": container.get_container_host_ip(), + "LOGSTASHUI_DB_PORT": str(container.get_exposed_port(3306)), + "LOGSTASHUI_DB_USER": "root", + "LOGSTASHUI_DB_PASSWORD": "logstashui", + "LOGSTASHUI_DB_NAME": dbname, + } + + +def new_dbname() -> str: + """Generate a unique database name for test isolation.""" + return f"logstashui_{uuid.uuid4().hex[:8]}" + + +# --------------------------------------------------------------------------- +# Fresh-database helpers (create/drop via native drivers) +# --------------------------------------------------------------------------- + +def create_pg_db(base_env: dict[str, str], dbname: str) -> None: + """Create *dbname* in the PostgreSQL container reachable via *base_env*.""" + import psycopg + + connstr = ( + f"host={base_env['LOGSTASHUI_DB_HOST']} " + f"port={base_env['LOGSTASHUI_DB_PORT']} " + f"user={base_env['LOGSTASHUI_DB_USER']} " + f"password={base_env['LOGSTASHUI_DB_PASSWORD']} " + f"dbname={base_env['LOGSTASHUI_DB_NAME']}" + ) + with psycopg.connect(connstr, autocommit=True) as conn: + conn.execute(f'CREATE DATABASE "{dbname}"') + + +def drop_pg_db(base_env: dict[str, str], dbname: str) -> None: + import psycopg + + connstr = ( + f"host={base_env['LOGSTASHUI_DB_HOST']} " + f"port={base_env['LOGSTASHUI_DB_PORT']} " + f"user={base_env['LOGSTASHUI_DB_USER']} " + f"password={base_env['LOGSTASHUI_DB_PASSWORD']} " + f"dbname={base_env['LOGSTASHUI_DB_NAME']}" + ) + with psycopg.connect(connstr, autocommit=True) as conn: + conn.execute(f'DROP DATABASE IF EXISTS "{dbname}"') + + +def create_mysql_db(base_env: dict[str, str], dbname: str) -> None: + """Create *dbname* with utf8mb4/utf8mb4_bin in the MySQL/MariaDB container.""" + import pymysql + + conn = pymysql.connect( + host=base_env["LOGSTASHUI_DB_HOST"], + port=int(base_env["LOGSTASHUI_DB_PORT"]), + user=base_env["LOGSTASHUI_DB_USER"], + password=base_env["LOGSTASHUI_DB_PASSWORD"], + autocommit=True, + ) + try: + with conn.cursor() as cur: + cur.execute( + f"CREATE DATABASE `{dbname}` " + f"CHARACTER SET utf8mb4 COLLATE utf8mb4_bin" + ) + finally: + conn.close() + + +def drop_mysql_db(base_env: dict[str, str], dbname: str) -> None: + import pymysql + + conn = pymysql.connect( + host=base_env["LOGSTASHUI_DB_HOST"], + port=int(base_env["LOGSTASHUI_DB_PORT"]), + user=base_env["LOGSTASHUI_DB_USER"], + password=base_env["LOGSTASHUI_DB_PASSWORD"], + autocommit=True, + ) + try: + with conn.cursor() as cur: + cur.execute(f"DROP DATABASE IF EXISTS `{dbname}`") + finally: + conn.close() + + +# --------------------------------------------------------------------------- +# Parametrized engine fixture (postgres + mysql) +# --------------------------------------------------------------------------- + +@pytest.fixture(params=["postgres", "mysql"]) +def engine_env(request, postgres_container, mysql_container): + """Yield (engine_name, env_dict) for each supported engine.""" + if request.param == "postgres": + yield "postgresql", pg_env(postgres_container) + else: + yield "mysql", mysql_env(mysql_container) + + +# --------------------------------------------------------------------------- +# Fresh per-test database fixture (for migration / round-trip tests) +# --------------------------------------------------------------------------- + +@pytest.fixture +def fresh_db_env(engine_env, tmp_path): + """ + Yield (engine_name, env_dict) with a unique throwaway database and + LOGSTASHUI_DATA_DIR set. The database is created before the test and + dropped afterwards. + """ + engine, base_env = engine_env + dbname = new_dbname() + + if engine == "postgresql": + create_pg_db(base_env, dbname) + full_env = { + **base_env, + "LOGSTASHUI_DB_NAME": dbname, + "LOGSTASHUI_DATA_DIR": str(tmp_path), + } + yield engine, full_env + drop_pg_db(base_env, dbname) + else: + create_mysql_db(base_env, dbname) + full_env = { + **base_env, + "LOGSTASHUI_DB_NAME": dbname, + "LOGSTASHUI_DATA_DIR": str(tmp_path), + } + yield engine, full_env + drop_mysql_db(base_env, dbname) diff --git a/tests/integration/test_db_config.py b/tests/integration/test_db_config.py new file mode 100644 index 0000000..0aed525 --- /dev/null +++ b/tests/integration/test_db_config.py @@ -0,0 +1,166 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +""" +Integration tests for database configuration and server version checking. +All tests that open real connections use subprocesses so Django settings +are configured in an isolated interpreter with the container env vars. +""" + +import os +import subprocess +import sys + +import pytest + +from LogstashUI.database import build_databases + + +# --------------------------------------------------------------------------- +# Subprocess helper +# --------------------------------------------------------------------------- + +def _run_python(code: str, extra_env: dict[str, str]) -> str: + from LogstashUI import migrate_engine as me + + env = os.environ.copy() + env.update(extra_env) + env.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") + me._with_package_pythonpath(env) + proc = subprocess.run( + [sys.executable, "-c", code], + env=env, + check=False, + capture_output=True, + text=True, + ) + if proc.returncode != 0: + raise AssertionError( + f"python -c exited {proc.returncode}\n" + f"stdout:\n{proc.stdout}\nstderr:\n{proc.stderr}" + ) + return proc.stdout + + +# --------------------------------------------------------------------------- +# Inline scripts +# --------------------------------------------------------------------------- + +_CHECK_SERVER_VERSION = """ +import os +import django +os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") +django.setup() +from django.db import connection +from LogstashUI.database import check_server_version +connection.ensure_connection() +check_server_version(connection) +print("OK") +""" + +_ENSURE_CONNECTION = """ +import os +import django +os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") +django.setup() +from django.db import connection +connection.ensure_connection() +assert connection.connection is not None, "connection is None" +print("OK") +""" + + +# --------------------------------------------------------------------------- +# Tests — build_databases() dict structure (no Docker needed) +# --------------------------------------------------------------------------- + +def test_mysql_options_include_utf8mb4(monkeypatch, tmp_path): + """build_databases() for MySQL must include utf8mb4 charset and utf8mb4_bin collation.""" + for key in ( + "LOGSTASHUI_DB_ENGINE", "LOGSTASHUI_DB_HOST", "LOGSTASHUI_DB_PORT", + "LOGSTASHUI_DB_USER", "LOGSTASHUI_DB_PASSWORD", "LOGSTASHUI_DB_NAME", + "LOGSTASHUI_DB_CONN_MAX_AGE", + ): + monkeypatch.delenv(key, raising=False) + monkeypatch.setenv("LOGSTASHUI_DB_ENGINE", "mysql") + monkeypatch.setenv("LOGSTASHUI_DB_HOST", "127.0.0.1") + monkeypatch.setenv("LOGSTASHUI_DB_USER", "root") + monkeypatch.setattr("LogstashUI.database._import_or_raise", lambda *a, **k: _fake_pymysql()) + db = build_databases(tmp_path)["default"] + assert db["OPTIONS"]["charset"] == "utf8mb4" + assert "utf8mb4_bin" in db["OPTIONS"]["init_command"] + assert db["TEST"]["CHARSET"] == "utf8mb4" + assert db["TEST"]["COLLATION"] == "utf8mb4_bin" + + +def _fake_pymysql(): + from types import SimpleNamespace + fake = SimpleNamespace( + version_info=(1, 1, 1, "final", 0), + install_as_MySQLdb=lambda: None, + ) + return fake + + +def test_conn_max_age_applied(monkeypatch, tmp_path): + """LOGSTASHUI_DB_CONN_MAX_AGE overrides the default 60s for postgres.""" + for key in ( + "LOGSTASHUI_DB_ENGINE", "LOGSTASHUI_DB_HOST", "LOGSTASHUI_DB_PORT", + "LOGSTASHUI_DB_USER", "LOGSTASHUI_DB_PASSWORD", "LOGSTASHUI_DB_NAME", + "LOGSTASHUI_DB_CONN_MAX_AGE", + ): + monkeypatch.delenv(key, raising=False) + monkeypatch.setenv("LOGSTASHUI_DB_ENGINE", "postgresql") + monkeypatch.setenv("LOGSTASHUI_DB_HOST", "db.example") + monkeypatch.setenv("LOGSTASHUI_DB_USER", "lsui") + monkeypatch.setenv("LOGSTASHUI_DB_CONN_MAX_AGE", "120") + monkeypatch.setattr("LogstashUI.database._import_or_raise", lambda *a, **k: None) + db = build_databases(tmp_path)["default"] + assert db["CONN_MAX_AGE"] == 120 + + +def test_build_databases_returns_valid_dict(engine_env, tmp_path, monkeypatch): + """build_databases() produces a valid DATABASES dict for each engine.""" + engine, env = engine_env + for k, v in env.items(): + monkeypatch.setenv(k, v) + monkeypatch.setenv("LOGSTASHUI_DATA_DIR", str(tmp_path)) + # For MySQL, stub _import_or_raise so it runs without the spoof side-effect + if engine == "mysql": + monkeypatch.setattr( + "LogstashUI.database._import_or_raise", + lambda *a, **k: _fake_pymysql(), + ) + db = build_databases(tmp_path)["default"] + assert "ENGINE" in db + assert "HOST" in db + assert "PORT" in db + assert "NAME" in db + assert isinstance(db["PORT"], str) + + +# --------------------------------------------------------------------------- +# Tests — real container connections (subprocess) +# --------------------------------------------------------------------------- + +def test_real_connection_opens(engine_env, tmp_path): + """Django can open a connection to the container database.""" + engine, env = engine_env + full_env = {**env, "LOGSTASHUI_DATA_DIR": str(tmp_path)} + _run_python(_ENSURE_CONNECTION, full_env) + + +def test_check_server_version_passes_on_real_connection(engine_env, tmp_path): + """check_server_version() passes without error on a real container connection.""" + engine, env = engine_env + full_env = {**env, "LOGSTASHUI_DATA_DIR": str(tmp_path)} + _run_python(_CHECK_SERVER_VERSION, full_env) + + +def test_check_server_version_mariadb_branch(mariadb_container, tmp_path): + """check_server_version() MariaDB detection branch passes on a real MariaDB server.""" + from tests.integration.conftest import mysql_env + env = mysql_env(mariadb_container) + full_env = {**env, "LOGSTASHUI_DATA_DIR": str(tmp_path)} + _run_python(_CHECK_SERVER_VERSION, full_env) diff --git a/tests/integration/test_migrate_engine.py b/tests/integration/test_migrate_engine.py new file mode 100644 index 0000000..0c59a74 --- /dev/null +++ b/tests/integration/test_migrate_engine.py @@ -0,0 +1,293 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +""" +Integration tests for cmd_migrate_engine (SQLite → PostgreSQL / MySQL / MariaDB). + +Each test uses a unique throwaway database in the session-scoped container +to prevent cross-test contamination. The pattern follows test_migrate_live.py +but testcontainers replaces the external Docker Compose dependency. +""" + +import json +import os +import subprocess +import sys +from argparse import Namespace + +import pytest + +from LogstashUI import migrate_engine as me +from tests.integration.conftest import ( + create_mysql_db, + create_pg_db, + drop_mysql_db, + drop_pg_db, + mysql_env, + new_dbname, + pg_env, +) + + +# --------------------------------------------------------------------------- +# Subprocess helper (mirrors test_migrate_live._run_python) +# --------------------------------------------------------------------------- + +def _run_python(code: str, extra_env: dict[str, str]) -> str: + env = os.environ.copy() + env.update(extra_env) + env.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") + me._with_package_pythonpath(env) + proc = subprocess.run( + [sys.executable, "-c", code], + env=env, + check=False, + capture_output=True, + text=True, + ) + if proc.returncode != 0: + raise AssertionError( + f"python -c exited {proc.returncode}\n" + f"stdout:\n{proc.stdout}\nstderr:\n{proc.stderr}" + ) + return proc.stdout + + +# --------------------------------------------------------------------------- +# Inline scripts (seed and count — identical to test_migrate_live) +# --------------------------------------------------------------------------- + +_SEED = """ +import os +import django +os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") +django.setup() +from django.contrib.auth import get_user_model +from PipelineManager.models import Connection, Policy +User = get_user_model() +User.objects.create_user(username="migrate-user", password="migrate-pass") +policy = Policy.objects.create( + name="Migrate Policy", + logstash_yml="http.host: 0.0.0.0", + jvm_options="-Xms1g", + log4j2_properties="status = error", +) +Connection.objects.create( + name="Migrate Conn", + connection_type=Connection.ConnectionType.AGENT, + host="127.0.0.1", + policy=policy, + status_blob={"health": "green", "n": 1}, +) +""" + +_COUNT = """ +import json +import os +import django +os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") +django.setup() +from django.contrib.auth import get_user_model +from PipelineManager.models import Connection, Policy +User = get_user_model() +conn = Connection.objects.filter(name="Migrate Conn").first() +print(json.dumps({ + "users": User.objects.count(), + "policies": Policy.objects.count(), + "migrate_user": User.objects.filter(username="migrate-user").count(), + "migrate_policy": Policy.objects.filter(name="Migrate Policy").count(), + "status_blob": conn.status_blob if conn else None, +})) +""" + +_POST_MIGRATE_INSERT = """ +import json +import os +import django +os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") +django.setup() +from django.contrib.auth import get_user_model +User = get_user_model() +u = User.objects.create_user(username="post-migrate-user", password="x") +assert u.pk is not None +print(json.dumps({"pk": u.pk})) +""" + +_TARGET_KEYS = ( + "LOGSTASHUI_DATA_DIR", + "LOGSTASHUI_DB_ENGINE", + "LOGSTASHUI_DB_HOST", + "LOGSTASHUI_DB_PORT", + "LOGSTASHUI_DB_USER", + "LOGSTASHUI_DB_PASSWORD", + "LOGSTASHUI_DB_NAME", +) + + +# --------------------------------------------------------------------------- +# Core migration helper +# --------------------------------------------------------------------------- + +def _run_to(tmp_path, target_env: dict[str, str]) -> dict: + """ + Seed a fresh SQLite database, run cmd_migrate_engine to the target, + then assert data counts. Returns the parsed count dict. + """ + data_dir = tmp_path + sqlite_path = data_dir / "db.sqlite3" + sqlite_env = { + "LOGSTASHUI_DATA_DIR": str(data_dir), + "LOGSTASHUI_DB_ENGINE": "sqlite", + "LOGSTASHUI_DB_NAME": str(sqlite_path), + } + me.run_manage(["migrate", "--noinput", "--verbosity", "0"], sqlite_env) + _run_python(_SEED, sqlite_env) + + full_target = {**target_env, "LOGSTASHUI_DATA_DIR": str(data_dir)} + previous = {key: os.environ.get(key) for key in _TARGET_KEYS} + try: + os.environ.update(full_target) + ns = Namespace( + to=full_target["LOGSTASHUI_DB_ENGINE"], + i_have_a_backup=True, + pid=None, + write_env=None, + ) + try: + rc = me.cmd_migrate_engine(ns) + except SystemExit as exc: + raise AssertionError( + f"cmd_migrate_engine SystemExit {exc.code}" + ) from exc + assert rc == 0 + raw = _run_python(_COUNT, full_target) + finally: + for key, value in previous.items(): + if value is None: + os.environ.pop(key, None) + else: + os.environ[key] = value + + counts = json.loads(raw.strip().splitlines()[-1]) + assert counts["users"] >= 1 + assert counts["policies"] >= 1 + assert counts["migrate_user"] == 1 + assert counts["migrate_policy"] == 1 + assert counts["status_blob"] == {"health": "green", "n": 1} + return counts + + +# --------------------------------------------------------------------------- +# Tests +# --------------------------------------------------------------------------- + +def test_migrate_engine_to_postgres(postgres_container, tmp_path): + dbname = new_dbname() + base = pg_env(postgres_container) + create_pg_db(base, dbname) + try: + target = pg_env(postgres_container, dbname=dbname) + _run_to(tmp_path, target) + finally: + drop_pg_db(base, dbname) + + +def test_migrate_engine_to_mysql(mysql_container, tmp_path): + dbname = new_dbname() + base = mysql_env(mysql_container) + create_mysql_db(base, dbname) + try: + target = mysql_env(mysql_container, dbname=dbname) + _run_to(tmp_path, target) + finally: + drop_mysql_db(base, dbname) + + +def test_migrate_engine_to_mariadb(mariadb_container, tmp_path): + dbname = new_dbname() + base = mysql_env(mariadb_container) + create_mysql_db(base, dbname) + try: + target = mysql_env(mariadb_container, dbname=dbname) + _run_to(tmp_path, target) + finally: + drop_mysql_db(base, dbname) + + +def test_sequence_reset_postgres(postgres_container, tmp_path): + """After SQLite→Postgres migration, inserting a new User does not fail on sequence.""" + dbname = new_dbname() + base = pg_env(postgres_container) + create_pg_db(base, dbname) + try: + target = pg_env(postgres_container, dbname=dbname) + _run_to(tmp_path, target) + full_env = {**target, "LOGSTASHUI_DATA_DIR": str(tmp_path)} + out = _run_python(_POST_MIGRATE_INSERT, full_env) + result = json.loads(out.strip()) + assert isinstance(result["pk"], int) and result["pk"] > 0 + finally: + drop_pg_db(base, dbname) + + +def test_migrate_engine_write_env(postgres_container, tmp_path): + """--write-env produces a file with engine/host keys but no PASSWORD.""" + dbname = new_dbname() + base = pg_env(postgres_container) + create_pg_db(base, dbname) + env_file = tmp_path / "logstashui.env" + try: + target = pg_env(postgres_container, dbname=dbname) + full_target = {**target, "LOGSTASHUI_DATA_DIR": str(tmp_path)} + sqlite_path = tmp_path / "db.sqlite3" + sqlite_env = { + "LOGSTASHUI_DATA_DIR": str(tmp_path), + "LOGSTASHUI_DB_ENGINE": "sqlite", + "LOGSTASHUI_DB_NAME": str(sqlite_path), + } + me.run_manage(["migrate", "--noinput", "--verbosity", "0"], sqlite_env) + _run_python(_SEED, sqlite_env) + previous = {key: os.environ.get(key) for key in _TARGET_KEYS} + try: + os.environ.update(full_target) + ns = Namespace( + to="postgresql", + i_have_a_backup=True, + pid=None, + write_env=str(env_file), + ) + rc = me.cmd_migrate_engine(ns) + finally: + for key, value in previous.items(): + if value is None: + os.environ.pop(key, None) + else: + os.environ[key] = value + assert rc == 0 + text = env_file.read_text() + assert "LOGSTASHUI_DB_ENGINE=postgresql" in text + assert "LOGSTASHUI_DB_HOST=" in text + assert "PASSWORD" not in text + finally: + drop_pg_db(base, dbname) + + +def test_migrate_engine_idempotent_env(postgres_container, tmp_path): + """Running write_env twice produces no duplicate keys in the output file.""" + dbname = new_dbname() + base = pg_env(postgres_container) + create_pg_db(base, dbname) + env_file = tmp_path / "logstashui.env" + try: + target = pg_env(postgres_container, dbname=dbname) + full_target = {**target, "LOGSTASHUI_DATA_DIR": str(tmp_path)} + # Run migration once + _run_to(tmp_path, target) + # write_env twice, pointing at the already-migrated DB + me.write_env_file(env_file, "postgresql") + me.write_env_file(env_file, "postgresql") + text = env_file.read_text() + assert text.count("LOGSTASHUI_DB_ENGINE=postgresql") == 1 + finally: + drop_pg_db(base, dbname) diff --git a/tests/integration/test_migrations.py b/tests/integration/test_migrations.py new file mode 100644 index 0000000..b20b524 --- /dev/null +++ b/tests/integration/test_migrations.py @@ -0,0 +1,101 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +""" +Integration tests for Django migrations against real database containers. +Each parametrized test gets a fresh isolated database (created and dropped +per-test via the fresh_db_env fixture). +""" + +import os +import subprocess +import sys + +import pytest + +from LogstashUI import migrate_engine as me + + +# --------------------------------------------------------------------------- +# Subprocess helper +# --------------------------------------------------------------------------- + +def _run_python(code: str, extra_env: dict[str, str]) -> str: + env = os.environ.copy() + env.update(extra_env) + env.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") + me._with_package_pythonpath(env) + proc = subprocess.run( + [sys.executable, "-c", code], + env=env, + check=False, + capture_output=True, + text=True, + ) + if proc.returncode != 0: + raise AssertionError( + f"python -c exited {proc.returncode}\n" + f"stdout:\n{proc.stdout}\nstderr:\n{proc.stderr}" + ) + return proc.stdout + + +# --------------------------------------------------------------------------- +# Inline scripts +# --------------------------------------------------------------------------- + +_NO_UNAPPLIED = """ +import os +import django +os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") +django.setup() +from django.db import connection +from django.db.migrations.executor import MigrationExecutor +from django.db.migrations.loader import MigrationLoader +loader = MigrationLoader(connection) +executor = MigrationExecutor(connection) +plan = executor.migration_plan(loader.graph.leaf_nodes()) +assert plan == [], f"Unapplied migrations: {[str(m) for m, _ in plan]}" +print("OK") +""" + + +# --------------------------------------------------------------------------- +# Tests — container migrations +# --------------------------------------------------------------------------- + +def test_migrate_runs_clean(fresh_db_env): + """migrate --noinput completes without error on a fresh container database.""" + engine, env = fresh_db_env + me.run_manage(["migrate", "--noinput", "--verbosity", "0"], env) + + +def test_migrate_is_idempotent(fresh_db_env): + """Running migrate twice is safe (no errors, no unexpected state).""" + engine, env = fresh_db_env + me.run_manage(["migrate", "--noinput", "--verbosity", "0"], env) + me.run_manage(["migrate", "--noinput", "--verbosity", "0"], env) + + +def test_no_unapplied_migrations(fresh_db_env): + """After migrate, MigrationExecutor reports an empty plan.""" + engine, env = fresh_db_env + me.run_manage(["migrate", "--noinput", "--verbosity", "0"], env) + _run_python(_NO_UNAPPLIED, env) + + +# --------------------------------------------------------------------------- +# SQLite baseline (no Docker needed — fast sanity check) +# --------------------------------------------------------------------------- + +def test_sqlite_migrate_baseline(tmp_path): + """migrate --noinput works against SQLite; ensures the test runner itself is healthy.""" + sqlite_path = tmp_path / "db.sqlite3" + env = { + "LOGSTASHUI_DATA_DIR": str(tmp_path), + "LOGSTASHUI_DB_ENGINE": "sqlite", + "LOGSTASHUI_DB_NAME": str(sqlite_path), + } + me.run_manage(["migrate", "--noinput", "--verbosity", "0"], env) + assert sqlite_path.exists() diff --git a/tests/integration/test_orm.py b/tests/integration/test_orm.py new file mode 100644 index 0000000..5aef6ab --- /dev/null +++ b/tests/integration/test_orm.py @@ -0,0 +1,389 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +""" +ORM integration tests against real database containers. + +All Django interaction happens in subprocesses so each test gets a clean +interpreter with the container env applied to settings. Each test migrates +(idempotently) then runs its CRUD script which self-cleans at the end. +""" + +import json +import os +import subprocess +import sys + +import pytest + +from LogstashUI import migrate_engine as me + + +# --------------------------------------------------------------------------- +# Subprocess helper +# --------------------------------------------------------------------------- + +def _run_python(code: str, extra_env: dict[str, str]) -> str: + env = os.environ.copy() + env.update(extra_env) + env.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") + me._with_package_pythonpath(env) + proc = subprocess.run( + [sys.executable, "-c", code], + env=env, + check=False, + capture_output=True, + text=True, + ) + if proc.returncode != 0: + raise AssertionError( + f"python -c exited {proc.returncode}\n" + f"stdout:\n{proc.stdout}\nstderr:\n{proc.stderr}" + ) + return proc.stdout + + +def _migrate(env: dict[str, str]) -> None: + me.run_manage(["migrate", "--noinput", "--verbosity", "0"], env) + + +# --------------------------------------------------------------------------- +# Inline CRUD scripts (each cleans up its own data) +# --------------------------------------------------------------------------- + +_POLICY_CRUD = """ +import json, os, django +os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") +django.setup() +from PipelineManager.models import Policy +TAG = "crud-policy-test" +Policy.objects.filter(name=TAG).delete() +p = Policy.objects.create( + name=TAG, + logstash_yml="http.host: 0.0.0.0", + jvm_options="-Xms512m", + log4j2_properties="status = error", +) +pk = p.pk +assert Policy.objects.get(pk=pk).name == TAG +Policy.objects.filter(pk=pk).update(logstash_yml="http.host: 127.0.0.1") +assert Policy.objects.get(pk=pk).logstash_yml == "http.host: 127.0.0.1" +Policy.objects.filter(pk=pk).delete() +assert Policy.objects.filter(pk=pk).count() == 0 +print(json.dumps({"ok": True})) +""" + +_CONNECTION_CRUD = """ +import json, os, django +os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") +django.setup() +from PipelineManager.models import Connection, Policy +Policy.objects.filter(name="crud-conn-policy").delete() +policy = Policy.objects.create( + name="crud-conn-policy", + logstash_yml="http.host: 0.0.0.0", + jvm_options="-Xms512m", + log4j2_properties="status = error", +) +Connection.objects.filter(name="crud-conn-test").delete() +c = Connection.objects.create( + name="crud-conn-test", + connection_type=Connection.ConnectionType.AGENT, + host="127.0.0.1", + policy=policy, + status_blob={"health": "green"}, +) +pk = c.pk +assert Connection.objects.get(pk=pk).name == "crud-conn-test" +Connection.objects.filter(pk=pk).update(status_blob={"health": "yellow"}) +assert Connection.objects.get(pk=pk).status_blob["health"] == "yellow" +Connection.objects.filter(pk=pk).delete() +policy.delete() +assert Connection.objects.filter(pk=pk).count() == 0 +print(json.dumps({"ok": True})) +""" + +_PIPELINE_CRUD = """ +import json, os, django +os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") +django.setup() +from PipelineManager.models import Pipeline, Policy +Policy.objects.filter(name="crud-pipe-policy").delete() +policy = Policy.objects.create( + name="crud-pipe-policy", + logstash_yml="http.host: 0.0.0.0", + jvm_options="-Xms512m", + log4j2_properties="status = error", +) +Pipeline.objects.filter(policy=policy, name="test-pipeline").delete() +p = Pipeline.objects.create( + policy=policy, + name="test-pipeline", + lscl="input { stdin{} } output { stdout{} }", +) +pk = p.pk +assert Pipeline.objects.get(pk=pk).name == "test-pipeline" +Pipeline.objects.filter(pk=pk).delete() +policy.delete() +assert Pipeline.objects.filter(pk=pk).count() == 0 +print(json.dumps({"ok": True})) +""" + +_REVISION_JSON = """ +import json, os, django +os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") +django.setup() +from PipelineManager.models import Policy, Revision +Policy.objects.filter(name="rev-json-policy").delete() +policy = Policy.objects.create( + name="rev-json-policy", + logstash_yml="http.host: 0.0.0.0", + jvm_options="-Xms512m", + log4j2_properties="status = error", +) +snapshot = { + "pipelines": [{"name": "main", "lscl": "input{} output{}"}], + "meta": {"tags": ["a", "b"], "nested": {"k": 1}}, +} +r = Revision.objects.create( + policy=policy, revision_number=1, snapshot_json=snapshot, created_by="testrunner" +) +pk = r.pk +fetched = Revision.objects.get(pk=pk) +assert fetched.snapshot_json == snapshot, f"mismatch: {fetched.snapshot_json!r}" +Revision.objects.filter(pk=pk).delete() +policy.delete() +print(json.dumps({"ok": True})) +""" + +_STATUS_BLOB_JSON = """ +import json, os, django +os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") +django.setup() +from PipelineManager.models import Connection, Policy +Policy.objects.filter(name="blob-policy").delete() +policy = Policy.objects.create( + name="blob-policy", + logstash_yml="http.host: 0.0.0.0", + jvm_options="-Xms512m", + log4j2_properties="status = error", +) +blob = {"health": "green", "n": 42, "nested": {"k": [1, 2, 3]}} +Connection.objects.filter(name="blob-conn").delete() +c = Connection.objects.create( + name="blob-conn", + connection_type=Connection.ConnectionType.AGENT, + host="127.0.0.1", + policy=policy, + status_blob=blob, +) +pk = c.pk +fetched = Connection.objects.get(pk=pk) +assert fetched.status_blob == blob, f"mismatch: {fetched.status_blob!r}" +Connection.objects.filter(pk=pk).delete() +policy.delete() +print(json.dumps({"ok": True})) +""" + +_STATUS_BLOB_NULL = """ +import json, os, django +os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") +django.setup() +from PipelineManager.models import Connection, Policy +Policy.objects.filter(name="null-blob-policy").delete() +policy = Policy.objects.create( + name="null-blob-policy", + logstash_yml="http.host: 0.0.0.0", + jvm_options="-Xms512m", + log4j2_properties="status = error", +) +Connection.objects.filter(name="null-blob-conn").delete() +c = Connection.objects.create( + name="null-blob-conn", + connection_type=Connection.ConnectionType.AGENT, + host="127.0.0.1", + policy=policy, + status_blob=None, +) +pk = c.pk +assert Connection.objects.get(pk=pk).status_blob is None +Connection.objects.filter(pk=pk).delete() +policy.delete() +print(json.dumps({"ok": True})) +""" + +_SNMP_NETWORK_CRUD = """ +import json, os, django +os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") +django.setup() +from SNMP.models import Network +Network.objects.filter(name="crud-network-test").delete() +n = Network.objects.create(name="crud-network-test", network_range="10.0.0.0/24") +pk = n.pk +assert Network.objects.get(pk=pk).name == "crud-network-test" +Network.objects.filter(pk=pk).delete() +assert Network.objects.filter(pk=pk).count() == 0 +print(json.dumps({"ok": True})) +""" + +_UNIQUE_POLICY_NAME = """ +import json, os, django +os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") +django.setup() +from django.db import IntegrityError +from PipelineManager.models import Policy +Policy.objects.filter(name="dupe-policy").delete() +p1 = Policy.objects.create( + name="dupe-policy", + logstash_yml="http.host: 0.0.0.0", + jvm_options="-Xms512m", + log4j2_properties="status = error", +) +try: + Policy.objects.create( + name="dupe-policy", + logstash_yml="different", + jvm_options="-Xmx512m", + log4j2_properties="status = warn", + ) + raise AssertionError("Expected IntegrityError for duplicate Policy.name") +except IntegrityError: + pass +finally: + Policy.objects.filter(name="dupe-policy").delete() +print(json.dumps({"ok": True})) +""" + +_UNIQUE_PIPELINE_PER_POLICY = """ +import json, os, django +os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") +django.setup() +from django.db import IntegrityError +from PipelineManager.models import Pipeline, Policy +Policy.objects.filter(name__in=["uq-pol-a", "uq-pol-b"]).delete() +pol_a = Policy.objects.create( + name="uq-pol-a", + logstash_yml="http.host: 0.0.0.0", + jvm_options="-Xms512m", + log4j2_properties="status = error", +) +pol_b = Policy.objects.create( + name="uq-pol-b", + logstash_yml="http.host: 0.0.0.0", + jvm_options="-Xms512m", + log4j2_properties="status = error", +) +# Same name under different policies is allowed +Pipeline.objects.create(policy=pol_a, name="shared-pipe", lscl="input{} output{}") +Pipeline.objects.create(policy=pol_b, name="shared-pipe", lscl="input{} output{}") +# Same name under same policy must raise +try: + Pipeline.objects.create(policy=pol_a, name="shared-pipe", lscl="input{} output{}") + raise AssertionError("Expected IntegrityError for duplicate pipeline name per policy") +except IntegrityError: + pass +pol_a.delete() +pol_b.delete() +print(json.dumps({"ok": True})) +""" + +# Validates utf8mb4_bin on MySQL and native case-sensitivity on PostgreSQL. +# "CaseNet" and "casenet" must be distinct; second "CaseNet" must fail. +_CASE_SENSITIVE_UNIQUE = """ +import json, os, django +os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") +django.setup() +from django.db import IntegrityError +from SNMP.models import Network +Network.objects.filter(name__in=["CaseNet", "casenet"]).delete() +Network.objects.create(name="CaseNet", network_range="10.1.0.0/24") +# Different case must succeed +Network.objects.create(name="casenet", network_range="10.2.0.0/24") +# Exact duplicate must fail +try: + Network.objects.create(name="CaseNet", network_range="10.3.0.0/24") + raise AssertionError("Expected IntegrityError for duplicate Network.name") +except IntegrityError: + pass +finally: + Network.objects.filter(name__in=["CaseNet", "casenet"]).delete() +print(json.dumps({"ok": True})) +""" + + +# --------------------------------------------------------------------------- +# Tests (all parametrized over postgres + mysql via engine_env) +# --------------------------------------------------------------------------- + +def test_policy_crud(engine_env, tmp_path): + engine, env = engine_env + full_env = {**env, "LOGSTASHUI_DATA_DIR": str(tmp_path)} + _migrate(full_env) + assert json.loads(_run_python(_POLICY_CRUD, full_env).strip())["ok"] is True + + +def test_connection_crud(engine_env, tmp_path): + engine, env = engine_env + full_env = {**env, "LOGSTASHUI_DATA_DIR": str(tmp_path)} + _migrate(full_env) + assert json.loads(_run_python(_CONNECTION_CRUD, full_env).strip())["ok"] is True + + +def test_pipeline_crud(engine_env, tmp_path): + engine, env = engine_env + full_env = {**env, "LOGSTASHUI_DATA_DIR": str(tmp_path)} + _migrate(full_env) + assert json.loads(_run_python(_PIPELINE_CRUD, full_env).strip())["ok"] is True + + +def test_revision_json_roundtrip(engine_env, tmp_path): + engine, env = engine_env + full_env = {**env, "LOGSTASHUI_DATA_DIR": str(tmp_path)} + _migrate(full_env) + assert json.loads(_run_python(_REVISION_JSON, full_env).strip())["ok"] is True + + +def test_status_blob_roundtrip(engine_env, tmp_path): + engine, env = engine_env + full_env = {**env, "LOGSTASHUI_DATA_DIR": str(tmp_path)} + _migrate(full_env) + assert json.loads(_run_python(_STATUS_BLOB_JSON, full_env).strip())["ok"] is True + + +def test_status_blob_null_roundtrip(engine_env, tmp_path): + engine, env = engine_env + full_env = {**env, "LOGSTASHUI_DATA_DIR": str(tmp_path)} + _migrate(full_env) + assert json.loads(_run_python(_STATUS_BLOB_NULL, full_env).strip())["ok"] is True + + +def test_snmp_network_crud(engine_env, tmp_path): + engine, env = engine_env + full_env = {**env, "LOGSTASHUI_DATA_DIR": str(tmp_path)} + _migrate(full_env) + assert json.loads(_run_python(_SNMP_NETWORK_CRUD, full_env).strip())["ok"] is True + + +def test_unique_policy_name(engine_env, tmp_path): + engine, env = engine_env + full_env = {**env, "LOGSTASHUI_DATA_DIR": str(tmp_path)} + _migrate(full_env) + assert json.loads(_run_python(_UNIQUE_POLICY_NAME, full_env).strip())["ok"] is True + + +def test_unique_pipeline_per_policy(engine_env, tmp_path): + engine, env = engine_env + full_env = {**env, "LOGSTASHUI_DATA_DIR": str(tmp_path)} + _migrate(full_env) + assert json.loads(_run_python(_UNIQUE_PIPELINE_PER_POLICY, full_env).strip())["ok"] is True + + +def test_case_sensitive_unique(engine_env, tmp_path): + """Both engines treat unique names as case-sensitive. + On MySQL this validates utf8mb4_bin is active; on PostgreSQL it's the default. + """ + engine, env = engine_env + full_env = {**env, "LOGSTASHUI_DATA_DIR": str(tmp_path)} + _migrate(full_env) + assert json.loads(_run_python(_CASE_SENSITIVE_UNIQUE, full_env).strip())["ok"] is True diff --git a/tests/unit/__init__.py b/tests/unit/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/src/logstashui/LogstashUI/tests/test_database.py b/tests/unit/test_database.py similarity index 100% rename from src/logstashui/LogstashUI/tests/test_database.py rename to tests/unit/test_database.py diff --git a/src/logstashui/LogstashUI/tests/test_migrate_engine.py b/tests/unit/test_migrate_engine.py similarity index 100% rename from src/logstashui/LogstashUI/tests/test_migrate_engine.py rename to tests/unit/test_migrate_engine.py diff --git a/uv.lock b/uv.lock index a57289b..9feeb07 100644 --- a/uv.lock +++ b/uv.lock @@ -267,55 +267,52 @@ wheels = [ [[package]] name = "cryptography" -version = "46.0.6" +version = "50.0.1" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "cffi", marker = "platform_python_implementation != 'PyPy'" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/a4/ba/04b1bd4218cbc58dc90ce967106d51582371b898690f3ae0402876cc4f34/cryptography-46.0.6.tar.gz", hash = "sha256:27550628a518c5c6c903d84f637fbecf287f6cb9ced3804838a1295dc1fd0759", size = 750542, upload-time = "2026-03-25T23:34:53.396Z" } +sdist = { url = "https://files.pythonhosted.org/packages/bb/ad/5d6702db60b1e40b41ef513b6967ff5848f307d50f8449baf1634f5908f1/cryptography-50.0.1.tar.gz", hash = "sha256:5dd9bda1c12b4162f6ff568eeb5e0ff956c28d14406e875cfe8a63a2d414ff20", size = 880381, upload-time = "2026-08-25T19:45:45.499Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/47/23/9285e15e3bc57325b0a72e592921983a701efc1ee8f91c06c5f0235d86d9/cryptography-46.0.6-cp311-abi3-macosx_10_9_universal2.whl", hash = "sha256:64235194bad039a10bb6d2d930ab3323baaec67e2ce36215fd0952fad0930ca8", size = 7176401, upload-time = "2026-03-25T23:33:22.096Z" }, - { url = "https://files.pythonhosted.org/packages/60/f8/e61f8f13950ab6195b31913b42d39f0f9afc7d93f76710f299b5ec286ae6/cryptography-46.0.6-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:26031f1e5ca62fcb9d1fcb34b2b60b390d1aacaa15dc8b895a9ed00968b97b30", size = 4275275, upload-time = "2026-03-25T23:33:23.844Z" }, - { url = "https://files.pythonhosted.org/packages/19/69/732a736d12c2631e140be2348b4ad3d226302df63ef64d30dfdb8db7ad1c/cryptography-46.0.6-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:9a693028b9cbe51b5a1136232ee8f2bc242e4e19d456ded3fa7c86e43c713b4a", size = 4425320, upload-time = "2026-03-25T23:33:25.703Z" }, - { url = "https://files.pythonhosted.org/packages/d4/12/123be7292674abf76b21ac1fc0e1af50661f0e5b8f0ec8285faac18eb99e/cryptography-46.0.6-cp311-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:67177e8a9f421aa2d3a170c3e56eca4e0128883cf52a071a7cbf53297f18b175", size = 4278082, upload-time = "2026-03-25T23:33:27.423Z" }, - { url = "https://files.pythonhosted.org/packages/5b/ba/d5e27f8d68c24951b0a484924a84c7cdaed7502bac9f18601cd357f8b1d2/cryptography-46.0.6-cp311-abi3-manylinux_2_28_ppc64le.whl", hash = "sha256:d9528b535a6c4f8ff37847144b8986a9a143585f0540fbcb1a98115b543aa463", size = 4926514, upload-time = "2026-03-25T23:33:29.206Z" }, - { url = "https://files.pythonhosted.org/packages/34/71/1ea5a7352ae516d5512d17babe7e1b87d9db5150b21f794b1377eac1edc0/cryptography-46.0.6-cp311-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:22259338084d6ae497a19bae5d4c66b7ca1387d3264d1c2c0e72d9e9b6a77b97", size = 4457766, upload-time = "2026-03-25T23:33:30.834Z" }, - { url = "https://files.pythonhosted.org/packages/01/59/562be1e653accee4fdad92c7a2e88fced26b3fdfce144047519bbebc299e/cryptography-46.0.6-cp311-abi3-manylinux_2_31_armv7l.whl", hash = "sha256:760997a4b950ff00d418398ad73fbc91aa2894b5c1db7ccb45b4f68b42a63b3c", size = 3986535, upload-time = "2026-03-25T23:33:33.02Z" }, - { url = "https://files.pythonhosted.org/packages/d6/8b/b1ebfeb788bf4624d36e45ed2662b8bd43a05ff62157093c1539c1288a18/cryptography-46.0.6-cp311-abi3-manylinux_2_34_aarch64.whl", hash = "sha256:3dfa6567f2e9e4c5dceb8ccb5a708158a2a871052fa75c8b78cb0977063f1507", size = 4277618, upload-time = "2026-03-25T23:33:34.567Z" }, - { url = "https://files.pythonhosted.org/packages/dd/52/a005f8eabdb28df57c20f84c44d397a755782d6ff6d455f05baa2785bd91/cryptography-46.0.6-cp311-abi3-manylinux_2_34_ppc64le.whl", hash = "sha256:cdcd3edcbc5d55757e5f5f3d330dd00007ae463a7e7aa5bf132d1f22a4b62b19", size = 4890802, upload-time = "2026-03-25T23:33:37.034Z" }, - { url = "https://files.pythonhosted.org/packages/ec/4d/8e7d7245c79c617d08724e2efa397737715ca0ec830ecb3c91e547302555/cryptography-46.0.6-cp311-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:d4e4aadb7fc1f88687f47ca20bb7227981b03afaae69287029da08096853b738", size = 4457425, upload-time = "2026-03-25T23:33:38.904Z" }, - { url = "https://files.pythonhosted.org/packages/1d/5c/f6c3596a1430cec6f949085f0e1a970638d76f81c3ea56d93d564d04c340/cryptography-46.0.6-cp311-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:2b417edbe8877cda9022dde3a008e2deb50be9c407eef034aeeb3a8b11d9db3c", size = 4405530, upload-time = "2026-03-25T23:33:40.842Z" }, - { url = "https://files.pythonhosted.org/packages/7e/c9/9f9cea13ee2dbde070424e0c4f621c091a91ffcc504ffea5e74f0e1daeff/cryptography-46.0.6-cp311-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:380343e0653b1c9d7e1f55b52aaa2dbb2fdf2730088d48c43ca1c7c0abb7cc2f", size = 4667896, upload-time = "2026-03-25T23:33:42.781Z" }, - { url = "https://files.pythonhosted.org/packages/ad/b5/1895bc0821226f129bc74d00eccfc6a5969e2028f8617c09790bf89c185e/cryptography-46.0.6-cp311-abi3-win32.whl", hash = "sha256:bcb87663e1f7b075e48c3be3ecb5f0b46c8fc50b50a97cf264e7f60242dca3f2", size = 3026348, upload-time = "2026-03-25T23:33:45.021Z" }, - { url = "https://files.pythonhosted.org/packages/c3/f8/c9bcbf0d3e6ad288b9d9aa0b1dee04b063d19e8c4f871855a03ab3a297ab/cryptography-46.0.6-cp311-abi3-win_amd64.whl", hash = "sha256:6739d56300662c468fddb0e5e291f9b4d084bead381667b9e654c7dd81705124", size = 3483896, upload-time = "2026-03-25T23:33:46.649Z" }, - { url = "https://files.pythonhosted.org/packages/01/41/3a578f7fd5c70611c0aacba52cd13cb364a5dee895a5c1d467208a9380b0/cryptography-46.0.6-cp314-cp314t-macosx_10_9_universal2.whl", hash = "sha256:2ef9e69886cbb137c2aef9772c2e7138dc581fad4fcbcf13cc181eb5a3ab6275", size = 7117147, upload-time = "2026-03-25T23:33:48.249Z" }, - { url = "https://files.pythonhosted.org/packages/fa/87/887f35a6fca9dde90cad08e0de0c89263a8e59b2d2ff904fd9fcd8025b6f/cryptography-46.0.6-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:7f417f034f91dcec1cb6c5c35b07cdbb2ef262557f701b4ecd803ee8cefed4f4", size = 4266221, upload-time = "2026-03-25T23:33:49.874Z" }, - { url = "https://files.pythonhosted.org/packages/aa/a8/0a90c4f0b0871e0e3d1ed126aed101328a8a57fd9fd17f00fb67e82a51ca/cryptography-46.0.6-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:d24c13369e856b94892a89ddf70b332e0b70ad4a5c43cf3e9cb71d6d7ffa1f7b", size = 4408952, upload-time = "2026-03-25T23:33:52.128Z" }, - { url = "https://files.pythonhosted.org/packages/16/0b/b239701eb946523e4e9f329336e4ff32b1247e109cbab32d1a7b61da8ed7/cryptography-46.0.6-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:aad75154a7ac9039936d50cf431719a2f8d4ed3d3c277ac03f3339ded1a5e707", size = 4270141, upload-time = "2026-03-25T23:33:54.11Z" }, - { url = "https://files.pythonhosted.org/packages/0f/a8/976acdd4f0f30df7b25605f4b9d3d89295351665c2091d18224f7ad5cdbf/cryptography-46.0.6-cp314-cp314t-manylinux_2_28_ppc64le.whl", hash = "sha256:3c21d92ed15e9cfc6eb64c1f5a0326db22ca9c2566ca46d845119b45b4400361", size = 4904178, upload-time = "2026-03-25T23:33:55.725Z" }, - { url = "https://files.pythonhosted.org/packages/b1/1b/bf0e01a88efd0e59679b69f42d4afd5bced8700bb5e80617b2d63a3741af/cryptography-46.0.6-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:4668298aef7cddeaf5c6ecc244c2302a2b8e40f384255505c22875eebb47888b", size = 4441812, upload-time = "2026-03-25T23:33:57.364Z" }, - { url = "https://files.pythonhosted.org/packages/bb/8b/11df86de2ea389c65aa1806f331cae145f2ed18011f30234cc10ca253de8/cryptography-46.0.6-cp314-cp314t-manylinux_2_31_armv7l.whl", hash = "sha256:8ce35b77aaf02f3b59c90b2c8a05c73bac12cea5b4e8f3fbece1f5fddea5f0ca", size = 3963923, upload-time = "2026-03-25T23:33:59.361Z" }, - { url = "https://files.pythonhosted.org/packages/91/e0/207fb177c3a9ef6a8108f234208c3e9e76a6aa8cf20d51932916bd43bda0/cryptography-46.0.6-cp314-cp314t-manylinux_2_34_aarch64.whl", hash = "sha256:c89eb37fae9216985d8734c1afd172ba4927f5a05cfd9bf0e4863c6d5465b013", size = 4269695, upload-time = "2026-03-25T23:34:00.909Z" }, - { url = "https://files.pythonhosted.org/packages/21/5e/19f3260ed1e95bced52ace7501fabcd266df67077eeb382b79c81729d2d3/cryptography-46.0.6-cp314-cp314t-manylinux_2_34_ppc64le.whl", hash = "sha256:ed418c37d095aeddf5336898a132fba01091f0ac5844e3e8018506f014b6d2c4", size = 4869785, upload-time = "2026-03-25T23:34:02.796Z" }, - { url = "https://files.pythonhosted.org/packages/10/38/cd7864d79aa1d92ef6f1a584281433419b955ad5a5ba8d1eb6c872165bcb/cryptography-46.0.6-cp314-cp314t-manylinux_2_34_x86_64.whl", hash = "sha256:69cf0056d6947edc6e6760e5f17afe4bea06b56a9ac8a06de9d2bd6b532d4f3a", size = 4441404, upload-time = "2026-03-25T23:34:04.35Z" }, - { url = "https://files.pythonhosted.org/packages/09/0a/4fe7a8d25fed74419f91835cf5829ade6408fd1963c9eae9c4bce390ecbb/cryptography-46.0.6-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:8e7304c4f4e9490e11efe56af6713983460ee0780f16c63f219984dab3af9d2d", size = 4397549, upload-time = "2026-03-25T23:34:06.342Z" }, - { url = "https://files.pythonhosted.org/packages/5f/a0/7d738944eac6513cd60a8da98b65951f4a3b279b93479a7e8926d9cd730b/cryptography-46.0.6-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:b928a3ca837c77a10e81a814a693f2295200adb3352395fad024559b7be7a736", size = 4651874, upload-time = "2026-03-25T23:34:07.916Z" }, - { url = "https://files.pythonhosted.org/packages/cb/f1/c2326781ca05208845efca38bf714f76939ae446cd492d7613808badedf1/cryptography-46.0.6-cp314-cp314t-win32.whl", hash = "sha256:97c8115b27e19e592a05c45d0dd89c57f81f841cc9880e353e0d3bf25b2139ed", size = 3001511, upload-time = "2026-03-25T23:34:09.892Z" }, - { url = "https://files.pythonhosted.org/packages/c9/57/fe4a23eb549ac9d903bd4698ffda13383808ef0876cc912bcb2838799ece/cryptography-46.0.6-cp314-cp314t-win_amd64.whl", hash = "sha256:c797e2517cb7880f8297e2c0f43bb910e91381339336f75d2c1c2cbf811b70b4", size = 3471692, upload-time = "2026-03-25T23:34:11.613Z" }, - { url = "https://files.pythonhosted.org/packages/c4/cc/f330e982852403da79008552de9906804568ae9230da8432f7496ce02b71/cryptography-46.0.6-cp38-abi3-macosx_10_9_universal2.whl", hash = "sha256:12cae594e9473bca1a7aceb90536060643128bb274fcea0fc459ab90f7d1ae7a", size = 7162776, upload-time = "2026-03-25T23:34:13.308Z" }, - { url = "https://files.pythonhosted.org/packages/49/b3/dc27efd8dcc4bff583b3f01d4a3943cd8b5821777a58b3a6a5f054d61b79/cryptography-46.0.6-cp38-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:639301950939d844a9e1c4464d7e07f902fe9a7f6b215bb0d4f28584729935d8", size = 4270529, upload-time = "2026-03-25T23:34:15.019Z" }, - { url = "https://files.pythonhosted.org/packages/e6/05/e8d0e6eb4f0d83365b3cb0e00eb3c484f7348db0266652ccd84632a3d58d/cryptography-46.0.6-cp38-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:ed3775295fb91f70b4027aeba878d79b3e55c0b3e97eaa4de71f8f23a9f2eb77", size = 4414827, upload-time = "2026-03-25T23:34:16.604Z" }, - { url = "https://files.pythonhosted.org/packages/2f/97/daba0f5d2dc6d855e2dcb70733c812558a7977a55dd4a6722756628c44d1/cryptography-46.0.6-cp38-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:8927ccfbe967c7df312ade694f987e7e9e22b2425976ddbf28271d7e58845290", size = 4271265, upload-time = "2026-03-25T23:34:18.586Z" }, - { url = "https://files.pythonhosted.org/packages/89/06/fe1fce39a37ac452e58d04b43b0855261dac320a2ebf8f5260dd55b201a9/cryptography-46.0.6-cp38-abi3-manylinux_2_28_ppc64le.whl", hash = "sha256:b12c6b1e1651e42ab5de8b1e00dc3b6354fdfd778e7fa60541ddacc27cd21410", size = 4916800, upload-time = "2026-03-25T23:34:20.561Z" }, - { url = "https://files.pythonhosted.org/packages/ff/8a/b14f3101fe9c3592603339eb5d94046c3ce5f7fc76d6512a2d40efd9724e/cryptography-46.0.6-cp38-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:063b67749f338ca9c5a0b7fe438a52c25f9526b851e24e6c9310e7195aad3b4d", size = 4448771, upload-time = "2026-03-25T23:34:22.406Z" }, - { url = "https://files.pythonhosted.org/packages/01/b3/0796998056a66d1973fd52ee89dc1bb3b6581960a91ad4ac705f182d398f/cryptography-46.0.6-cp38-abi3-manylinux_2_31_armv7l.whl", hash = "sha256:02fad249cb0e090b574e30b276a3da6a149e04ee2f049725b1f69e7b8351ec70", size = 3978333, upload-time = "2026-03-25T23:34:24.281Z" }, - { url = "https://files.pythonhosted.org/packages/c5/3d/db200af5a4ffd08918cd55c08399dc6c9c50b0bc72c00a3246e099d3a849/cryptography-46.0.6-cp38-abi3-manylinux_2_34_aarch64.whl", hash = "sha256:7e6142674f2a9291463e5e150090b95a8519b2fb6e6aaec8917dd8d094ce750d", size = 4271069, upload-time = "2026-03-25T23:34:25.895Z" }, - { url = "https://files.pythonhosted.org/packages/d7/18/61acfd5b414309d74ee838be321c636fe71815436f53c9f0334bf19064fa/cryptography-46.0.6-cp38-abi3-manylinux_2_34_ppc64le.whl", hash = "sha256:456b3215172aeefb9284550b162801d62f5f264a081049a3e94307fe20792cfa", size = 4878358, upload-time = "2026-03-25T23:34:27.67Z" }, - { url = "https://files.pythonhosted.org/packages/8b/65/5bf43286d566f8171917cae23ac6add941654ccf085d739195a4eacf1674/cryptography-46.0.6-cp38-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:341359d6c9e68834e204ceaf25936dffeafea3829ab80e9503860dcc4f4dac58", size = 4448061, upload-time = "2026-03-25T23:34:29.375Z" }, - { url = "https://files.pythonhosted.org/packages/e0/25/7e49c0fa7205cf3597e525d156a6bce5b5c9de1fd7e8cb01120e459f205a/cryptography-46.0.6-cp38-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:9a9c42a2723999a710445bc0d974e345c32adfd8d2fac6d8a251fa829ad31cfb", size = 4399103, upload-time = "2026-03-25T23:34:32.036Z" }, - { url = "https://files.pythonhosted.org/packages/44/46/466269e833f1c4718d6cd496ffe20c56c9c8d013486ff66b4f69c302a68d/cryptography-46.0.6-cp38-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:6617f67b1606dfd9fe4dbfa354a9508d4a6d37afe30306fe6c101b7ce3274b72", size = 4659255, upload-time = "2026-03-25T23:34:33.679Z" }, - { url = "https://files.pythonhosted.org/packages/0a/09/ddc5f630cc32287d2c953fc5d32705e63ec73e37308e5120955316f53827/cryptography-46.0.6-cp38-abi3-win32.whl", hash = "sha256:7f6690b6c55e9c5332c0b59b9c8a3fb232ebf059094c17f9019a51e9827df91c", size = 3010660, upload-time = "2026-03-25T23:34:35.418Z" }, - { url = "https://files.pythonhosted.org/packages/1b/82/ca4893968aeb2709aacfb57a30dec6fa2ab25b10fa9f064b8882ce33f599/cryptography-46.0.6-cp38-abi3-win_amd64.whl", hash = "sha256:79e865c642cfc5c0b3eb12af83c35c5aeff4fa5c672dc28c43721c2c9fdd2f0f", size = 3471160, upload-time = "2026-03-25T23:34:37.191Z" }, + { url = "https://files.pythonhosted.org/packages/ba/19/797e2aaac9df6a66f1550f49979dc1b1e39ecd2077501c30efa81e8d5d67/cryptography-50.0.1-cp311-abi3-macosx_11_0_arm64.whl", hash = "sha256:b8f852c65863251b9e3a1b8c150ce21e59b522dbb6a7d4bc80e680d38388e986", size = 4010153, upload-time = "2026-08-25T19:44:03.155Z" }, + { url = "https://files.pythonhosted.org/packages/90/34/9ce9a62ed9dc82ca9fd6a34445b6904af56e5f38b3eae2ed32e49c36053d/cryptography-50.0.1-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:53e279950892dc102c6b4e52af03ae5ea92fac572a1ddab78ca73a997f62b69f", size = 4723133, upload-time = "2026-08-25T19:44:05.461Z" }, + { url = "https://files.pythonhosted.org/packages/57/26/e6d4fc8512a51a5f9ee7bfdbfb853bce1197087df40c9ad993ad370b846f/cryptography-50.0.1-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:ff838d62ec1bfce4f9ba7fa16f4a7b554cd8d0c299e6be37502161a660c84eef", size = 4712478, upload-time = "2026-08-25T19:44:07.375Z" }, + { url = "https://files.pythonhosted.org/packages/e6/de/d3cdc2815697aae84126cbd6a030ca7b6b452e28a88b501b836bd3aa7a86/cryptography-50.0.1-cp311-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:e74591e283fe6eb956416c929eb58262a719fe0311fd9054c62c3350ed8760d8", size = 4730726, upload-time = "2026-08-25T19:44:09.294Z" }, + { url = "https://files.pythonhosted.org/packages/55/32/38c0d344b98c06d34b5df8946565a9c0d6dbf32c8e0730a7f05f0a3c6cab/cryptography-50.0.1-cp311-abi3-manylinux_2_28_ppc64le.whl", hash = "sha256:5fe002589592ed749ce77fe0695fcbd3500dd61d7d6db5858a7544c612fa8e45", size = 5353524, upload-time = "2026-08-25T19:44:11.96Z" }, + { url = "https://files.pythonhosted.org/packages/e1/1b/82f0f0d8858d4432be1af790477edf62aef90324041aa07c57e57bef1af7/cryptography-50.0.1-cp311-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:51593d180cf6d179bde5c5d065bed81386b1f381656ae7d042b7ffc87a9895ad", size = 4746720, upload-time = "2026-08-25T19:44:14.051Z" }, + { url = "https://files.pythonhosted.org/packages/29/ba/042ca458b8c64348c768284b5d23e69b92ed53d057ab779fee628564676d/cryptography-50.0.1-cp311-abi3-manylinux_2_31_armv7l.whl", hash = "sha256:359e62deae718bce96170e223fdcb6357e4fbd3bb7a3a75f4430763532560e49", size = 4361866, upload-time = "2026-08-25T19:44:16.167Z" }, + { url = "https://files.pythonhosted.org/packages/39/3b/e96c1ef71edef71057c7e3c3d982ce8fda554e0c52d0cc19c18845cde3eb/cryptography-50.0.1-cp311-abi3-manylinux_2_34_aarch64.whl", hash = "sha256:e2ca8fd1b6b4b82a1c4cb02841d0837e3c12336c2e24b520ab8ab3b969733d8f", size = 4730028, upload-time = "2026-08-25T19:44:18.085Z" }, + { url = "https://files.pythonhosted.org/packages/e3/38/45abd72ef63f2e7d0754a6cacf97bd8b69512ace7f6130d24c39ece65da2/cryptography-50.0.1-cp311-abi3-manylinux_2_34_ppc64le.whl", hash = "sha256:76de83fbd91ac49c0feaaa983d0748fd7a53176afac5fb3bf7478d244f0eb527", size = 5308405, upload-time = "2026-08-25T19:44:20.197Z" }, + { url = "https://files.pythonhosted.org/packages/85/66/6ccca4722987ddedaa7fc9c3f4708af7431f5535666c174350830888c6b7/cryptography-50.0.1-cp311-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:51afcfceb15597cf2635068e4ac9a56b2abde622edde17f37d85fd7b5306497a", size = 4746230, upload-time = "2026-08-25T19:44:22.376Z" }, + { url = "https://files.pythonhosted.org/packages/13/0e/b1f92e013228111413f2e6743948b80bc24dfd3c1b87ba98ceea16f5df89/cryptography-50.0.1-cp311-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:be224a65493ec5b74a158ff22a5522ce4a5ca1e543c647a3a4730d4a09e5f959", size = 4862596, upload-time = "2026-08-25T19:44:24.472Z" }, + { url = "https://files.pythonhosted.org/packages/7e/22/c3654cccc856e9d682817b04ac3ee79731cb09ca6f95996a95c904de2883/cryptography-50.0.1-cp311-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:9ebcdd5519be9b652a46f507817a74591774fc3d6923ac364e4dfa64e36b291b", size = 5014082, upload-time = "2026-08-25T19:44:26.709Z" }, + { url = "https://files.pythonhosted.org/packages/42/8b/cb12b1b60c91b074ca6bf0fdd59aa8f10d8bc5f73af8faece86ef0421b37/cryptography-50.0.1-cp311-abi3-win_amd64.whl", hash = "sha256:aed8db4f6d71c51efb89530e12d9464e7bf2923d46c3205dc794a2a93f8c0648", size = 3842826, upload-time = "2026-08-25T19:44:28.784Z" }, + { url = "https://files.pythonhosted.org/packages/5b/f0/424cb557d99aa86ac55da5e2add02e2882e44047b6264f93ade1b975a993/cryptography-50.0.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:30a125032e5642a21ff816e021152bd4e7e94f03eff3f4b7fca41cd22bc3110f", size = 3973525, upload-time = "2026-08-25T19:44:30.7Z" }, + { url = "https://files.pythonhosted.org/packages/4d/72/3a2711d967977ab5fc80b782837c7e8d1ac7445e764c20c381a265c57ef3/cryptography-50.0.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:a0b1a59e3a089064a0ec309e9428c8e3ae4e161419d20ac33600767e83fc658a", size = 4708817, upload-time = "2026-08-25T19:44:32.773Z" }, + { url = "https://files.pythonhosted.org/packages/b4/f2/bb1f56e10815b789df0b409a69fa4992ff3d3fef9c72747f4a6b26fed38e/cryptography-50.0.1-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:8921d58f426793c5f1b47f0b59575780de9a095214958d0eb37d909593db8367", size = 4697300, upload-time = "2026-08-25T19:44:35.144Z" }, + { url = "https://files.pythonhosted.org/packages/08/bd/ed5396be499ffcf8807a585bfe38b71a1fbdd1c342b4f9b6d0ef5162a946/cryptography-50.0.1-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:a8f40ea47330e71b594a7e246898f93177c259490c63183dbaf9e571d71ed9a5", size = 4716039, upload-time = "2026-08-25T19:44:37.192Z" }, + { url = "https://files.pythonhosted.org/packages/f6/6e/1cf405c5c8e8df7545378048e954792f00b7f2367af8863ce8b8f3e10607/cryptography-50.0.1-cp314-cp314t-manylinux_2_28_ppc64le.whl", hash = "sha256:a255449073358275b64b67d3f595f268bbef70e72b6edb65e0c70c735bf739c9", size = 5332388, upload-time = "2026-08-25T19:44:39.16Z" }, + { url = "https://files.pythonhosted.org/packages/47/92/b4317e8c32c4f47b062f5398bd79106b220a124546f42be83bf32b761e2a/cryptography-50.0.1-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:8df2de9102026855887e4587084f6eabd80ed0f345b8ad8a7ac27ab9bf4723e0", size = 4730293, upload-time = "2026-08-25T19:44:41.298Z" }, + { url = "https://files.pythonhosted.org/packages/39/0d/a1e7633e2c744d0f2983320a27e924ef2264c79c56e1a58d5fb0a1cfd413/cryptography-50.0.1-cp314-cp314t-manylinux_2_31_armv7l.whl", hash = "sha256:ac02b07824d4d1001bd4367599f839c19cb171924c796e52c23508ac14c2c0cc", size = 4346031, upload-time = "2026-08-25T19:44:43.245Z" }, + { url = "https://files.pythonhosted.org/packages/88/dd/b215616f9bab3fc18510c78a4e5c9f362d77838503c363dc747c7d4f5c6f/cryptography-50.0.1-cp314-cp314t-manylinux_2_34_aarch64.whl", hash = "sha256:cbf74a81765ee67413503ca6e26dcc4f6f5a519822436cc0a1b97aab6c1b8a17", size = 4715344, upload-time = "2026-08-25T19:44:45.291Z" }, + { url = "https://files.pythonhosted.org/packages/b1/1b/ec3ebd31741d0e963612c4fe43caa39341b9b1e031e469820e42e4c83918/cryptography-50.0.1-cp314-cp314t-manylinux_2_34_ppc64le.whl", hash = "sha256:16c5ecd954b3330ebfb6605eca4fd952da8bef376551d5cc264534e3770a9ee6", size = 5287201, upload-time = "2026-08-25T19:44:47.297Z" }, + { url = "https://files.pythonhosted.org/packages/1a/01/0127d11a762b31a9ee0221894f540318761783f3fdc4bc5d057698caebd5/cryptography-50.0.1-cp314-cp314t-manylinux_2_34_x86_64.whl", hash = "sha256:79bf008d1f9af6071c797ad133e39915dfee7614f18f18f4db9072eb715064a3", size = 4730023, upload-time = "2026-08-25T19:44:49.435Z" }, + { url = "https://files.pythonhosted.org/packages/9e/b9/e7425ebfb599241a0c1d7000f1b466c3062da66c19d9525031315dff7213/cryptography-50.0.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:330fbb252391c596f1ae42c5754449dc924e6ad012dca8efe0d703f9f2d12ec6", size = 4847362, upload-time = "2026-08-25T19:44:51.94Z" }, + { url = "https://files.pythonhosted.org/packages/2d/fd/60d0ddf4defa12e482c9d5e0f554384d6e8ab25341fd15f060028fd92e6a/cryptography-50.0.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:42be3bb70596b3abe4ac097b75be223e8b3ab614a0e5de068e3dcc54d71d6149", size = 4999247, upload-time = "2026-08-25T19:44:53.876Z" }, + { url = "https://files.pythonhosted.org/packages/4d/56/bc4f2b209e766c93372cfcd59b781a0b2b59700f62a969580415b699c2b2/cryptography-50.0.1-cp314-cp314t-win_amd64.whl", hash = "sha256:f74455bb086a85d5e81246412602aaa97ed095e504cd40dd261ef50be42205bf", size = 3825806, upload-time = "2026-08-25T19:44:56.209Z" }, + { url = "https://files.pythonhosted.org/packages/84/a9/ee16a903f13755e914d1eecc482fe64d1f10761c3960e5d8fa6837377aff/cryptography-50.0.1-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:ca83d00d9e69cd5eb63f2e69c3a5a59e0cecae5ae14c6ae0b35830fe3b37bad0", size = 4035307, upload-time = "2026-08-25T19:44:58.305Z" }, + { url = "https://files.pythonhosted.org/packages/5e/a5/9ec7e81e8526c0d7a387d73386b2daed3f39e10d81a85930bd1b6bfba65c/cryptography-50.0.1-cp39-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:05ba322c4da95b262a212c345af888ef2c37c88c0509756ea00a0e6d68850f23", size = 4751900, upload-time = "2026-08-25T19:45:00.401Z" }, + { url = "https://files.pythonhosted.org/packages/7e/3c/0e77bd5ffcf078e9dd27d3074aad6c030d9b10d0bf69329d573c927a188c/cryptography-50.0.1-cp39-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:e22dfed744bd4002e909464cb23d2f0b05c6f3113a79ef2e9864a53db737c733", size = 4738357, upload-time = "2026-08-25T19:45:02.786Z" }, + { url = "https://files.pythonhosted.org/packages/27/3a/3c5f80daa4dcd47323c7af8a2fcb90de27a33564d4fcac69846c0972691a/cryptography-50.0.1-cp39-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:4c4188f7c0cf655be5c06342b817ed0f9595b69ffa2b12026e5353eed29dea88", size = 4758474, upload-time = "2026-08-25T19:45:04.889Z" }, + { url = "https://files.pythonhosted.org/packages/6e/2b/214cf0cf93db9628c3c20c896b229f327f6fb1b20e4b3743d8ad3f00af8b/cryptography-50.0.1-cp39-abi3-manylinux_2_28_ppc64le.whl", hash = "sha256:2ebbfb0f1fed745e91796e3e1080a1440423fdae8ece1b995a1d80883a409054", size = 5375862, upload-time = "2026-08-25T19:45:07.163Z" }, + { url = "https://files.pythonhosted.org/packages/d6/51/3f9701867a46b6c1740c9b52fc4d3bed6cbdcfedcc9b6e64305c07f39cff/cryptography-50.0.1-cp39-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:407fe2b6db00939c05c0e945e9914238f2f0a430974839429dafc82b1ee6bee5", size = 4772942, upload-time = "2026-08-25T19:45:09.396Z" }, + { url = "https://files.pythonhosted.org/packages/0d/5c/13ea642e08e2544d0f5396122055f4820cfacb3203562197b5967125ea97/cryptography-50.0.1-cp39-abi3-manylinux_2_31_armv7l.whl", hash = "sha256:2b34d76a652ea2b6faf777c35df230c5637842cd904e04f16230c3f9f03e4361", size = 4383347, upload-time = "2026-08-25T19:45:11.659Z" }, + { url = "https://files.pythonhosted.org/packages/84/d5/7d1fe1cb93f91c428093ff234e128c89ba8ea61a6f26aab406081f9b996e/cryptography-50.0.1-cp39-abi3-manylinux_2_34_aarch64.whl", hash = "sha256:01f41478cf33fc605a6a089cd56d28b45c6c0b45a1928b61797f2621a04bac71", size = 4758050, upload-time = "2026-08-25T19:45:13.745Z" }, + { url = "https://files.pythonhosted.org/packages/dd/04/557fc5ead96a829e0bc812a3b9dc4a52a2f27e4f7f5950da7ff27653a805/cryptography-50.0.1-cp39-abi3-manylinux_2_34_ppc64le.whl", hash = "sha256:fc3ed7ebd2a8c96f5b166de0ab9b624996bef3b07bbeb19364dfb78222c22c80", size = 5332955, upload-time = "2026-08-25T19:45:16.193Z" }, + { url = "https://files.pythonhosted.org/packages/8c/eb/5d7124083e8d8cda8f5b348f544b71ad6f707ad63193758ef4d8e569da02/cryptography-50.0.1-cp39-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:9dde0a357190eb3b1da1bb9ab750e9c85cba82ca5977aa0836cbb94e92611239", size = 4772694, upload-time = "2026-08-25T19:45:18.315Z" }, + { url = "https://files.pythonhosted.org/packages/63/8e/f1f955e0921dd2b6d22eae7e8d24a4c4b638d10735ffbf6a71f99eb0fcb8/cryptography-50.0.1-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:fd3718b960d0b5dd213cdf03f3bcb7000e69dda0de8b956061947ff6bcff5558", size = 4888413, upload-time = "2026-08-25T19:45:20.4Z" }, + { url = "https://files.pythonhosted.org/packages/1f/ab/89e2b798d2c3925f82e2bb72d5979f3d2f6da2dd22ef4a8cd8b70d920039/cryptography-50.0.1-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:2a93d05e34d5f67fba6f891fe85d929999baa7195e853923ea6d7576c9e68c5e", size = 5044355, upload-time = "2026-08-25T19:45:22.353Z" }, + { url = "https://files.pythonhosted.org/packages/99/89/87ef49ffe383ef4e147d27b7bf2088fb0b54ea409dd87b5a89442e5828a5/cryptography-50.0.1-cp39-abi3-win_amd64.whl", hash = "sha256:55d16b1ef3ee0958d893a977b19777887e546c9954ea81b200c3301a864013f2", size = 3875429, upload-time = "2026-08-25T19:45:24.418Z" }, ] [[package]] @@ -386,6 +383,20 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/02/ed/85113d22ab4268600542152bc4b5512abb1204552b90e04848ebc496aa5c/django_tailwind-4.4.2-py3-none-any.whl", hash = "sha256:0e4a2836cb36e8952700457d049fadb8743583017cef80fa3a374f8597c289f4", size = 23358, upload-time = "2025-12-05T18:23:39.462Z" }, ] +[[package]] +name = "docker" +version = "7.2.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "pywin32", marker = "sys_platform == 'win32'" }, + { name = "requests" }, + { name = "urllib3" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/88/7f/731ff914b0255d3d065f45fd4e626d4b8c95dbcbaada049f337a6ac16410/docker-7.2.0.tar.gz", hash = "sha256:cebb93773d334f778e023a7ee352a8d6e13ab1bd3b863a4d4a59dec897df43ac", size = 118731, upload-time = "2026-07-09T14:53:46.39Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/75/23/529140fe1aab80fc6992f93a706deec709140a6397439139a054e1515c45/docker-7.2.0-py3-none-any.whl", hash = "sha256:a3f45fdeb9165e2d25d9a1d02ddf3bc70fb572cf5ebbf9b58558c22caf29b71f", size = 148775, upload-time = "2026-07-09T14:53:45.224Z" }, +] + [[package]] name = "elastic-transport" version = "9.2.1" @@ -600,6 +611,7 @@ dev = [ { name = "pytest" }, { name = "pytest-cov" }, { name = "pytest-django" }, + { name = "testcontainers", extra = ["mysql"] }, ] [package.metadata] @@ -634,6 +646,8 @@ dev = [ { name = "pytest", specifier = ">=9.0.2" }, { name = "pytest-cov", specifier = ">=7.1.0" }, { name = "pytest-django", specifier = ">=4.10.0" }, + { name = "testcontainers", extras = ["mysql"], specifier = ">=4.8.0" }, + { name = "testcontainers", extras = ["postgres"], specifier = ">=4.8.0" }, ] [[package]] @@ -800,6 +814,11 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/c4/bd/2534e130295c8cfd4f0a2e31623baab7502278f1e97bcfe61db75656a77f/pymysql-1.2.0-py3-none-any.whl", hash = "sha256:62169ce6d5510f08e140c5e7990ee884a9764024e4a9a27b2cc11f1099322ae0", size = 45716, upload-time = "2026-05-19T08:26:20.974Z" }, ] +[package.optional-dependencies] +rsa = [ + { name = "cryptography" }, +] + [[package]] name = "pysnmp" version = "7.1.26" @@ -888,6 +907,34 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/67/0f/019d3949a40280f6193b62bc010177d4ce702d0fce424322286488569cd3/python_discovery-1.2.1-py3-none-any.whl", hash = "sha256:b6a957b24c1cd79252484d3566d1b49527581d46e789aaf43181005e56201502", size = 31674, upload-time = "2026-03-26T22:30:43.396Z" }, ] +[[package]] +name = "python-dotenv" +version = "1.2.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/6a/53/ed9d74092561d4b01a2ef1349d52cdbc135e526c245f366b089cfca6de49/python_dotenv-1.2.3.tar.gz", hash = "sha256:a20a594dabeaa385725aa239d5244871c143ecb356add8a20fcf23773a6c3a35", size = 58945, upload-time = "2026-08-16T16:54:54.067Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/0d/17/c5c6b53ddc18f297992099b3d9ec16c855c0ccc83263a21fe4d1c625ec6c/python_dotenv-1.2.3-py3-none-any.whl", hash = "sha256:904552145e8bfed22162c09dab1c2b9b54fefa7b23ba780f4f26ca0316b0f0d9", size = 22780, upload-time = "2026-08-16T16:54:52.473Z" }, +] + +[[package]] +name = "pywin32" +version = "312" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/83/ff/32aa7d2ed0ab12b323aaa64f9b75e6ad4f8fd09f9ccfc28c79414d46838d/pywin32-312-cp312-cp312-win32.whl", hash = "sha256:dab4f65ac9c4e48400a2a0530c46c3c579cd5905ecd11b80692373915269208b", size = 6371877, upload-time = "2026-06-04T07:49:28.836Z" }, + { url = "https://files.pythonhosted.org/packages/03/d9/77040d3b43df3f3be32ea289433d660d2727f5ba327bc73be835127d9d60/pywin32-312-cp312-cp312-win_amd64.whl", hash = "sha256:b457f6d628a47e8a7346ce22acb7e1a46a4a78b52e1d17e1af56871bd19a93bc", size = 6914841, upload-time = "2026-06-04T07:49:31.85Z" }, + { url = "https://files.pythonhosted.org/packages/e3/cc/7b1ec671775756020a0ee7f4feeaf3c568f0ab86bd3900088cf986937a92/pywin32-312-cp312-cp312-win_arm64.whl", hash = "sha256:6017c58e12f6809fbb0555b75df144c2922a9ffd18e4b9b5afa863b6c1a9d950", size = 6727901, upload-time = "2026-06-04T07:49:34.244Z" }, + { url = "https://files.pythonhosted.org/packages/2d/41/12fbfd7f36ed2146d8bc9de96c2741296bf0d490b98508496cff322e274c/pywin32-312-cp313-cp313-win32.whl", hash = "sha256:7a27df850933d16a8eabfbaeb73d52b273e2da667f80d70b01a89d1f6828d02c", size = 6370184, upload-time = "2026-06-04T07:49:36.253Z" }, + { url = "https://files.pythonhosted.org/packages/ba/db/36a78e3403099d31d9746d13fdcde5accc43c1155f375a34d15983a479a7/pywin32-312-cp313-cp313-win_amd64.whl", hash = "sha256:c53e878d15a1c44788082bfe712a905433473aa38f86375b7cf8b45e3acbaaf9", size = 6914298, upload-time = "2026-06-04T07:49:38.876Z" }, + { url = "https://files.pythonhosted.org/packages/84/37/c1697194092b76de9ed47ca124323f02c57ffc8a45c06f88a3d5acaf01eb/pywin32-312-cp313-cp313-win_arm64.whl", hash = "sha256:59aba5d5940842075343a5ddc6b11f1cdf0d1567fe745290359dfbcc7c2eb831", size = 6727640, upload-time = "2026-06-04T07:49:41.083Z" }, + { url = "https://files.pythonhosted.org/packages/fc/2b/1f3cded5822fd49c02f40544cbb5f58c7cfd6b1694869fd476cb6170ee97/pywin32-312-cp314-cp314-win32.whl", hash = "sha256:a77a90fbb6881238d2ca9c6fd797b25817f3768fe78d214a90137ff055a75f5b", size = 6468928, upload-time = "2026-06-04T07:49:43.188Z" }, + { url = "https://files.pythonhosted.org/packages/21/82/3bf86d2e2808902013132e1ce905a7da0da53790f3836c64bf44d55e24f3/pywin32-312-cp314-cp314-win_amd64.whl", hash = "sha256:a4dd3a848290ef724347b19f301045831d8e802fa4464f491b98b1e0a081432e", size = 7024157, upload-time = "2026-06-04T07:49:45.34Z" }, + { url = "https://files.pythonhosted.org/packages/a4/0e/73f6d6800b4f27655abd9e9f6aaeaefcddb2b946e4674efa2bab184a7f7b/pywin32-312-cp314-cp314-win_arm64.whl", hash = "sha256:9fce94568364e0155e6dfb781ac5d95903be8baf28670632beab1b523f300daa", size = 6839598, upload-time = "2026-06-04T07:49:47.613Z" }, + { url = "https://files.pythonhosted.org/packages/eb/61/caa39686032d2ebdd04ff0ab5cbe163126c0066d98e00c9018646e42393b/pywin32-312-cp315-cp315-win32.whl", hash = "sha256:5c1fbe4a937a73ae9297384a3da38518cbc694c68ad8a809b2e19acd350f03ed", size = 6471159, upload-time = "2026-06-04T07:49:50.035Z" }, + { url = "https://files.pythonhosted.org/packages/0f/cd/7e1de64a4a6f69c04214169657ccab0d93a670ea50e35eb8f489d7378249/pywin32-312-cp315-cp315-win_amd64.whl", hash = "sha256:c2f03a0f73f804a13c2735b99392b0cd426bb4f2c4d0178e5ac966a0f21618d5", size = 7025293, upload-time = "2026-06-04T07:49:54.857Z" }, + { url = "https://files.pythonhosted.org/packages/23/ed/4532e9388e65fa16b46776ef47ad631a64eda1631884488af707666350ed/pywin32-312-cp315-cp315-win_arm64.whl", hash = "sha256:a8597d28f267b39074aef51fa593530082b39cbe5a074226096857b1fed2dfb9", size = 6840337, upload-time = "2026-06-04T07:49:57.531Z" }, +] + [[package]] name = "pyyaml" version = "6.0.3" @@ -1055,6 +1102,41 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/e9/44/75a9c9421471a6c4805dbf2356f7c181a29c1879239abab1ea2cc8f38b40/sniffio-1.3.1-py3-none-any.whl", hash = "sha256:2f6da418d1f1e0fddd844478f41680e794e6051915791a034ff65e5f100525a2", size = 10235, upload-time = "2024-02-25T23:20:01.196Z" }, ] +[[package]] +name = "sqlalchemy" +version = "2.0.52" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "greenlet", marker = "platform_machine == 'AMD64' or platform_machine == 'WIN32' or platform_machine == 'aarch64' or platform_machine == 'amd64' or platform_machine == 'ppc64le' or platform_machine == 'win32' or platform_machine == 'x86_64'" }, + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/3b/21/77b4c147963073040dc3c3a5cb7a8c3001a1893c0209432cb77f9df836aa/sqlalchemy-2.0.52.tar.gz", hash = "sha256:5e2d46356ac2ccb7d268ab6c2319ac6a2b42f1b8d5fd8bd3d46855cd82abee97", size = 9945637, upload-time = "2026-08-11T19:07:09.829Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e0/d5/1b77a026d161f98a08f11af1a5f6c47b98ee7c7e2648af525a1004826c78/sqlalchemy-2.0.52-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:be8c49131665dfe2cc74c498aa1240ffb548d0fd901325dd11c2c7a18956f727", size = 2170940, upload-time = "2026-08-11T20:58:11.25Z" }, + { url = "https://files.pythonhosted.org/packages/54/bd/f444444adb37b5d53753fb1730ee7a421628e2e3b756c4da461af7e6394a/sqlalchemy-2.0.52-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1b2d9e507a458832adcfbd8af6e2036ddf069b7710b799448542ebccae2dceee", size = 3383415, upload-time = "2026-08-11T21:02:38.534Z" }, + { url = "https://files.pythonhosted.org/packages/be/57/2eadf93a552568c57e8680b7e58bb5e9770d80942a1bdbaf4f2f63f0d7c8/sqlalchemy-2.0.52-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8738008376d22f30f411ea3efecf39b51110b6996d80bb73786f30bcfdd5fd3b", size = 3398577, upload-time = "2026-08-11T21:16:59.092Z" }, + { url = "https://files.pythonhosted.org/packages/15/c3/2887cf9dd111d1fbf05d22165b404c221ef43e029f7a2695e7302f27a7cc/sqlalchemy-2.0.52-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:37a4d548327b6cab9c7d8cdb4e0e82feabee0110c4d150059068e2d1cfbd99ee", size = 3328225, upload-time = "2026-08-11T21:02:40.183Z" }, + { url = "https://files.pythonhosted.org/packages/02/0f/466bdf9e1feeeef5587f868c187d8687e21ff8c85b1775e9041130181132/sqlalchemy-2.0.52-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:e49f51a5d59857a7a0dcaf9469febf7197d9394bd88f00d69c2c4e848112cdbf", size = 3357374, upload-time = "2026-08-11T21:17:01.076Z" }, + { url = "https://files.pythonhosted.org/packages/22/20/5c2b4583904af4173076dda1c9e53c9e2ffc7a702d2efde0216bbacbf7cb/sqlalchemy-2.0.52-cp312-cp312-win32.whl", hash = "sha256:afda3ec521d0517d0de783fc70030775841900896d832de5bbd066549290470e", size = 2129366, upload-time = "2026-08-11T21:14:50.991Z" }, + { url = "https://files.pythonhosted.org/packages/ed/06/543dab8ef62d4e9fb96fb31a30c2b8b14a8763bccf48d428294d6b3041c0/sqlalchemy-2.0.52-cp312-cp312-win_amd64.whl", hash = "sha256:2d5e53e36e37129fe0be8b9d08b6e4052c10a963ee6cda56c8c10dcc194b99ca", size = 2157344, upload-time = "2026-08-11T21:14:52.453Z" }, + { url = "https://files.pythonhosted.org/packages/7f/18/e30c6fe1eca1bf34a39fbdd6066121cc9974c850faf6f349eac563697a26/sqlalchemy-2.0.52-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:2eb3c6a64b1bfe6704777cfd504e7b8ad093a5f3e03ce67663a5e6742f294e43", size = 2167724, upload-time = "2026-08-11T20:58:12.679Z" }, + { url = "https://files.pythonhosted.org/packages/d0/56/2e17d161a4f7ecc1c2ffb93e607b4e1898bb551b451b283235acb8f6ce47/sqlalchemy-2.0.52-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:923bb183c1dc64fdf7b717965e3d59938ec4f8b8710b419a21ce403e5da9a9e1", size = 3321189, upload-time = "2026-08-11T21:02:41.932Z" }, + { url = "https://files.pythonhosted.org/packages/cf/b8/8490916e893f3f8d74dc9cc54c078619364999dee37047a188e73abbc852/sqlalchemy-2.0.52-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:651d6d8782e80679e6151707c7b490834d46ada526328895abf567f25e63d29c", size = 3338185, upload-time = "2026-08-11T21:17:02.597Z" }, + { url = "https://files.pythonhosted.org/packages/8b/f7/752cc8ee453da222829b3f5c4613614bf750d97429363b70414fa10478e4/sqlalchemy-2.0.52-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:b08cddb8989775e3c88799d86704bdfc3ee6e9846118201aa5997f16f27e3a15", size = 3271698, upload-time = "2026-08-11T21:02:43.963Z" }, + { url = "https://files.pythonhosted.org/packages/51/e6/074ade0c07b9e4c8e8bca46820320ed94df9702afdb6f2af06623068d2e6/sqlalchemy-2.0.52-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:ab66fa9618269390d4dfa222f2f2f88f7bc4bf5da13905131b818217db7e8057", size = 3308936, upload-time = "2026-08-11T21:17:04.172Z" }, + { url = "https://files.pythonhosted.org/packages/66/07/557c0d04716705599227945ac14e0a17ad0338e899f37d8c2ddff4dcc663/sqlalchemy-2.0.52-cp313-cp313-win32.whl", hash = "sha256:c63bda077685c85ca513286547a531ba57e7a68cf0a7ed3bafcc2bbd18896f4d", size = 2127308, upload-time = "2026-08-11T21:14:53.879Z" }, + { url = "https://files.pythonhosted.org/packages/96/4e/226eda27654318ce525d043025221f689abef883da2c7126f9065121618c/sqlalchemy-2.0.52-cp313-cp313-win_amd64.whl", hash = "sha256:9876b09b9f1ce7398b0ffece585c0a911244c53191187341f6bcae640e133751", size = 2153876, upload-time = "2026-08-11T21:14:55.527Z" }, + { url = "https://files.pythonhosted.org/packages/d5/f5/71cb30af58c9b80a4e1fac0b73bb48f86d497a774a6a2eb6d2f1e657bb73/sqlalchemy-2.0.52-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:410d52be41d17f1a236d19520fbe776257dc16516ed06bd16d433311842aefd9", size = 2169537, upload-time = "2026-08-11T20:58:13.855Z" }, + { url = "https://files.pythonhosted.org/packages/4c/93/d07ebd645d1b07b6b5ed63450a70f063a346a7e0f2c8810daf2e532400cb/sqlalchemy-2.0.52-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:dfe9ce533dbe4d0a2ae1486546619bd30b76bcd670539a44d910361376175f5e", size = 3319606, upload-time = "2026-08-11T21:02:45.829Z" }, + { url = "https://files.pythonhosted.org/packages/ae/5c/290c84c7c2566ecd3b65baaae0fddec9bc33b033b398a06123bb86fbfc6e/sqlalchemy-2.0.52-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:812bae5138bfc0aa46fb0686da0fc7f581f68e2bbb05bc24c3713bebaedd1437", size = 3323642, upload-time = "2026-08-11T21:17:05.675Z" }, + { url = "https://files.pythonhosted.org/packages/13/f5/2cc160590ca49173359557880b92a0572293ccb899e8f6cedf150c5a3ddf/sqlalchemy-2.0.52-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:50bff43b632a56fbf5ed9afdd76307e1512b62051bcd5afb341ae67205bbb6c8", size = 3268125, upload-time = "2026-08-11T21:02:47.649Z" }, + { url = "https://files.pythonhosted.org/packages/35/f3/ea8933fc9f7d1353e9c2ff9965eae687c4cef181120574591ed2fa0633e1/sqlalchemy-2.0.52-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:49565daf5af554f538e23aef1fc81a95a4e49658f152285e45c02f5fc44f04cd", size = 3289516, upload-time = "2026-08-11T21:17:07.267Z" }, + { url = "https://files.pythonhosted.org/packages/45/67/05cf86541c1e1716fca1e4a996954a439cd74501707cda607fb7cb02ef50/sqlalchemy-2.0.52-cp314-cp314-win32.whl", hash = "sha256:ab9da41e61b9979b910499d633b241df20c51ee5037e5405b11c2faac3cbe1a2", size = 2130249, upload-time = "2026-08-11T21:14:57.273Z" }, + { url = "https://files.pythonhosted.org/packages/96/d7/8ac6ffa1e36169e762ef65bd835046abb2251b1bc17f8f6708e14ed8d31f/sqlalchemy-2.0.52-cp314-cp314-win_amd64.whl", hash = "sha256:a593db51b3bae75db17a5738ad5f992244b3a03863f83c28117ee482c6a3f76d", size = 2156718, upload-time = "2026-08-11T21:14:58.667Z" }, + { url = "https://files.pythonhosted.org/packages/dc/4b/e01a737eef378e734cc6394a82248a6ce13b167dfa36c731075ce9fc9c64/sqlalchemy-2.0.52-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:c1e61d08bdf4ee2f41024569e3400de7d6734ba498144766b11260936ccfa582", size = 2190344, upload-time = "2026-08-11T19:53:21.393Z" }, + { url = "https://files.pythonhosted.org/packages/b3/3f/3582293d1e185e71d19d7c731c3e2ee20ba21981c4a1115c0806c1f62120/sqlalchemy-2.0.52-py3-none-any.whl", hash = "sha256:3b81b8363a919ce53453591cdb93702e6bd54ade6c4fa2f468fc053baee5ed89", size = 1950700, upload-time = "2026-08-11T20:47:21.603Z" }, +] + [[package]] name = "sqlparse" version = "0.5.5" @@ -1064,6 +1146,28 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/49/4b/359f28a903c13438ef59ebeee215fb25da53066db67b305c125f1c6d2a25/sqlparse-0.5.5-py3-none-any.whl", hash = "sha256:12a08b3bf3eec877c519589833aed092e2444e68240a3577e8e26148acc7b1ba", size = 46138, upload-time = "2025-12-19T07:17:46.573Z" }, ] +[[package]] +name = "testcontainers" +version = "4.15.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "docker" }, + { name = "python-dotenv" }, + { name = "typing-extensions" }, + { name = "urllib3" }, + { name = "wrapt" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/4b/13/2cc466bddf26d0085f30a2b2bd56b7f8708b54a54db833eec97c5c69129b/testcontainers-4.15.0.tar.gz", hash = "sha256:085cde086337632e19002719460b7b80bbab2bdd51bb3ea04f77d0de96504706", size = 95340, upload-time = "2026-07-24T23:08:01.731Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/00/7e/424aac8b355597835deb333e757a0e94b5ccf38ad00f07fe6ed1f4e17c88/testcontainers-4.15.0-py3-none-any.whl", hash = "sha256:8796c14e76604031ad39cf0ed3b8e9806283a1fbf5270965c2b1c594caa31b74", size = 160771, upload-time = "2026-07-24T23:08:00.13Z" }, +] + +[package.optional-dependencies] +mysql = [ + { name = "pymysql", extra = ["rsa"] }, + { name = "sqlalchemy" }, +] + [[package]] name = "typing-extensions" version = "4.15.0" @@ -1115,6 +1219,81 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/db/eb/d5583a11486211f3ebd4b385545ae787f32363d453c19fffd81106c9c138/whitenoise-6.12.0-py3-none-any.whl", hash = "sha256:fc5e8c572e33ebf24795b47b6a7da8da3c00cff2349f5b04c02f28d0cc5a3cc2", size = 20302, upload-time = "2026-02-27T00:05:40.086Z" }, ] +[[package]] +name = "wrapt" +version = "2.4.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/65/ba/8dc25478ed234dacc7d83c671634f347d0bdfb65bf0502f41879cf2f15a9/wrapt-2.4.0.tar.gz", hash = "sha256:7082fc1f94b020ac275870c4af71b09cff22876fe6e9c4c0ad01ea21d217b288", size = 161179, upload-time = "2026-08-30T04:41:51.424Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f0/22/581a0b44349d5babe526c958f365b8126e0fbd8fc2810e80446c47358050/wrapt-2.4.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:ef4e2d6e399ce6eecc80179a6b9ef6544f121288f95fc132bc36c9d9503903af", size = 96374, upload-time = "2026-08-30T04:39:42.335Z" }, + { url = "https://files.pythonhosted.org/packages/5d/90/095984648cec62a786bb27c0b50f6cfa5856d1e073ba1006fe148d190084/wrapt-2.4.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:6b9b32d5e4f0a179cef5075cc79b79d6d3482c44c434c12969e48c6719e06d95", size = 96178, upload-time = "2026-08-30T04:39:43.789Z" }, + { url = "https://files.pythonhosted.org/packages/f8/fd/b20e3cb3cab35131b515edf18e8cd777dff680fc76fc00919481f4e536af/wrapt-2.4.0-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:d7dbbdbfdacb85c2d962fa52db791c77943fd777d600d74c95af2d53b32f5a94", size = 227806, upload-time = "2026-08-30T04:39:45.264Z" }, + { url = "https://files.pythonhosted.org/packages/08/75/c8dfba5e0caf17cd0718a0cbbe76cb85e637a2d65183fb728232419f6fca/wrapt-2.4.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:39cd68df4dff79f5336f9c745c06259d204bcb42d504040c9c91eac9e2abb39c", size = 229004, upload-time = "2026-08-30T04:39:47.068Z" }, + { url = "https://files.pythonhosted.org/packages/42/05/d4853fbd33e5860b10d5aec690f563547a92a82e61fb8bb2d4ece1ce3570/wrapt-2.4.0-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:2a9f1a2f75bb95257cc5744e255e10a5a86e923f328b40ad3dbf9d8d03430013", size = 208934, upload-time = "2026-08-30T04:39:48.73Z" }, + { url = "https://files.pythonhosted.org/packages/a3/66/23d0e8de9b411fd198af5121627587563657370c8d509fbe5ea8adb3df79/wrapt-2.4.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:8763ad01e3725b7751a4575f38bbcc19c0aa0822fec91c5c5bd21ce3ce7e1d2b", size = 225709, upload-time = "2026-08-30T04:39:50.287Z" }, + { url = "https://files.pythonhosted.org/packages/01/37/3b357bc90530d510ae59ae7ac48265c482ae899e47637ca4436645688b40/wrapt-2.4.0-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:9125c6dbe8b88c00dd8ef4fc1e55757e8eb4720b6b2b2cc610a45bd32bd28c57", size = 207090, upload-time = "2026-08-30T04:39:51.78Z" }, + { url = "https://files.pythonhosted.org/packages/6d/0c/d8a5c6dbcc2d221308223bcea4130c6332454a855cb4dbd5dcb2360b13b2/wrapt-2.4.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:28f5de1526831b8f173889a436e289fe181ede8c66c9feb669d1aca8fd602eaf", size = 216269, upload-time = "2026-08-30T04:39:53.641Z" }, + { url = "https://files.pythonhosted.org/packages/92/93/cc9fc8fef1d3d25edaa1c2dc2337b556dc1d0613ddc1c4a6fe9ee08ad705/wrapt-2.4.0-cp312-cp312-win32.whl", hash = "sha256:a9ca1cdb3f7facb4990c7739ea5afbaceeb6728d066feedde03a4cfe83b29b03", size = 91187, upload-time = "2026-08-30T04:39:55.38Z" }, + { url = "https://files.pythonhosted.org/packages/ad/ec/a7b10705172bdb669b9687a8ff68bbe5f566437d2a49ad6d976af48b6d10/wrapt-2.4.0-cp312-cp312-win_amd64.whl", hash = "sha256:8b464316489fb2fca0669ea0f8f07290054a0f26fc72982d3e4cf95469628ba9", size = 96423, upload-time = "2026-08-30T04:39:56.81Z" }, + { url = "https://files.pythonhosted.org/packages/83/7a/e838ac6463a1a1a1817b2f184ee2aa20c54692b80368c5063403c8d2461c/wrapt-2.4.0-cp312-cp312-win_arm64.whl", hash = "sha256:db1285071ea09a7767fac608e7b5c7b03c09833b06186875a359905fbc659d29", size = 93003, upload-time = "2026-08-30T04:39:58.237Z" }, + { url = "https://files.pythonhosted.org/packages/19/86/f9de4e11582ff96ad2199eeeceaa17faa27bbdc599243f520070c4f3de07/wrapt-2.4.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:5c5c4c728cd22a36e4b8bb5df4a7d3bccaa865d27725b36eeb3b6f18fb2e1bc2", size = 96041, upload-time = "2026-08-30T04:39:59.575Z" }, + { url = "https://files.pythonhosted.org/packages/c3/ab/1dbf50802bea3b46192fd0dc39bb0eb2e77a064c813b2bbd88d2888ad49f/wrapt-2.4.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:7de5b8d94417e55c02be50cc226e0ae1209bbc73813bf691dff3979c94438115", size = 96269, upload-time = "2026-08-30T04:40:01.182Z" }, + { url = "https://files.pythonhosted.org/packages/cb/a3/a3b5cde1cd06e04b6e95134eb3187a0a7da607a530e7795b221d4e4fa819/wrapt-2.4.0-cp313-cp313-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:6436e2bda993a3eb69a1b317fc831c8ebcafb5704c390859ebd49f81218c4bbb", size = 225787, upload-time = "2026-08-30T04:40:02.715Z" }, + { url = "https://files.pythonhosted.org/packages/cb/f7/d100f6c348b7669f19119cf890dcd4764623e2233af065586d110e0cd99e/wrapt-2.4.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e084558fbd112d2e1e34b0f5c71e45a3405bdad51a17150368a959bcf6697964", size = 226649, upload-time = "2026-08-30T04:40:04.647Z" }, + { url = "https://files.pythonhosted.org/packages/52/c6/3af8df515d5d7e92306957536f3468c6bdfecbe3659f99dbf09a468c2c4c/wrapt-2.4.0-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:e78c947e18fadfd690c9420c30a96d221feeb93fc8f1cc00509b370ac16c3114", size = 206760, upload-time = "2026-08-30T04:40:06.332Z" }, + { url = "https://files.pythonhosted.org/packages/b9/c5/40d355552bd3eb6c5186e26051c19b573d24d7896de42caa7937d6b5ca9f/wrapt-2.4.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:08d8378c4514ac8dcc0ace76044cf87a873e6a52b5e6109834c8fb9037f4441b", size = 223467, upload-time = "2026-08-30T04:40:07.829Z" }, + { url = "https://files.pythonhosted.org/packages/40/ab/d198eebdb39f0d7e182e771e590a36673489cd58cebdad8aa273dcf28e04/wrapt-2.4.0-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:93180c2199784dd6a1075b33f9ed636bd0966821edbece6b3d5379b1c4f0bb7d", size = 205358, upload-time = "2026-08-30T04:40:09.344Z" }, + { url = "https://files.pythonhosted.org/packages/1c/0e/974a60672ad507d39a3d8a1c6351ef37fe65b07240d000ceba5d2b83e9e9/wrapt-2.4.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:3d5e5eb76fb87e62752af751d2dcd9d1cd986b12037d2e1363d109ba716029e8", size = 214654, upload-time = "2026-08-30T04:40:10.923Z" }, + { url = "https://files.pythonhosted.org/packages/cc/5a/8b2db70206db0a4246758e0472ce344cb9636217113ef70640fc8d2ce874/wrapt-2.4.0-cp313-cp313-win32.whl", hash = "sha256:49bb5a572469e0e18163a8ec2aa972135a0929899ecbe627665f274506e1b5b4", size = 91171, upload-time = "2026-08-30T04:40:12.895Z" }, + { url = "https://files.pythonhosted.org/packages/fe/1e/e782b511c680dbe7369c92e7d981484aacca0cda584da1f28a84cd9a8e1a/wrapt-2.4.0-cp313-cp313-win_amd64.whl", hash = "sha256:b1737f46b1e4a81eb93500a7f2854319e1c7a86e8863fb050b7b4daadd5a4178", size = 96178, upload-time = "2026-08-30T04:40:14.336Z" }, + { url = "https://files.pythonhosted.org/packages/9f/62/095ba31123fa5dd482d6183c05200b061314aabbd5442c010aba4b03ff1c/wrapt-2.4.0-cp313-cp313-win_arm64.whl", hash = "sha256:f1e9e088094f4895f84ab043e7d59401df137d663efbf1e80c82144882960830", size = 92949, upload-time = "2026-08-30T04:40:15.935Z" }, + { url = "https://files.pythonhosted.org/packages/1f/dd/1f269e4daf0c992f675e1ca2de6b1683b761c6d0aeb6c7b4b412486823ea/wrapt-2.4.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:788e473d1a6786d29d577b1e2bd95e214c09cdafde84907c522c31069c9acfac", size = 96386, upload-time = "2026-08-30T04:40:17.584Z" }, + { url = "https://files.pythonhosted.org/packages/be/42/7ecef06d33c0121c68d66a8a695efe67ebaa57218c1c61c585eca2a6117a/wrapt-2.4.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:947bd4b3438167b3638bf5477cb83a068a586ffb6d331ac427f39839c2b93b3c", size = 96532, upload-time = "2026-08-30T04:40:19.116Z" }, + { url = "https://files.pythonhosted.org/packages/c2/e3/8fdc9eba0e6cbbfe8303e1e807d734691309a27970b2ea458d099f1a46b0/wrapt-2.4.0-cp314-cp314-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:3a69161cae7f0dca44c89c1d14146b4a0508a0c3cad98b3f2db1f4e9016c94ba", size = 228775, upload-time = "2026-08-30T04:40:20.604Z" }, + { url = "https://files.pythonhosted.org/packages/f4/77/4ac5882abfb29bf9821c5fa5cf9f30241a194e0f47faa2682b9b29765278/wrapt-2.4.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0536f5d85ff6a157ebe7e0fe08c5479943742cf1ce59569075a66159efcbc495", size = 229029, upload-time = "2026-08-30T04:40:22.186Z" }, + { url = "https://files.pythonhosted.org/packages/c9/c5/8a3608311a02faf3e5c072da38d06a7c623150fc258e29f18fe377d91703/wrapt-2.4.0-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:5f041ed6a4d571010944bd6cfad9072db463e1851877b6d3227467a44af37456", size = 210436, upload-time = "2026-08-30T04:40:23.953Z" }, + { url = "https://files.pythonhosted.org/packages/de/90/e0cbc43f435fd39df25460e9f173e7b96f3dac5c7f66be41c7227166f021/wrapt-2.4.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:f7fed45dbadf5d98a52bfff9624d3cca00affeb9543d493c9632b7a53cdd35c9", size = 226586, upload-time = "2026-08-30T04:40:25.507Z" }, + { url = "https://files.pythonhosted.org/packages/81/6c/7e5f2143228635ec139ef6df733dc477049f7d96a0c49deb23944a73ed6a/wrapt-2.4.0-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:5cc2e7c7b6032e11a2b367a9baadaf0c5241feff2d8205260d87f1aa6dbdf84b", size = 208880, upload-time = "2026-08-30T04:40:27.128Z" }, + { url = "https://files.pythonhosted.org/packages/10/16/1de84402bb7a0916e10739bf6586e031244172b299e87c8cff2a04baf9ff/wrapt-2.4.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:72826910a1cf5a081234720fd43011304b899acfee219af49148155b4d795533", size = 216689, upload-time = "2026-08-30T04:40:28.844Z" }, + { url = "https://files.pythonhosted.org/packages/20/19/cd6bd5050381a541b44be97c4e0994eed60c5f439f4314f95eb5777d6c1a/wrapt-2.4.0-cp314-cp314-win32.whl", hash = "sha256:0eca69c9e93518240abe8801fb9b2726116a6e48172e4564c2651a2e14521747", size = 91581, upload-time = "2026-08-30T04:40:30.592Z" }, + { url = "https://files.pythonhosted.org/packages/a4/f8/b642f3184619adde676ad449030bcbeae6cc78ea07a92f0b5fddeec4c4e6/wrapt-2.4.0-cp314-cp314-win_amd64.whl", hash = "sha256:63b94f401d7ae3a9a3027472fd3a3ff38afd2ed293b2f0b3b84a6d133a9f99a3", size = 96510, upload-time = "2026-08-30T04:40:32.1Z" }, + { url = "https://files.pythonhosted.org/packages/4d/3b/3415a18b91221261eeac85bf8ee23dfb0e2a39d76b9703a797efca177439/wrapt-2.4.0-cp314-cp314-win_arm64.whl", hash = "sha256:6b3e082d43f592fcd381aee46354a11ce887a813ce5bbcedd9766fd681723c09", size = 93648, upload-time = "2026-08-30T04:40:33.563Z" }, + { url = "https://files.pythonhosted.org/packages/ac/90/80cf6a09e9599a11249775928df9bb790b82471e4312b847a861ffb2c2ed/wrapt-2.4.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:09064c7be688c38c3ff125ce86bc26b69b5d78dd56062c3ddd9c814b2a25f1e1", size = 99615, upload-time = "2026-08-30T04:40:35.134Z" }, + { url = "https://files.pythonhosted.org/packages/b2/da/c1d3245abb911a42584f8f7e9781995bdc41345c7affba75cf7e376c85ac/wrapt-2.4.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:4f8ddff4bbb75916be36da5169b8b9d475b59a1bd24acdb7551bb2c71be9aaac", size = 100031, upload-time = "2026-08-30T04:40:36.641Z" }, + { url = "https://files.pythonhosted.org/packages/84/46/8ec4941d0abbb010df7caf0a34840ca0128177389843b0f5ef2f9ee48ac5/wrapt-2.4.0-cp314-cp314t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:e9f8017443595870aa31f46125553a5c55ce95a26a267b96261baee6ba566d83", size = 269389, upload-time = "2026-08-30T04:40:38.212Z" }, + { url = "https://files.pythonhosted.org/packages/14/b5/a0ae1b431cc1f49a545d32b8b678a5788c50583ecf0ecb85dc0c7f95b4f6/wrapt-2.4.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:328eb2d978ca3a6ae25f8d8fe560bf8f4bc9778b5932e7b142664eef05b92e8f", size = 281081, upload-time = "2026-08-30T04:40:40.045Z" }, + { url = "https://files.pythonhosted.org/packages/c7/24/dfaf53dd3bdb0703524a9367b48e2a64ea86433fcc854b5f14be6a8e0e39/wrapt-2.4.0-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:7a057d376d994da6bd1bbf955ecfda699aa7353826f98847f5605e1801abdfd4", size = 249637, upload-time = "2026-08-30T04:40:41.657Z" }, + { url = "https://files.pythonhosted.org/packages/3e/27/bdd82044d7503c2bfa78afcc89881f82a1b82b5d2013aabab853d339ce2a/wrapt-2.4.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:3367a5212212c9393e0d3ca6ae029b3a8fa40c5896e4a985d43fe8a4b8322f0d", size = 275322, upload-time = "2026-08-30T04:40:43.408Z" }, + { url = "https://files.pythonhosted.org/packages/c4/82/04f4228eb3fb348d660dd1ea7225e53665b1809df2273ff4861d4d33b741/wrapt-2.4.0-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:c4fca1e63af6675af3df7cdfcd5a0c878b5e655c7e48611ced9dc8d62183a11d", size = 247292, upload-time = "2026-08-30T04:40:45.457Z" }, + { url = "https://files.pythonhosted.org/packages/a2/20/67b2968fa9200458446c51b36a435adb6906083428b70fafb4caf92d4dc2/wrapt-2.4.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:694005fdc3002ade0f21641408c588028abde03c85961f3ba7727d8bead3ed6b", size = 264586, upload-time = "2026-08-30T04:40:47.079Z" }, + { url = "https://files.pythonhosted.org/packages/d7/fd/0db9ba03e08a7663f52455e95520c723f567bc037bffc6699950fcc456c4/wrapt-2.4.0-cp314-cp314t-win32.whl", hash = "sha256:332d9bad7e9b718974bb2a576504c4956f45b4a0fcd7b3bb7827279167550464", size = 93752, upload-time = "2026-08-30T04:40:48.81Z" }, + { url = "https://files.pythonhosted.org/packages/3f/87/ced171220935c696b157207385fa6be5675558a74655479f071d95a00f1d/wrapt-2.4.0-cp314-cp314t-win_amd64.whl", hash = "sha256:6d57264c9dfcf37d2bf0b0fbec68d0f6184fc5617267619ada04d03e8b0231f3", size = 99890, upload-time = "2026-08-30T04:40:50.407Z" }, + { url = "https://files.pythonhosted.org/packages/a3/af/4a10c9a6d3b7ae41f830978c28d33a59ceb29537bd6875d2abfe78db4b41/wrapt-2.4.0-cp314-cp314t-win_arm64.whl", hash = "sha256:f43af38a642c3d6062e9740d8f5cc0feb5dbe0da516702df892147393b8cb14d", size = 96033, upload-time = "2026-08-30T04:40:51.933Z" }, + { url = "https://files.pythonhosted.org/packages/a0/df/3a0b6225ab88bd47090df70391c059a3308057638f8fc0ae32e8ac9d1886/wrapt-2.4.0-cp315-cp315-macosx_10_15_x86_64.whl", hash = "sha256:430fde1a116df3ceb5c29035de1da6609b70e680d9b8ce3ee624422f3fe0978c", size = 96389, upload-time = "2026-08-30T04:40:53.555Z" }, + { url = "https://files.pythonhosted.org/packages/ea/6f/803b0d0e14de11781f0e938e6f7d6e29e79652139fe70d7513460357ac78/wrapt-2.4.0-cp315-cp315-macosx_11_0_arm64.whl", hash = "sha256:7d28f8f35a02d49f75f57fa4e755db4ba33f65841c0de64cd65b253916f5bf06", size = 96557, upload-time = "2026-08-30T04:40:55.033Z" }, + { url = "https://files.pythonhosted.org/packages/aa/e8/46571e1218d0494604a7aadc4c898c738c4b179052327ee1e57e278cebd6/wrapt-2.4.0-cp315-cp315-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:efd9a4be6785295e471f71efdf5682bd11d5b822b9665e6e1b4844917cf2f7ac", size = 229230, upload-time = "2026-08-30T04:40:56.703Z" }, + { url = "https://files.pythonhosted.org/packages/78/2e/0cab15fcaec56096a5734feace3620bc01edc885653be04bd756f84a6784/wrapt-2.4.0-cp315-cp315-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:75529a2fb569a671cf162f762c1b576f569f571b55ec7f3481258ca842ba507f", size = 229444, upload-time = "2026-08-30T04:40:58.51Z" }, + { url = "https://files.pythonhosted.org/packages/e7/9e/a92c049371a2675f98a0381ab2951f984866d1ba4de0e0771d6a31fdaa2b/wrapt-2.4.0-cp315-cp315-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:66e7512c0d324cc37bba1def2be1fc365cbb685d3aa393a8f6f4d2d00202881d", size = 212482, upload-time = "2026-08-30T04:41:00.224Z" }, + { url = "https://files.pythonhosted.org/packages/ee/3b/8b5b57d0ff24edcd3421dbaeb4e94c89be3616824e47708f4e13f25ae3d7/wrapt-2.4.0-cp315-cp315-musllinux_1_2_aarch64.whl", hash = "sha256:5f3bdfc35c83b562fcaebc0f24593045e5ed9f3b633adafd35222718a0ec38fa", size = 227017, upload-time = "2026-08-30T04:41:01.918Z" }, + { url = "https://files.pythonhosted.org/packages/0e/20/124b40bfd9585848db5a5aa6741d0c8dbf378dd995c6c2d95f090d9cf540/wrapt-2.4.0-cp315-cp315-musllinux_1_2_riscv64.whl", hash = "sha256:d5f45bead708e2c0014be5e98531ce7202916b098a208c7be83c6ceb0a2559fa", size = 210498, upload-time = "2026-08-30T04:41:03.617Z" }, + { url = "https://files.pythonhosted.org/packages/4b/bf/89db9d5a80a9f2af52b24bdfdb5392be80bc0f0fd39fc39d1aab72afd0bd/wrapt-2.4.0-cp315-cp315-musllinux_1_2_x86_64.whl", hash = "sha256:d294576fddac636589e4deccfe782e8f429da10f167c1985c4d51071de3672b7", size = 217046, upload-time = "2026-08-30T04:41:05.473Z" }, + { url = "https://files.pythonhosted.org/packages/3b/0b/021c9d6ce64c639894bffdaa7a895ddd4187abfefb2873ce55e536cd9d56/wrapt-2.4.0-cp315-cp315-win32.whl", hash = "sha256:0191d717dfbb8e519e7bfd4775e5b9bd57e359b3a09ab5db1ea47f6025b4d845", size = 91591, upload-time = "2026-08-30T04:41:07.086Z" }, + { url = "https://files.pythonhosted.org/packages/fd/d3/6ebd944041cea0ac4a108a4739510ed2dc891a3f3216e4f7bf0650f5b5a6/wrapt-2.4.0-cp315-cp315-win_amd64.whl", hash = "sha256:e8df31a126a0a247c1aa379e30873839de03912dea09ca360c680f3625d815df", size = 96517, upload-time = "2026-08-30T04:41:08.671Z" }, + { url = "https://files.pythonhosted.org/packages/96/84/7c5e52e450f80ba76fd0282dccf7c79cd004ebd8ccabd0903064d3d2c56e/wrapt-2.4.0-cp315-cp315-win_arm64.whl", hash = "sha256:e9e7e94472f0e3f1447caf27e1939eb384d0e87972a35a05f5c2e0968e9c01af", size = 93652, upload-time = "2026-08-30T04:41:10.258Z" }, + { url = "https://files.pythonhosted.org/packages/35/89/f08ff45d7646de29750932805cc3b1e86b6ac3128015b293ed45fa8efe86/wrapt-2.4.0-cp315-cp315t-macosx_10_15_x86_64.whl", hash = "sha256:8828369b7d3e93c547cc8ad931b5a57b4e8d174035c82762fb1091e7d05ac9f5", size = 99610, upload-time = "2026-08-30T04:41:11.933Z" }, + { url = "https://files.pythonhosted.org/packages/4b/c2/f9a3c40901a36c6bb7ecaff8e1e54af78fa7fa0b95a0e54d13d3a24c8a0a/wrapt-2.4.0-cp315-cp315t-macosx_11_0_arm64.whl", hash = "sha256:413e757dce7a43fcda8bb8441994b1127492ffac6a5803af777d44516df8c6e2", size = 100064, upload-time = "2026-08-30T04:41:13.492Z" }, + { url = "https://files.pythonhosted.org/packages/0b/e1/e2437f17f2a1ec292056e2fcafe1248269ebc39502f2ffe79424bf86f8a6/wrapt-2.4.0-cp315-cp315t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:75944792cf6b99262d649d55710bf5901f7013fbb212c7a1d736b97a20517607", size = 269421, upload-time = "2026-08-30T04:41:15.238Z" }, + { url = "https://files.pythonhosted.org/packages/c6/d0/c98d6548dc4c7d12ab9baa192234ca1a57e141afd283252b448faddbd9ef/wrapt-2.4.0-cp315-cp315t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:648d1d4f94e8a0a1656675c755f40d2f0ee5fe92c449ab45326f4ecc2738cbe8", size = 281452, upload-time = "2026-08-30T04:41:16.939Z" }, + { url = "https://files.pythonhosted.org/packages/a3/57/673168e00aa03725148ce621ed201b75df4e787a57acd48fecefd2725600/wrapt-2.4.0-cp315-cp315t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:a112a1bfdd2621e4344cb0a32dbaab80636b32dac1b055d03fbb2a67d806d1db", size = 250358, upload-time = "2026-08-30T04:41:18.716Z" }, + { url = "https://files.pythonhosted.org/packages/78/0b/f2e576de5bf53ef5b578470104ea93f33e273a704c825131bc1719fffc42/wrapt-2.4.0-cp315-cp315t-musllinux_1_2_aarch64.whl", hash = "sha256:0972cd025f4c86fa2d8abd953d9f875779935343af58b4ce019ff89573fc65bd", size = 275654, upload-time = "2026-08-30T04:41:20.418Z" }, + { url = "https://files.pythonhosted.org/packages/33/7f/9347b2e236346b1ba4cb28b82b205b8a377bb2da9417cb81bbe3d25816d7/wrapt-2.4.0-cp315-cp315t-musllinux_1_2_riscv64.whl", hash = "sha256:c246aaed719dcdb62eeb7b8d9306a6237777226ef3baad35919c4ae134c91ce7", size = 248662, upload-time = "2026-08-30T04:41:22.371Z" }, + { url = "https://files.pythonhosted.org/packages/a5/36/3b84d9e1ac8393bf2c94272760a2d361dc394ac30301e6d6dbd6583ade2d/wrapt-2.4.0-cp315-cp315t-musllinux_1_2_x86_64.whl", hash = "sha256:1656de3835f760781c9b974bce07d8c04edb9c9ad7ad67264aee69cd68a1db09", size = 264813, upload-time = "2026-08-30T04:41:24.116Z" }, + { url = "https://files.pythonhosted.org/packages/6e/a2/de7b1de1702667b4a048318e301e26887268c17b07c8b9797cea06b10aee/wrapt-2.4.0-cp315-cp315t-win32.whl", hash = "sha256:d8e6e1e5dc684dfce7c33fc8b67a08ba2af94f3a45cfc70d5c1d6a839d2caf97", size = 93753, upload-time = "2026-08-30T04:41:25.793Z" }, + { url = "https://files.pythonhosted.org/packages/09/50/4e7ef58c4eb058861ceddc0d1f94a6ed87f62e1cb27783c60b2897ef7e58/wrapt-2.4.0-cp315-cp315t-win_amd64.whl", hash = "sha256:85ed3c67fd39e8d9a36c224758cb6f2f4eb277d07ea677930caa0008c18ec002", size = 99888, upload-time = "2026-08-30T04:41:27.305Z" }, + { url = "https://files.pythonhosted.org/packages/68/64/d15740c763dd0ddea2338ad42e3bd4a84f8702e16083e7ff61674c504a13/wrapt-2.4.0-cp315-cp315t-win_arm64.whl", hash = "sha256:36b56a4fba13b34ed8ff307557325fff215de0a58b5dbaef2c50e4d8aa39dbd1", size = 96039, upload-time = "2026-08-30T04:41:29.062Z" }, + { url = "https://files.pythonhosted.org/packages/79/c8/fafe0002f572ced999c792cfe8b05d39269c63d8193d15d25bd828bcad7a/wrapt-2.4.0-py3-none-any.whl", hash = "sha256:18aabd9301d06026f5900538051773d6f87f65ae02cdc60de482df978513dc0a", size = 73713, upload-time = "2026-08-30T04:41:49.805Z" }, +] + [[package]] name = "zope-event" version = "6.1" From 73dcc51e294aa42c912d700bb50c3365c76ad458 Mon Sep 17 00:00:00 2001 From: Aaron Mildenstein Date: Thu, 3 Sep 2026 10:45:18 -0600 Subject: [PATCH 27/62] fix: CI database testing --- CHANGELOG.md | 3 ++- bin/test_databases.bat | 6 +----- bin/test_databases.sh | 4 ++-- tests/integration/test_migrate_engine.py | 7 +++---- 4 files changed, 8 insertions(+), 12 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 45cac5e..3b859d9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -33,7 +33,8 @@ SQLite does not scale under gunicorn/gevent. Operators can now run LogstashUI on ### Testing - Default `pytest` stays SQLite (no Docker, no extras). -- `bin/test_databases.sh` / `bin/test_databases.bat` start local Docker Postgres 16, MariaDB 11, and MySQL 8.0, run the suite on each engine, and run live dump/load tests. CI workflow `.github/workflows/test-databases.yml` calls the same script. +- `bin/test_databases.sh` / `bin/test_databases.bat` start local Docker Postgres 16, MariaDB 11, and MySQL 8.0 and run the full suite on each engine, then run `tests/integration/` via testcontainers. CI workflow `.github/workflows/test-databases.yml` calls the same script. +- Self-contained integration suite at `tests/integration/` uses `testcontainers` (postgres:16, mysql:8.0, mariadb:11 — no external Docker Compose). Parametrized over PostgreSQL and MySQL; MariaDB covered for `check_server_version` version-detection. Skips gracefully when Docker is unavailable. Covers DB config, migrations (clean, idempotent, no unapplied), ORM CRUD/JSON/uniqueness, and full SQLite → PG/MySQL/MariaDB `migrate-engine` round-trips. Run with `uv run pytest tests/integration/ -v --no-cov` after `uv sync --group dev --extra databases`. - Smoke compose is still SQLite (product CA / PUID unchanged). ### Kubernetes and database docs diff --git a/bin/test_databases.bat b/bin/test_databases.bat index 4272c16..f8d0dad 100644 --- a/bin/test_databases.bat +++ b/bin/test_databases.bat @@ -41,11 +41,7 @@ if errorlevel 1 ( goto :down ) -set LOGSTASHUI_LIVE_DB=1 -set LOGSTASHUI_LIVE_PG_PORT=55432 -set LOGSTASHUI_LIVE_MARIA_PORT=53306 -set LOGSTASHUI_LIVE_MYSQL_PORT=53307 -uv run pytest src\logstashui\LogstashUI\tests\test_migrate_live.py -v --no-cov +uv run pytest tests\integration\ -v --no-cov if errorlevel 1 set FAIL=1 :down diff --git a/bin/test_databases.sh b/bin/test_databases.sh index 798ceb9..c1b31c1 100755 --- a/bin/test_databases.sh +++ b/bin/test_databases.sh @@ -23,6 +23,6 @@ run_engine () { run_engine postgresql LOGSTASHUI_DB_ENGINE=postgresql LOGSTASHUI_DB_HOST=127.0.0.1 LOGSTASHUI_DB_PORT=55432 LOGSTASHUI_DB_NAME=logstashui LOGSTASHUI_DB_USER=logstashui LOGSTASHUI_DB_PASSWORD=logstashui run_engine mariadb LOGSTASHUI_DB_ENGINE=mysql LOGSTASHUI_DB_HOST=127.0.0.1 LOGSTASHUI_DB_PORT=53306 LOGSTASHUI_DB_NAME=logstashui LOGSTASHUI_DB_USER=root LOGSTASHUI_DB_PASSWORD=logstashui run_engine mysql LOGSTASHUI_DB_ENGINE=mysql LOGSTASHUI_DB_HOST=127.0.0.1 LOGSTASHUI_DB_PORT=53307 LOGSTASHUI_DB_NAME=logstashui LOGSTASHUI_DB_USER=root LOGSTASHUI_DB_PASSWORD=logstashui -echo "==> Live migrator tests" -env LOGSTASHUI_LIVE_DB=1 LOGSTASHUI_LIVE_PG_PORT=55432 LOGSTASHUI_LIVE_MARIA_PORT=53306 LOGSTASHUI_LIVE_MYSQL_PORT=53307 LOGSTASHUI_LIVE_DB_USER=root LOGSTASHUI_LIVE_DB_PASSWORD=logstashui LOGSTASHUI_LIVE_PG_USER=logstashui uv run pytest src/logstashui/LogstashUI/tests/test_migrate_live.py -v --no-cov +echo "==> Integration suite (testcontainers)" +uv run pytest tests/integration/ -v --no-cov if [[ "$KEEP" -eq 0 ]]; then "${COMPOSE[@]}" down -v; fi diff --git a/tests/integration/test_migrate_engine.py b/tests/integration/test_migrate_engine.py index 0c59a74..fab32f8 100644 --- a/tests/integration/test_migrate_engine.py +++ b/tests/integration/test_migrate_engine.py @@ -6,8 +6,7 @@ Integration tests for cmd_migrate_engine (SQLite → PostgreSQL / MySQL / MariaDB). Each test uses a unique throwaway database in the session-scoped container -to prevent cross-test contamination. The pattern follows test_migrate_live.py -but testcontainers replaces the external Docker Compose dependency. +to prevent cross-test contamination. """ import json @@ -31,7 +30,7 @@ # --------------------------------------------------------------------------- -# Subprocess helper (mirrors test_migrate_live._run_python) +# Subprocess helper # --------------------------------------------------------------------------- def _run_python(code: str, extra_env: dict[str, str]) -> str: @@ -55,7 +54,7 @@ def _run_python(code: str, extra_env: dict[str, str]) -> str: # --------------------------------------------------------------------------- -# Inline scripts (seed and count — identical to test_migrate_live) +# Inline scripts # --------------------------------------------------------------------------- _SEED = """ From d5445813cd798e7127f105e8915a0a8008851306 Mon Sep 17 00:00:00 2001 From: Aaron Mildenstein Date: Thu, 3 Sep 2026 11:06:29 -0600 Subject: [PATCH 28/62] fix: FULLY honor LOGSTASHUI_TLS env var There was a hidden env var in `settings.py` that was missed with regards to disabling TLS. If DEBUG=false, it would set SECURE_SSL_REDIRECT to true. Now it sets SECURE_SSL_REDIRECT to the value of LOGSTASHUI_TLS, which defaults to true if unset. --- CHANGELOG.md | 4 ++++ src/logstashui/LogstashUI/settings.py | 12 +++++++----- 2 files changed, 11 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3b859d9..7a6ec27 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -44,6 +44,10 @@ SQLite does not scale under gunicorn/gevent. Operators can now run LogstashUI on - Example manifests under `docs/docs/logstashui/kubernetes/examples/{sqlite,postgresql,mysql}/`. - Database subsection: engines, every `LOGSTASHUI_DB_*` default, offline dump/load, BETA `migrate-engine`, CREATE DATABASE scripts (`utf8mb4_bin` for MySQL/MariaDB), schema snapshots from 0.5.2 `migrate`. +### Fixes + +- `LOGSTASHUI_TLS=false` now suppresses the Django-level HTTP→HTTPS redirect (`SECURE_SSL_REDIRECT`) in addition to disabling the Gunicorn TLS certificate. Previously, running the container with `-e LOGSTASHUI_TLS=false` still returned a `301` because `SECURE_SSL_REDIRECT` was gated on `DEBUG` only. Both knobs are now independent. + ### Agent version display - ConnectionManager and Policy editor Agents tab show a cyan **LS X.Y.Z** pill for the Logstash version the agent reported (`logstash_version_resolved`, else Logstash API version). Hidden until known. diff --git a/src/logstashui/LogstashUI/settings.py b/src/logstashui/LogstashUI/settings.py index 06334cc..e4a850a 100644 --- a/src/logstashui/LogstashUI/settings.py +++ b/src/logstashui/LogstashUI/settings.py @@ -277,20 +277,22 @@ def _get_version(): # Security Headers # These settings protect against common web vulnerabilities # Only enforce in production (when DEBUG=False) +_tls_env = os.environ.get("LOGSTASHUI_TLS", "true") +TLS_ENABLED = _tls_env.strip().lower() not in ("0", "false", "no", "off") + if not DEBUG: # Ensure cookies are only sent over HTTPS SESSION_COOKIE_SECURE = True CSRF_COOKIE_SECURE = True - + # HTTP Strict Transport Security (HSTS) # Tells browsers to only access the site via HTTPS for the next year SECURE_HSTS_SECONDS = 31536000 # 1 year SECURE_HSTS_INCLUDE_SUBDOMAINS = True SECURE_HSTS_PRELOAD = True - - # Redirect all HTTP requests to HTTPS at Django level - # Note: nginx already does this, but this adds defense in depth - SECURE_SSL_REDIRECT = True + + # Only redirect when TLS is actually on — LOGSTASHUI_TLS=false must suppress this + SECURE_SSL_REDIRECT = TLS_ENABLED # Prevent the site from being embedded in iframes (clickjacking protection) X_FRAME_OPTIONS = 'DENY' From c396829f4b90077867d5c5fab36e3034e8a6f478 Mon Sep 17 00:00:00 2001 From: Aaron Mildenstein Date: Thu, 3 Sep 2026 12:21:48 -0600 Subject: [PATCH 29/62] Migrated all tests to be in tests/ --- CHANGELOG.md | 5 +- bin/test_databases.bat | 2 +- bin/test_databases.sh | 4 +- pyproject.toml | 2 +- src/logstashui/AI/tests.py | 7 - src/logstashui/Common/test_resources.py | 67 - src/logstashui/Common/tests/__init__.py | 0 .../components/ls-repo-apache2.json | 162 -- .../components/test-asa-new.json | 1846 ------------ .../conversion_data/components/test-asa.json | 1846 ------------ .../components/test-boolean-numeric.json | 134 - .../test-comments-brace-in-comment.json | 72 - .../components/test-comments-mixed.json | 154 - .../test-comments-plugin-inline.json | 111 - .../test-comments-section-opener.json | 100 - .../test-comments-standalone-in-plugin.json | 61 - .../components/test-complex2.json | 952 ------- .../components/test-complex3.json | 1497 ---------- .../components/test-data-types.json | 52 - .../components/test-datatypes.json | 52 - .../components/test-devopsschool-1.json | 51 - .../components/test-devopsschool-2.json | 39 - .../components/test-devopsschool-4.json | 62 - .../components/test-devopsschool-5.json | 48 - .../components/test-elasticdocs-apache.json | 86 - .../test-elasticdocs-configuring_filters.json | 60 - .../components/test-elasticdocs-syslog.json | 92 - .../components/test-es-input.json | 48 - .../components/test-ls-repo-mysql.json | 156 -- .../components/test-ls-repo-nginx.json | 156 -- .../components/test-ls-repo-system.json | 135 - .../test-multiline-ruby-with-hash.json | 60 - .../test-nested-conditionals-comments.json | 266 -- .../components/test-regex-conditions.json | 228 -- .../components/test-sample-nginx.json | 183 -- .../components/test-snmp-v0.2.json | 212 -- .../components/test-string-escaping.json | 60 - .../components/test-twitter.json | 89 - .../components/test_complex1.json | 952 ------- .../test_elasticdocs-conditional.json | 116 - .../components/text-complex4.json | 386 --- .../components/text-complex5.json | 94 - .../components/text-complex6.json | 469 ---- .../components/text-complex7.json | 558 ---- .../components/text-ls-repo-nginx-error.json | 183 -- .../pipelines/ls-repo-apache2.conf | 69 - .../pipelines/test-asa-new.conf | 897 ------ .../conversion_data/pipelines/test-asa.conf | 897 ------ .../pipelines/test-boolean-numeric.conf | 58 - .../test-comments-brace-in-comment.conf | 39 - .../pipelines/test-comments-mixed.conf | 57 - .../test-comments-plugin-inline.conf | 46 - .../test-comments-section-opener.conf | 29 - .../test-comments-standalone-in-plugin.conf | 29 - .../pipelines/test-complex2.conf | 329 --- .../pipelines/test-complex3.conf | 652 ----- .../pipelines/test-data-types.conf | 33 - .../pipelines/test-datatypes.conf | 33 - .../pipelines/test-devopsschool-1.conf | 21 - .../pipelines/test-devopsschool-2.conf | 17 - .../pipelines/test-devopsschool-4.conf | 25 - .../pipelines/test-devopsschool-5.conf | 20 - .../pipelines/test-elasticdocs-apache.conf | 31 - .../test-elasticdocs-configuring_filters.conf | 22 - .../pipelines/test-elasticdocs-syslog.conf | 31 - .../pipelines/test-es-input.conf | 26 - .../pipelines/test-ls-repo-mysql.conf | 69 - .../pipelines/test-ls-repo-nginx.conf | 64 - .../pipelines/test-ls-repo-system.conf | 61 - .../test-multiline-ruby-with-hash.conf | 49 - .../test-nested-conditionals-comments.conf | 71 - .../pipelines/test-regex-conditions.conf | 62 - .../pipelines/test-sample-nginx.conf | 67 - .../pipelines/test-snmp-v0.2.conf | 213 -- .../pipelines/test-string-escaping.conf | 33 - .../pipelines/test-twitter.conf | 41 - .../pipelines/test_complex1.conf | 329 --- .../test_elasticdocs-conditional.conf | 44 - .../pipelines/text-complex4.conf | 123 - .../pipelines/text-complex5.conf | 40 - .../pipelines/text-complex6.conf | 192 -- .../pipelines/text-complex7.conf | 222 -- .../pipelines/text-ls-repo-nginx-error.conf | 67 - .../tests/test_components_to_pipeline.py | 57 - .../Common/tests/test_context_processors.py | 224 -- .../Common/tests/test_decorators.py | 269 -- .../Common/tests/test_elastic_utils.py | 654 ----- .../Common/tests/test_encryption.py | 267 -- .../Common/tests/test_error_handlers.py | 263 -- .../Common/tests/test_formatters.py | 407 --- .../tests/test_logstash_config_parse.py | 542 ---- .../Common/tests/test_logstash_utils.py | 163 -- .../Common/tests/test_middleware.py | 242 -- .../tests/test_pipeline_to_components.py | 51 - .../Common/tests/test_product_ca.py | 334 --- .../Common/tests/test_validators.py | 175 -- src/logstashui/LogstashUI/tests/test_cli.py | 230 -- .../LogstashUI/tests/test_config.py | 61 - .../LogstashUI/tests/test_logging_config.py | 50 - src/logstashui/LogstashUI/tests/test_paths.py | 91 - src/logstashui/Management/tests/__init__.py | 0 src/logstashui/Management/tests/test_views.py | 1067 ------- src/logstashui/Monitoring/tests/__init__.py | 0 src/logstashui/Monitoring/tests/test_views.py | 882 ------ .../PipelineManager/tests/__init__.py | 0 .../PipelineManager/tests/test_agent_api.py | 1109 -------- .../PipelineManager/tests/test_agent_modes.py | 731 ----- .../tests/test_agent_policies.py | 1004 ------- .../tests/test_agent_versions.py | 128 - .../tests/test_connections_crud.py | 603 ---- .../tests/test_editor_views.py | 751 ----- .../tests/test_elasticsearch_queries.py | 491 ---- .../tests/test_manager_views.py | 792 ------ .../tests/test_pipeline_editor.py | 548 ---- .../tests/test_pipelines_crud.py | 775 ----- .../tests/test_policies_crud.py | 1096 -------- .../tests/test_sim_keystore.py | 157 -- .../PipelineManager/tests/test_simulation.py | 818 ------ src/logstashui/SNMP/tests/__init__.py | 0 src/logstashui/SNMP/tests/test_commands.py | 776 ----- .../SNMP/tests/test_inline_grounding.py | 66 - src/logstashui/SNMP/tests/test_models.py | 498 ---- src/logstashui/SNMP/tests/test_network_map.py | 618 ---- src/logstashui/SNMP/tests/test_overview.py | 482 ---- src/logstashui/SNMP/tests/test_snmp_crud.py | 2485 ----------------- .../SNMP/tests/test_snmp_grounding.py | 263 -- .../SNMP/tests/test_snmp_normalizers.py | 545 ---- .../tests/test_snmp_pipeline_generator.py | 789 ------ src/logstashui/SNMP/tests/test_snmp_test.py | 920 ------ src/logstashui/SNMP/tests/test_views.py | 1098 -------- src/logstashui/Site/tests/__init__.py | 0 src/logstashui/Site/tests/test_views.py | 381 --- src/logstashui/Utilities/tests/__init__.py | 1 - .../Utilities/tests/test_grok_patterns.py | 138 - src/logstashui/Utilities/tests/test_views.py | 672 ----- tests/integration/__init__.py | 0 tests/integration/conftest.py | 244 -- tests/integration/test_db_config.py | 166 -- tests/integration/test_migrate_engine.py | 292 -- tests/integration/test_migrations.py | 101 - tests/integration/test_orm.py | 389 --- tests/unit/__init__.py | 0 tests/unit/test_database.py | 245 -- tests/unit/test_migrate_engine.py | 111 - 144 files changed, 7 insertions(+), 43328 deletions(-) delete mode 100644 src/logstashui/AI/tests.py delete mode 100644 src/logstashui/Common/test_resources.py delete mode 100644 src/logstashui/Common/tests/__init__.py delete mode 100644 src/logstashui/Common/tests/conversion_data/components/ls-repo-apache2.json delete mode 100644 src/logstashui/Common/tests/conversion_data/components/test-asa-new.json delete mode 100644 src/logstashui/Common/tests/conversion_data/components/test-asa.json delete mode 100644 src/logstashui/Common/tests/conversion_data/components/test-boolean-numeric.json delete mode 100644 src/logstashui/Common/tests/conversion_data/components/test-comments-brace-in-comment.json delete mode 100644 src/logstashui/Common/tests/conversion_data/components/test-comments-mixed.json delete mode 100644 src/logstashui/Common/tests/conversion_data/components/test-comments-plugin-inline.json delete mode 100644 src/logstashui/Common/tests/conversion_data/components/test-comments-section-opener.json delete mode 100644 src/logstashui/Common/tests/conversion_data/components/test-comments-standalone-in-plugin.json delete mode 100644 src/logstashui/Common/tests/conversion_data/components/test-complex2.json delete mode 100644 src/logstashui/Common/tests/conversion_data/components/test-complex3.json delete mode 100644 src/logstashui/Common/tests/conversion_data/components/test-data-types.json delete mode 100644 src/logstashui/Common/tests/conversion_data/components/test-datatypes.json delete mode 100644 src/logstashui/Common/tests/conversion_data/components/test-devopsschool-1.json delete mode 100644 src/logstashui/Common/tests/conversion_data/components/test-devopsschool-2.json delete mode 100644 src/logstashui/Common/tests/conversion_data/components/test-devopsschool-4.json delete mode 100644 src/logstashui/Common/tests/conversion_data/components/test-devopsschool-5.json delete mode 100644 src/logstashui/Common/tests/conversion_data/components/test-elasticdocs-apache.json delete mode 100644 src/logstashui/Common/tests/conversion_data/components/test-elasticdocs-configuring_filters.json delete mode 100644 src/logstashui/Common/tests/conversion_data/components/test-elasticdocs-syslog.json delete mode 100644 src/logstashui/Common/tests/conversion_data/components/test-es-input.json delete mode 100644 src/logstashui/Common/tests/conversion_data/components/test-ls-repo-mysql.json delete mode 100644 src/logstashui/Common/tests/conversion_data/components/test-ls-repo-nginx.json delete mode 100644 src/logstashui/Common/tests/conversion_data/components/test-ls-repo-system.json delete mode 100644 src/logstashui/Common/tests/conversion_data/components/test-multiline-ruby-with-hash.json delete mode 100644 src/logstashui/Common/tests/conversion_data/components/test-nested-conditionals-comments.json delete mode 100644 src/logstashui/Common/tests/conversion_data/components/test-regex-conditions.json delete mode 100644 src/logstashui/Common/tests/conversion_data/components/test-sample-nginx.json delete mode 100644 src/logstashui/Common/tests/conversion_data/components/test-snmp-v0.2.json delete mode 100644 src/logstashui/Common/tests/conversion_data/components/test-string-escaping.json delete mode 100644 src/logstashui/Common/tests/conversion_data/components/test-twitter.json delete mode 100644 src/logstashui/Common/tests/conversion_data/components/test_complex1.json delete mode 100644 src/logstashui/Common/tests/conversion_data/components/test_elasticdocs-conditional.json delete mode 100644 src/logstashui/Common/tests/conversion_data/components/text-complex4.json delete mode 100644 src/logstashui/Common/tests/conversion_data/components/text-complex5.json delete mode 100644 src/logstashui/Common/tests/conversion_data/components/text-complex6.json delete mode 100644 src/logstashui/Common/tests/conversion_data/components/text-complex7.json delete mode 100644 src/logstashui/Common/tests/conversion_data/components/text-ls-repo-nginx-error.json delete mode 100644 src/logstashui/Common/tests/conversion_data/pipelines/ls-repo-apache2.conf delete mode 100644 src/logstashui/Common/tests/conversion_data/pipelines/test-asa-new.conf delete mode 100644 src/logstashui/Common/tests/conversion_data/pipelines/test-asa.conf delete mode 100644 src/logstashui/Common/tests/conversion_data/pipelines/test-boolean-numeric.conf delete mode 100644 src/logstashui/Common/tests/conversion_data/pipelines/test-comments-brace-in-comment.conf delete mode 100644 src/logstashui/Common/tests/conversion_data/pipelines/test-comments-mixed.conf delete mode 100644 src/logstashui/Common/tests/conversion_data/pipelines/test-comments-plugin-inline.conf delete mode 100644 src/logstashui/Common/tests/conversion_data/pipelines/test-comments-section-opener.conf delete mode 100644 src/logstashui/Common/tests/conversion_data/pipelines/test-comments-standalone-in-plugin.conf delete mode 100644 src/logstashui/Common/tests/conversion_data/pipelines/test-complex2.conf delete mode 100644 src/logstashui/Common/tests/conversion_data/pipelines/test-complex3.conf delete mode 100644 src/logstashui/Common/tests/conversion_data/pipelines/test-data-types.conf delete mode 100644 src/logstashui/Common/tests/conversion_data/pipelines/test-datatypes.conf delete mode 100644 src/logstashui/Common/tests/conversion_data/pipelines/test-devopsschool-1.conf delete mode 100644 src/logstashui/Common/tests/conversion_data/pipelines/test-devopsschool-2.conf delete mode 100644 src/logstashui/Common/tests/conversion_data/pipelines/test-devopsschool-4.conf delete mode 100644 src/logstashui/Common/tests/conversion_data/pipelines/test-devopsschool-5.conf delete mode 100644 src/logstashui/Common/tests/conversion_data/pipelines/test-elasticdocs-apache.conf delete mode 100644 src/logstashui/Common/tests/conversion_data/pipelines/test-elasticdocs-configuring_filters.conf delete mode 100644 src/logstashui/Common/tests/conversion_data/pipelines/test-elasticdocs-syslog.conf delete mode 100644 src/logstashui/Common/tests/conversion_data/pipelines/test-es-input.conf delete mode 100644 src/logstashui/Common/tests/conversion_data/pipelines/test-ls-repo-mysql.conf delete mode 100644 src/logstashui/Common/tests/conversion_data/pipelines/test-ls-repo-nginx.conf delete mode 100644 src/logstashui/Common/tests/conversion_data/pipelines/test-ls-repo-system.conf delete mode 100644 src/logstashui/Common/tests/conversion_data/pipelines/test-multiline-ruby-with-hash.conf delete mode 100644 src/logstashui/Common/tests/conversion_data/pipelines/test-nested-conditionals-comments.conf delete mode 100644 src/logstashui/Common/tests/conversion_data/pipelines/test-regex-conditions.conf delete mode 100644 src/logstashui/Common/tests/conversion_data/pipelines/test-sample-nginx.conf delete mode 100644 src/logstashui/Common/tests/conversion_data/pipelines/test-snmp-v0.2.conf delete mode 100644 src/logstashui/Common/tests/conversion_data/pipelines/test-string-escaping.conf delete mode 100644 src/logstashui/Common/tests/conversion_data/pipelines/test-twitter.conf delete mode 100644 src/logstashui/Common/tests/conversion_data/pipelines/test_complex1.conf delete mode 100644 src/logstashui/Common/tests/conversion_data/pipelines/test_elasticdocs-conditional.conf delete mode 100644 src/logstashui/Common/tests/conversion_data/pipelines/text-complex4.conf delete mode 100644 src/logstashui/Common/tests/conversion_data/pipelines/text-complex5.conf delete mode 100644 src/logstashui/Common/tests/conversion_data/pipelines/text-complex6.conf delete mode 100644 src/logstashui/Common/tests/conversion_data/pipelines/text-complex7.conf delete mode 100644 src/logstashui/Common/tests/conversion_data/pipelines/text-ls-repo-nginx-error.conf delete mode 100644 src/logstashui/Common/tests/test_components_to_pipeline.py delete mode 100644 src/logstashui/Common/tests/test_context_processors.py delete mode 100644 src/logstashui/Common/tests/test_decorators.py delete mode 100644 src/logstashui/Common/tests/test_elastic_utils.py delete mode 100644 src/logstashui/Common/tests/test_encryption.py delete mode 100644 src/logstashui/Common/tests/test_error_handlers.py delete mode 100644 src/logstashui/Common/tests/test_formatters.py delete mode 100644 src/logstashui/Common/tests/test_logstash_config_parse.py delete mode 100644 src/logstashui/Common/tests/test_logstash_utils.py delete mode 100644 src/logstashui/Common/tests/test_middleware.py delete mode 100644 src/logstashui/Common/tests/test_pipeline_to_components.py delete mode 100644 src/logstashui/Common/tests/test_product_ca.py delete mode 100644 src/logstashui/Common/tests/test_validators.py delete mode 100644 src/logstashui/LogstashUI/tests/test_cli.py delete mode 100644 src/logstashui/LogstashUI/tests/test_config.py delete mode 100644 src/logstashui/LogstashUI/tests/test_logging_config.py delete mode 100644 src/logstashui/LogstashUI/tests/test_paths.py delete mode 100644 src/logstashui/Management/tests/__init__.py delete mode 100644 src/logstashui/Management/tests/test_views.py delete mode 100644 src/logstashui/Monitoring/tests/__init__.py delete mode 100644 src/logstashui/Monitoring/tests/test_views.py delete mode 100644 src/logstashui/PipelineManager/tests/__init__.py delete mode 100644 src/logstashui/PipelineManager/tests/test_agent_api.py delete mode 100644 src/logstashui/PipelineManager/tests/test_agent_modes.py delete mode 100644 src/logstashui/PipelineManager/tests/test_agent_policies.py delete mode 100644 src/logstashui/PipelineManager/tests/test_agent_versions.py delete mode 100644 src/logstashui/PipelineManager/tests/test_connections_crud.py delete mode 100644 src/logstashui/PipelineManager/tests/test_editor_views.py delete mode 100644 src/logstashui/PipelineManager/tests/test_elasticsearch_queries.py delete mode 100644 src/logstashui/PipelineManager/tests/test_manager_views.py delete mode 100644 src/logstashui/PipelineManager/tests/test_pipeline_editor.py delete mode 100644 src/logstashui/PipelineManager/tests/test_pipelines_crud.py delete mode 100644 src/logstashui/PipelineManager/tests/test_policies_crud.py delete mode 100644 src/logstashui/PipelineManager/tests/test_sim_keystore.py delete mode 100644 src/logstashui/PipelineManager/tests/test_simulation.py delete mode 100644 src/logstashui/SNMP/tests/__init__.py delete mode 100644 src/logstashui/SNMP/tests/test_commands.py delete mode 100644 src/logstashui/SNMP/tests/test_inline_grounding.py delete mode 100644 src/logstashui/SNMP/tests/test_models.py delete mode 100644 src/logstashui/SNMP/tests/test_network_map.py delete mode 100644 src/logstashui/SNMP/tests/test_overview.py delete mode 100644 src/logstashui/SNMP/tests/test_snmp_crud.py delete mode 100644 src/logstashui/SNMP/tests/test_snmp_grounding.py delete mode 100644 src/logstashui/SNMP/tests/test_snmp_normalizers.py delete mode 100644 src/logstashui/SNMP/tests/test_snmp_pipeline_generator.py delete mode 100644 src/logstashui/SNMP/tests/test_snmp_test.py delete mode 100644 src/logstashui/SNMP/tests/test_views.py delete mode 100644 src/logstashui/Site/tests/__init__.py delete mode 100644 src/logstashui/Site/tests/test_views.py delete mode 100644 src/logstashui/Utilities/tests/__init__.py delete mode 100644 src/logstashui/Utilities/tests/test_grok_patterns.py delete mode 100644 src/logstashui/Utilities/tests/test_views.py delete mode 100644 tests/integration/__init__.py delete mode 100644 tests/integration/conftest.py delete mode 100644 tests/integration/test_db_config.py delete mode 100644 tests/integration/test_migrate_engine.py delete mode 100644 tests/integration/test_migrations.py delete mode 100644 tests/integration/test_orm.py delete mode 100644 tests/unit/__init__.py delete mode 100644 tests/unit/test_database.py delete mode 100644 tests/unit/test_migrate_engine.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 7a6ec27..819badc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -33,8 +33,9 @@ SQLite does not scale under gunicorn/gevent. Operators can now run LogstashUI on ### Testing - Default `pytest` stays SQLite (no Docker, no extras). -- `bin/test_databases.sh` / `bin/test_databases.bat` start local Docker Postgres 16, MariaDB 11, and MySQL 8.0 and run the full suite on each engine, then run `tests/integration/` via testcontainers. CI workflow `.github/workflows/test-databases.yml` calls the same script. -- Self-contained integration suite at `tests/integration/` uses `testcontainers` (postgres:16, mysql:8.0, mariadb:11 — no external Docker Compose). Parametrized over PostgreSQL and MySQL; MariaDB covered for `check_server_version` version-detection. Skips gracefully when Docker is unavailable. Covers DB config, migrations (clean, idempotent, no unapplied), ORM CRUD/JSON/uniqueness, and full SQLite → PG/MySQL/MariaDB `migrate-engine` round-trips. Run with `uv run pytest tests/integration/ -v --no-cov` after `uv sync --group dev --extra databases`. +- `bin/test_databases.sh` / `bin/test_databases.bat` start local Docker Postgres 16, MariaDB 11, and MySQL 8.0 and run the full suite on each engine, then run `tests/Database/` via testcontainers. CI workflow `.github/workflows/test-databases.yml` calls the same script. +- Self-contained database test suite at `tests/Database/integration/` uses `testcontainers` (postgres:16, mysql:8.0, mariadb:11 — no external Docker Compose). Parametrized over PostgreSQL and MySQL; MariaDB covered for `check_server_version` version-detection. Skips gracefully when Docker is unavailable. Covers DB config, migrations (clean, idempotent, no unapplied), ORM CRUD/JSON/uniqueness, and full SQLite → PG/MySQL/MariaDB `migrate-engine` round-trips. Run with `uv run pytest tests/Database/ -v --no-cov` after `uv sync --group dev --extra databases`. +- All tests migrated from `src/logstashui//tests/` to a dedicated `tests/` tree at the project root. Layout: `tests//unit/` per Django app, `tests/Database/unit|integration/` for database and migration tests. Shared fixtures moved from `Common/test_resources.py` to `tests/conftest.py` (auto-discovered by pytest). `testpaths` trimmed to `["tests"]`; `pythonpath = ["src/logstashui"]` unchanged so app imports still resolve. Run any app in isolation: `uv run pytest tests/SNMP -v`. - Smoke compose is still SQLite (product CA / PUID unchanged). ### Kubernetes and database docs diff --git a/bin/test_databases.bat b/bin/test_databases.bat index f8d0dad..4e4281e 100644 --- a/bin/test_databases.bat +++ b/bin/test_databases.bat @@ -41,7 +41,7 @@ if errorlevel 1 ( goto :down ) -uv run pytest tests\integration\ -v --no-cov +uv run pytest tests\Database\ -v --no-cov if errorlevel 1 set FAIL=1 :down diff --git a/bin/test_databases.sh b/bin/test_databases.sh index c1b31c1..63a0825 100755 --- a/bin/test_databases.sh +++ b/bin/test_databases.sh @@ -23,6 +23,6 @@ run_engine () { run_engine postgresql LOGSTASHUI_DB_ENGINE=postgresql LOGSTASHUI_DB_HOST=127.0.0.1 LOGSTASHUI_DB_PORT=55432 LOGSTASHUI_DB_NAME=logstashui LOGSTASHUI_DB_USER=logstashui LOGSTASHUI_DB_PASSWORD=logstashui run_engine mariadb LOGSTASHUI_DB_ENGINE=mysql LOGSTASHUI_DB_HOST=127.0.0.1 LOGSTASHUI_DB_PORT=53306 LOGSTASHUI_DB_NAME=logstashui LOGSTASHUI_DB_USER=root LOGSTASHUI_DB_PASSWORD=logstashui run_engine mysql LOGSTASHUI_DB_ENGINE=mysql LOGSTASHUI_DB_HOST=127.0.0.1 LOGSTASHUI_DB_PORT=53307 LOGSTASHUI_DB_NAME=logstashui LOGSTASHUI_DB_USER=root LOGSTASHUI_DB_PASSWORD=logstashui -echo "==> Integration suite (testcontainers)" -uv run pytest tests/integration/ -v --no-cov +echo "==> Database test suite (testcontainers)" +uv run pytest tests/Database/ -v --no-cov if [[ "$KEEP" -eq 0 ]]; then "${COMPOSE[@]}" down -v; fi diff --git a/pyproject.toml b/pyproject.toml index c54754a..26f98ed 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -107,7 +107,7 @@ dev = [ DJANGO_SETTINGS_MODULE = "LogstashUI.settings" python_files = ["tests.py", "test_*.py", "*_tests.py"] pythonpath = ["src/logstashui"] -testpaths = ["src/logstashui", "tests"] +testpaths = ["tests"] markers = [ "integration: marks tests that require running Docker containers", ] diff --git a/src/logstashui/AI/tests.py b/src/logstashui/AI/tests.py deleted file mode 100644 index 9b26a3a..0000000 --- a/src/logstashui/AI/tests.py +++ /dev/null @@ -1,7 +0,0 @@ -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. - -from django.test import TestCase - -# Create your tests here. diff --git a/src/logstashui/Common/test_resources.py b/src/logstashui/Common/test_resources.py deleted file mode 100644 index 5a0cf54..0000000 --- a/src/logstashui/Common/test_resources.py +++ /dev/null @@ -1,67 +0,0 @@ -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. - -from django.contrib.auth.models import User -from django.test import Client, RequestFactory -from PipelineManager.models import Connection - -import pytest - - -##### Fixtures ##### -@pytest.fixture -def request_factory(): - """Django RequestFactory for creating mock requests""" - return RequestFactory() - - -@pytest.fixture -def authenticated_client(client, test_user): - """Client with authenticated user""" - client.login(username='testuser', password='testpass123') - return client - - -@pytest.fixture -def client(): - """Django test client""" - return Client() - - -@pytest.fixture -def test_user(db): - """Create a test user with admin profile""" - from Management.models import UserProfile - - user = User.objects.create_user( - username='testuser', - password='testpass123', - email='test@example.com' - ) - user.is_superuser = True - user.is_staff = True - user.save() - - # Create admin profile (use get_or_create to avoid UNIQUE constraint errors) - UserProfile.objects.get_or_create( - user=user, - defaults={'role': 'admin'} - ) - - return user - - - - -@pytest.fixture -def test_connection(db): - """Create a test connection""" - connection = Connection.objects.create( - name='Test Connection', - connection_type='CENTRALIZED', - host='https://localhost:9200', - username='elastic', - password='changeme' - ) - return connection diff --git a/src/logstashui/Common/tests/__init__.py b/src/logstashui/Common/tests/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/src/logstashui/Common/tests/conversion_data/components/ls-repo-apache2.json b/src/logstashui/Common/tests/conversion_data/components/ls-repo-apache2.json deleted file mode 100644 index 439a46f..0000000 --- a/src/logstashui/Common/tests/conversion_data/components/ls-repo-apache2.json +++ /dev/null @@ -1,162 +0,0 @@ -{ - "input": [ - { - "id": "input_beats_0", - "type": "input", - "plugin": "beats", - "config": { - "port": 5044, - "host": "0.0.0.0" - }, - "comments": [] - } - ], - "filter": [ - { - "id": "filter_if_1", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[fileset][module] == \"apache2\"", - "plugins": [ - { - "id": "filter_if_2", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[fileset][name] == \"access\"", - "plugins": [ - { - "id": "filter_grok_3", - "type": "filter", - "plugin": "grok", - "config": { - "match": { - "message": [ - "%{IPORHOST:[apache2][access][remote_ip]} - %{DATA:[apache2][access][user_name]} \\[%{HTTPDATE:[apache2][access][time]}\\] \"%{WORD:[apache2][access][method]} %{DATA:[apache2][access][url]} HTTP/%{NUMBER:[apache2][access][http_version]}\" %{NUMBER:[apache2][access][response_code]} %{NUMBER:[apache2][access][body_sent][bytes]}( \"%{DATA:[apache2][access][referrer]}\")?( \"%{DATA:[apache2][access][agent]}\")?", - "%{IPORHOST:[apache2][access][remote_ip]} - %{DATA:[apache2][access][user_name]} \\[%{HTTPDATE:[apache2][access][time]}\\] \"-\" %{NUMBER:[apache2][access][response_code]} -" - ] - }, - "remove_field": "message" - }, - "comments": [] - }, - { - "id": "filter_mutate_4", - "type": "filter", - "plugin": "mutate", - "config": { - "add_field": { - "read_timestamp": "%{@timestamp}" - } - }, - "comments": [] - }, - { - "id": "filter_date_5", - "type": "filter", - "plugin": "date", - "config": { - "match": [ - "[apache2][access][time]", - "dd/MMM/YYYY:H:m:s Z" - ], - "remove_field": "[apache2][access][time]" - }, - "comments": [] - }, - { - "id": "filter_useragent_6", - "type": "filter", - "plugin": "useragent", - "config": { - "source": "[apache2][access][agent]", - "target": "[apache2][access][user_agent]", - "remove_field": "[apache2][access][agent]" - }, - "comments": [] - }, - { - "id": "filter_geoip_7", - "type": "filter", - "plugin": "geoip", - "config": { - "source": "[apache2][access][remote_ip]", - "target": "[apache2][access][geoip]" - }, - "comments": [] - } - ], - "else_ifs": [ - { - "condition": "[fileset][name] == \"error\"", - "plugins": [ - { - "id": "filter_grok_8", - "type": "filter", - "plugin": "grok", - "config": { - "match": { - "message": [ - "\\[%{APACHE_TIME:[apache2][error][timestamp]}\\] \\[%{LOGLEVEL:[apache2][error][level]}\\]( \\[client %{IPORHOST:[apache2][error][client]}\\])? %{GREEDYDATA:[apache2][error][message]}", - "\\[%{APACHE_TIME:[apache2][error][timestamp]}\\] \\[%{DATA:[apache2][error][module]}:%{LOGLEVEL:[apache2][error][level]}\\] \\[pid %{NUMBER:[apache2][error][pid]}(:tid %{NUMBER:[apache2][error][tid]})?\\]( \\[client %{IPORHOST:[apache2][error][client]}\\])? %{GREEDYDATA:[apache2][error][message1]}" - ] - }, - "pattern_definitions": { - "APACHE_TIME": "%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR}" - }, - "remove_field": "message" - }, - "comments": [] - }, - { - "id": "filter_mutate_9", - "type": "filter", - "plugin": "mutate", - "config": { - "rename": { - "[apache2][error][message1]": "[apache2][error][message]" - } - }, - "comments": [] - }, - { - "id": "filter_date_10", - "type": "filter", - "plugin": "date", - "config": { - "match": [ - "[apache2][error][timestamp]", - "EEE MMM dd H:m:s YYYY", - "EEE MMM dd H:m:s.SSSSSS YYYY" - ], - "remove_field": "[apache2][error][timestamp]" - }, - "comments": [] - } - ] - } - ], - "else": null - } - } - ], - "else_ifs": [], - "else": null - } - } - ], - "output": [ - { - "id": "output_elasticsearch_11", - "type": "output", - "plugin": "elasticsearch", - "config": { - "hosts": "localhost", - "manage_template": "false", - "index": "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" - }, - "comments": [] - } - ] -} diff --git a/src/logstashui/Common/tests/conversion_data/components/test-asa-new.json b/src/logstashui/Common/tests/conversion_data/components/test-asa-new.json deleted file mode 100644 index 3945c20..0000000 --- a/src/logstashui/Common/tests/conversion_data/components/test-asa-new.json +++ /dev/null @@ -1,1846 +0,0 @@ -{ - "input": [ - { - "id": "input_udp_0", - "type": "input", - "plugin": "udp", - "config": { - "id": "input_udp_1", - "port": "5119" - }, - "comments": [] - }, - { - "id": "input_cloudwatch_1", - "type": "input", - "plugin": "cloudwatch", - "config": {}, - "comments": [] - } - ], - "filter": [ - { - "id": "filter_mutate_2", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_1", - "rename": { - "message": "log.original", - "host": "observer.ip" - }, - "copy": { - "host": "sysloghost" - } - }, - "comments": [] - }, - { - "id": "filter_grok_3", - "type": "filter", - "plugin": "grok", - "config": { - "id": "filter_grok_1", - "match": { - "log.original": [ - "%{CISCO_TAGGED_SYSLOG} %{GREEDYDATA:message}", - "^<%{POSINT:syslog_pri}>%{DATA}: %%{DATA:ciscotag}: %{GREEDYDATA:message}", - "^<%{POSINT:syslog_pri}>%%{DATA:ciscotag}: %{GREEDYDATA:message}" - ] - } - }, - "comments": [] - }, - { - "id": "filter_grok_4", - "type": "filter", - "plugin": "grok", - "config": { - "id": "filter_grok_2", - "match": { - "ciscotag": [ - "%{WORD}-%{INT:event.severity}-%{INT:event.code}", - "%{WORD}-%{WORD}-%{INT:event.severity}-%{INT:event.code}" - ] - } - }, - "comments": [] - }, - { - "id": "filter_mutate_5", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_2", - "add_field": { - "event.action": "firewall-rule" - } - }, - "comments": [] - }, - { - "id": "filter_if_6", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[event.code] == \"105012\"", - "plugins": [ - { - "id": "filter_grok_7", - "type": "filter", - "plugin": "grok", - "config": { - "id": "filter_grok_3", - "match": { - "message": [ - "Teardown dynamic %{WORD:network.transport} translation from %{DATA:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} to %{DATA:cisco.destination_interface}:%{IP:destination.address}/%{INT:destination.port} duration %{DATA:cisco.duration_hms}$" - ] - } - }, - "comments": [] - } - ], - "else_ifs": [ - { - "condition": "[event.code] == \"106001\"", - "plugins": [ - { - "id": "filter_dissect_8", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_1", - "mapping": { - "message": "%{network.direction} %{network.transport} connection %{event.outcome} from %{source.address}/%{source.port} to %{destination.address}/%{destination.port} flags %{} on interface %{source_interface}" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_9", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_3", - "add_field": { - "event.category": "network_traffic" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"106002\"", - "plugins": [ - { - "id": "filter_dissect_10", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_2", - "mapping": { - "message": "%{network.transport} Connection %{event.outcome} by %{network.direction} list %{cisco.list_id} src %{source.address} dest %{destination.address}" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_11", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_4", - "add_field": { - "event.category": "network_traffic" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"106006\"", - "plugins": [ - { - "id": "filter_dissect_12", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_3", - "mapping": { - "message": "%{event.outcome} %{network.direction} %{network.transport} from %{source.address}/%{source.port} to %{destination.address}/%{destination.port} on interface %{cisco.source_interface}" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_13", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_5", - "add_field": { - "event.category": "network_traffic" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"106007\"", - "plugins": [ - { - "id": "filter_dissect_14", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_4", - "mapping": { - "message": "%{event.outcome} %{network.direction} %{network.transport} from %{source.address}/%{source.port} to %{destination.address}/%{destination.port} due to %{network.protocol} %{}" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_15", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_6", - "add_field": { - "event.category": "network_traffic" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"106010\"", - "plugins": [ - { - "id": "filter_dissect_16", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_5", - "mapping": { - "message": "%{event.outcome} %{network.direction} %{network.transport} src %{cisco.source_interface}:%{source.address}/%{source.port} %{} dst %{cisco.destination_interface}:%{destination.address}/%{destination.port} %{}" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_17", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_7", - "add_field": { - "event.category": "network_traffic" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"106013\"", - "plugins": [ - { - "id": "filter_dissect_18", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_6", - "mapping": { - "message": "Dropping echo request from %{source.address} to PAT address %{destination.address}" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_19", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_8", - "add_field": { - "network.transport": "icmp", - "network.direction": "inbound" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"106014\"", - "plugins": [ - { - "id": "filter_dissect_20", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_7", - "mapping": { - "message": "%{event.outcome} %{network.direction} %{network.transport} src %{cisco.source_interface}:%{source.address} %{}dst %{cisco.destination_interface}:%{destination.address} %{}" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_21", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_9", - "add_field": { - "event.category": "network_traffic" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"106015\"", - "plugins": [ - { - "id": "filter_dissect_22", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_8", - "mapping": { - "message": "%{event.outcome} %{network.transport} (no connection) from %{source.address}/%{source.port} to %{destination.address}/%{destination.port} flags %{} on interface %{cisco.source_interface}" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_23", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_10", - "add_field": { - "event.category": "nat_translation" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"106016\"", - "plugins": [ - { - "id": "filter_dissect_24", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_9", - "mapping": { - "message": "%{event.outcome} IP spoof from (%{source.address}) to %{destination.address} on interface %{cisco.source_interface}" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"106017\"", - "plugins": [ - { - "id": "filter_dissect_25", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_10", - "mapping": { - "message": "%{event.outcome} IP due to Land Attack from %{source.address} to %{destination.address}" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"106018\"", - "plugins": [ - { - "id": "filter_dissect_26", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_11", - "mapping": { - "message": "%{network.transport} packet type %{cisco.icmp_type} %{event.outcome} by %{network.direction} list %{cisco.list_id} src %{source.address} dest %{destination.address}" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"106020\"", - "plugins": [ - { - "id": "filter_dissect_27", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_12", - "mapping": { - "message": "%{event.outcome} IP teardrop fragment (size = %{}, offset = %{}) from %{source.address} to %{destination.address}" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"106021\"", - "plugins": [ - { - "id": "filter_dissect_28", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_13", - "mapping": { - "message": "%{event.outcome} %{network.transport} reverse path check from %{source.address} to %{destination.address} on interface %{cisco.source_interface}" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"106022\"", - "plugins": [ - { - "id": "filter_dissect_29", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_14", - "mapping": { - "message": "%{event.outcome} %{network.transport} connection spoof from %{source.address} to %{destination.address} on interface %{cisco.source_interface}" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"106023\"", - "plugins": [ - { - "id": "filter_grok_30", - "type": "filter", - "plugin": "grok", - "config": { - "id": "filter_grok_4", - "match": { - "message": [ - "%{WORD:event.outcome} %{WORD:network.transport} src %{WORD:cisco.source_interface}:%{IPORHOST:source.address}?(/%{INT:source.port}) dst %{WORD:cisco.destination_interface}:%{IPORHOST:destination.address}?(/%{INT:destination.port}) by access-group \"%{DATA:cisco.list_id}\"", - "%{WORD:event.outcome} %{WORD:network.transport} src %{WORD:cisco.source_interface}:%{IPORHOST:source.address} dst %{WORD:destination.direction}:%{IPORHOST:destination.address} \\(%{DATA}\\) by access-group \"%{DATA:cisco.list_id}\"", - "%{WORD:event.outcome} %{WORD:network.transport} src %{WORD:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} dst %{WORD:cisco.destination.interface}:%{IPORHOST:destination.address}/%{INT:destination.port} by access-group \"%{DATA:cisco.list_id}\"" - ] - } - }, - "comments": [] - }, - { - "id": "filter_mutate_31", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_11", - "add_field": { - "event.category": "network_traffic" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"106027\"", - "plugins": [ - { - "id": "filter_dissect_32", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_15", - "mapping": { - "message": "%{} %{event.outcome} src %{source.address} dst %{destination.address} by access-group \"%{cisco.list_id}\"%{}" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"106100\"", - "plugins": [ - { - "id": "filter_dissect_33", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_16", - "mapping": { - "message": "access-list %{cisco.list_id} %{event.outcome} %{network.transport} %{cisco.source_interface}/%{source.address}(%{source.port}) -> %{cisco.destination_interface}/%{destination.address}(%{destination.port}) %{}" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"106102\"", - "plugins": [ - { - "id": "filter_dissect_34", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_17", - "mapping": { - "message": "access-list %{cisco.list_id} %{event.outcome} %{network.transport} for user %{cisco.username} %{cisco.source_interface}/%{source.address} %{source.port} %{cisco.destination_interface}/%{destination.address} %{destination.port} %{}" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"106103\"", - "plugins": [ - { - "id": "filter_dissect_35", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_18", - "mapping": { - "message": "access-list %{cisco.list_id} %{event.outcome} %{network.transport} for user %{cisco.username} %{cisco.source_interface}/%{source.address} %{source.port} %{cisco.destination_interface}/%{destination.address} %{destination.port} %{}" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"113004\"", - "plugins": [ - { - "id": "filter_grok_36", - "type": "filter", - "plugin": "grok", - "config": { - "id": "filter_grok_5", - "match": { - "message": [ - "AAA user accounting %{WORD:cisco.auth_outcome} : server =%{SPACE}%{IP:source.address} : user =%{SPACE}%{DATA:source.user.name}$" - ] - } - }, - "comments": [] - }, - { - "id": "filter_mutate_37", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_12", - "add_field": { - "event.category": "authentication" - } - }, - "comments": [] - }, - { - "id": "filter_if_38", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[cisco.auth_outcome] == \"Successful\"", - "plugins": [ - { - "id": "filter_mutate_39", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_13", - "add_field": { - "event.action": "authentication_success" - } - }, - "comments": [] - } - ], - "else_ifs": [], - "else": { - "plugins": [ - { - "id": "filter_mutate_40", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_14", - "add_field": { - "event.action": "authentication_failure" - } - }, - "comments": [] - } - ] - } - } - } - ] - }, - { - "condition": "[event.code] == \"302015\" or [event.code] == \"302013\"", - "plugins": [ - { - "id": "filter_grok_41", - "type": "filter", - "plugin": "grok", - "config": { - "id": "filter_grok_6", - "match": { - "message": [ - "Built %{WORD:network.direction} %{WORD:network.transport} connection %{INT} for %{DATA:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} \\(%{IP}|\\) to %{DATA:cisco.destination_interface}:%{IPORHOST:destination.address}/%{INT:destination.port} \\(%{DATA}\\)", - "Built %{WORD:network.direction} %{WORD:network.transport} connection %{INT} for %{DATA:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} \\(%{DATA}\\)?(\\(%{DATA:cisco.source_username}\\)) to %{DATA:cisco.destination_interface}:%{IPORHOST:destination.address}/%{INT:destination.port} \\(%{DATA}\\) ?(\\(%{DATA:cisco.username}\\))", - "Built %{WORD:network.direction} %{WORD:network.transport} connection %{INT:cisco.connection_id} for %{WORD:cisco.source_interface}:%{IPORHOST:source.address}\\/%{INT:source.port} \\(%{DATA}\\) to %{WORD:cisco.destination_interface}:%{IPORHOST:destination.address}/%{INT:destination.port}" - ] - } - }, - "comments": [] - }, - { - "id": "filter_mutate_42", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_15", - "add_field": { - "event.category": "nat_translation" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"110003\"", - "plugins": [ - { - "id": "filter_grok_43", - "type": "filter", - "plugin": "grok", - "config": { - "id": "filter_grok_7", - "match": { - "message": [ - "%{DATA:cisco.event_error} for %{WORD:network.transport} from %{DATA:cisco.source_interface}:%{IP:source.address}\\/%{INT:source.port} to %{DATA:cisco.destination_interface}:%{IP:destination.address}\\/%{INT:destination.port}" - ] - } - }, - "comments": [] - }, - { - "id": "filter_mutate_44", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_16", - "add_field": { - "event.category": "error" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"113019\"", - "plugins": [ - { - "id": "filter_grok_45", - "type": "filter", - "plugin": "grok", - "config": { - "id": "filter_grok_8", - "match": { - "message": [ - "Group = %{DATA:cisco.group}, Username = %{DATA:user.name}, IP = %{IP:cisco.client_vpn_ip}, %{DATA:cisco.client_vpn_action}\\. Session Type: %{DATA:cisco.session_type}, Duration: %{DATA:cisco.duration}, Bytes xmt: %{INT:cisco.vpn_transmit_byte_summary}, Bytes rcv: %{INT:cisco.vpn_receive_byte_summary}, Reason: %{DATA:cisco.client_vpn_outcome}$" - ] - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"304001\"", - "plugins": [ - { - "id": "filter_dissect_46", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_19", - "mapping": { - "message": "%{source.address} %{}ccessed URL %{destination.address}:%{url.original}" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_47", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_17", - "add_field": { - "event.outcome": "allow" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"304002\"", - "plugins": [ - { - "id": "filter_dissect_48", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_20", - "mapping": { - "message": "Access %{event.outcome} URL %{url.original} SRC %{source.address} %{}EST %{destination.address} on interface %{cisco.source_interface}" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"305011\"", - "plugins": [ - { - "id": "filter_grok_49", - "type": "filter", - "plugin": "grok", - "config": { - "id": "filter_grok_9", - "match": { - "message": [ - "Built dynamic %{WORD:network.transport} translation from %{DATA:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} to %{DATA:cisco.destination_interface}:%{IP:destination.address}/%{INT:destination.port}" - ] - } - }, - "comments": [] - }, - { - "id": "filter_mutate_50", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_18", - "add_field": { - "event.category": "nat_translation" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"305012\"", - "plugins": [ - { - "id": "filter_grok_51", - "type": "filter", - "plugin": "grok", - "config": { - "id": "filter_grok_10", - "match": { - "message": [ - "Teardown dynamic %{WORD:network.transport} translation from %{DATA:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} to %{DATA:cisco.destination_interface}:%{IP:destination.address}/%{INT:destination.port}" - ] - } - }, - "comments": [] - }, - { - "id": "filter_mutate_52", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_19", - "add_field": { - "event.category": "nat_translation" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"313001\"", - "plugins": [ - { - "id": "filter_dissect_53", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_21", - "mapping": { - "message": "%{event.outcome} %{network.transport} type=%{cisco.icmp_type}, code=%{cisco.icmp_code} from %{source.address} on interface %{cisco.source_interface}" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"313004\"", - "plugins": [ - { - "id": "filter_dissect_54", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_22", - "mapping": { - "message": "%{event.outcome} %{network.transport} type=%{cisco.icmp_type}, from%{}addr %{source.address} on interface %{cisco.source_interface} to %{destination.address}: no matching session" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"313005\"", - "plugins": [ - { - "id": "filter_dissect_55", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_23", - "mapping": { - "message": "No matching connection for %{network.transport} error message: %{} on %{cisco.source_interface} interface.%{}riginal IP payload: %{}" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"313008\"", - "plugins": [ - { - "id": "filter_dissect_56", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_24", - "mapping": { - "message": "%{event.outcome} %{network.transport} type=%{cisco.icmp_type} , code=%{cisco.icmp_code} from %{source.address} on interface %{cisco.source_interface}" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"313009\"", - "plugins": [ - { - "id": "filter_dissect_57", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_25", - "mapping": { - "message": "%{event.outcome} invalid %{network.transport} code %{cisco.icmp_code} , for %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"322001\"", - "plugins": [ - { - "id": "filter_dissect_58", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_26", - "mapping": { - "message": "%{event.outcome} MAC address %{source.mac}, possible spoof attempt on interface %{cisco.source_interface}" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"338001\"", - "plugins": [ - { - "id": "filter_dissect_59", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_27", - "mapping": { - "message": "Dynamic filter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{source.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"338002\"", - "plugins": [ - { - "id": "filter_dissect_60", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_28", - "mapping": { - "message": "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{destination.domain}" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_61", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_20", - "add_field": { - "server.domain": "[destination.domain]" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"338003\"", - "plugins": [ - { - "id": "filter_dissect_62", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_29", - "mapping": { - "message": "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"338004\"", - "plugins": [ - { - "id": "filter_dissect_63", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_30", - "mapping": { - "message": "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"338005\"", - "plugins": [ - { - "id": "filter_dissect_64", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_31", - "mapping": { - "message": "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{source.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_65", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_21", - "add_field": { - "server.domain": "[source.domain]" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"338006\"", - "plugins": [ - { - "id": "filter_dissect_66", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_32", - "mapping": { - "message": "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{destination.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_67", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_22", - "add_field": { - "server.domain": "server.domain" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"338007\"", - "plugins": [ - { - "id": "filter_dissect_68", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_33", - "mapping": { - "message": "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"338008\"", - "plugins": [ - { - "id": "filter_dissect_69", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_34", - "mapping": { - "message": "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"338101\"", - "plugins": [ - { - "id": "filter_dissect_70", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_35", - "mapping": { - "message": "Dynamic %{}ilter %{event.outcome} white%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{source.domain}" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_71", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_23", - "add_field": { - "server.domain": "server.domain" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"338102\"", - "plugins": [ - { - "id": "filter_dissect_72", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_36", - "mapping": { - "message": "Dynamic %{}ilter %{event.outcome} white%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{destination.domain}" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_73", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_24", - "add_field": { - "server.domain": "server.domain" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"338103\"", - "plugins": [ - { - "id": "filter_dissect_74", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_37", - "mapping": { - "message": "Dynamic %{}ilter %{event.outcome} white%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{}" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"338104\"", - "plugins": [ - { - "id": "filter_dissect_75", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_38", - "mapping": { - "message": "Dynamic %{}ilter %{event.outcome} white%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{}" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"338201\"", - "plugins": [ - { - "id": "filter_dissect_76", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_39", - "mapping": { - "message": "Dynamic %{}ilter %{event.outcome} grey%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{source.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_77", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_25", - "add_field": { - "server.domain": "server.domain" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"338202\"", - "plugins": [ - { - "id": "filter_dissect_78", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_40", - "mapping": { - "message": "Dynamic %{}ilter %{event.outcome} grey%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{destination.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_79", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_26", - "add_field": { - "server.domain": "server.domain" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"338203\"", - "plugins": [ - { - "id": "filter_dissect_80", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_41", - "mapping": { - "message": "Dynamic %{}ilter %{event.outcome} grey%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{source.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_81", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_27", - "add_field": { - "server.domain": "server.domain" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"338204\"", - "plugins": [ - { - "id": "filter_dissect_82", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_42", - "mapping": { - "message": "Dynamic %{}ilter %{event.outcome} grey%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{destination.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_83", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_28", - "add_field": { - "server.domain": "server.domain" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"338301\"", - "plugins": [ - { - "id": "filter_dissect_84", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_43", - "mapping": { - "message": "Intercepted DNS reply for domain %{source.domain} from %{cisco.source_interface}:%{source.address}/%{source.port} to %{cisco.destination_interface}:%{destination.address}/%{destination.port}, matched %{cisco.list_id}" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_85", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_29", - "add_field": { - "client.address": "client.address" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_86", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_30", - "add_field": { - "client.port": "client.port" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_87", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_31", - "add_field": { - "server.address": "server.address" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_88", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_32", - "add_field": { - "server.port": "server.port" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] in [\"302014\", \"302016\", \"302018\", \"302021\", \"302036\", \"302304\", \"302306\", \"302020\"]", - "plugins": [ - { - "id": "filter_grok_89", - "type": "filter", - "plugin": "grok", - "config": { - "id": "filter_grok_11", - "pattern_definitions": { - "NOTCOLON": "[^:]*", - "ECSSOURCEIPORHOST": "(?:%{IP:source.address}|%{HOSTNAME:source.domain})", - "ECSDESTIPORHOST": "(?:%{IP:destination.address}|%{HOSTNAME:destination.domain})", - "MAPPEDSRC": "(?:%{DATA:cisco.mapped_source_ip}|%{HOSTNAME})" - }, - "match": { - "message": [ - "Teardown %{NOTSPACE:network.transport} (?:state-bypass )?connection %{NOTSPACE:cisco.connection_id} (?:for|from) %{NOTCOLON}:%{DATA:source.address}/%{NUMBER:source.port:int}?(\\(%{DATA:cisco.source_username}\\)|) ?to %{NOTCOLON:cisco.destination_interface}:%{DATA:destination.address}/%{NUMBER:destination.port:int}?(\\(%{DATA:cisco.source_username}\\)|) ?(?:duration %{TIME:cisco.duration_hms} bytes %{NUMBER:network.bytes:int})%{GREEDYDATA}", - "Teardown %{NOTSPACE:network.transport} (?:state-bypass )?connection %{NOTSPACE:cisco.connection_id} (?:for|from) %{NOTCOLON}:%{DATA:source.address}/%{NUMBER:source.port:int} (?:%{NOTSPACE:cisco.source_username} )?to %{NOTCOLON:cisco.destination_interface}:%{DATA:destination.address}/%{NUMBER:destination.port:int} (?:%{NOTSPACE:cisco.destination_username} )?(?:duration %{TIME:cisco.duration_hms} bytes %{NUMBER:network.bytes:int})%{GREEDYDATA}", - "Teardown %{NOTSPACE:network.transport} connection for faddr (?:%{NOTCOLON:cisco.source_interface}:)?%{ECSDESTIPORHOST}/%{NUMBER} (?:%{NOTSPACE:cisco.destination_username} )?gaddr (?:%{NOTCOLON}:)?%{MAPPEDSRC}/%{NUMBER} laddr (?:%{NOTCOLON:cisco.source_interface}:)?%{ECSSOURCEIPORHOST}/%{NUMBER}(?: %{NOTSPACE:cisco.source_username})?%{GREEDYDATA}", - "Built %{WORD:network.direction} %{NOTSPACE:network.transport} connection for faddr (?:%{NOTCOLON:cisco.source_interface}:)?%{ECSDESTIPORHOST}/%{NUMBER} (?:%{NOTSPACE:cisco.destination_username} )?gaddr (?:%{NOTCOLON}:)?%{MAPPEDSRC}/%{NUMBER} laddr (?:%{NOTCOLON:cisco.source_interface}:)?%{ECSSOURCEIPORHOST}/%{NUMBER}(?: %{NOTSPACE:cisco.source_username})?%{GREEDYDATA}" - ] - } - }, - "comments": [] - }, - { - "id": "filter_mutate_90", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_33", - "add_field": { - "event.category": "nat_translation" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"419002\"", - "plugins": [ - { - "id": "filter_grok_91", - "type": "filter", - "plugin": "grok", - "config": { - "id": "filter_grok_12", - "match": { - "message": [ - "%{DATA:cisco.event_error} from %{WORD:cisco.source_interface}:%{IPORHOST:source.address}\\/%{INT:source.port} to %{WORD:cisco.destination_interface}:%{IPORHOST:destination.address}\\/%{INT:destination.port}" - ] - } - }, - "comments": [] - }, - { - "id": "filter_mutate_92", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_34", - "add_field": { - "event.category": "error" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] in [\"733100\", \"752015\", \"752012\"]", - "plugins": [ - { - "id": "filter_grok_93", - "type": "filter", - "plugin": "grok", - "config": { - "id": "filter_grok_13", - "match": { - "message": [ - "%{GREEDYDATA:cisco.event_error}" - ] - } - }, - "comments": [] - }, - { - "id": "filter_mutate_94", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_35", - "add_field": { - "event.category": "error" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"716002\"", - "plugins": [ - { - "id": "filter_grok_95", - "type": "filter", - "plugin": "grok", - "config": { - "id": "filter_grok_14", - "match": { - "message": [ - "Group \\<%{DATA:cisco.group} User \\<%{DATA:user.name}\\> IP \\<%{IP:cisco.client_vpn_ip}\\> WebVPN session %{WORD:cisco.client_vpn_session_outcome}\\: %{DATA:cisco.web_vpn_action}\\." - ] - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] in ['722022', '722033', '722055', '722051', '113039', '722023', '722037']", - "plugins": [ - { - "id": "filter_grok_96", - "type": "filter", - "plugin": "grok", - "config": { - "id": "filter_grok_15", - "match": { - "message": [ - "Group \\<%{DATA:cisco.group} User \\<%{DATA:user.name}\\> IP \\<%{IP:cisco.client_vpn_ip}\\> %{GREEDYDATA:cisco.message}" - ] - } - }, - "comments": [] - } - ] - } - ], - "else": { - "plugins": [ - { - "id": "filter_grok_97", - "type": "filter", - "plugin": "grok", - "config": { - "id": "filter_grok_16", - "match": { - "message": [ - "forced_failure" - ] - } - }, - "comments": [] - } - ] - } - } - }, - { - "id": "filter_if_98", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[event.category] == \"nat_translation\"", - "plugins": [ - { - "id": "filter_drop_99", - "type": "filter", - "plugin": "drop", - "config": { - "id": "filter_drop_1" - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_if_100", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[source.address]", - "plugins": [ - { - "id": "filter_grok_101", - "type": "filter", - "plugin": "grok", - "config": { - "id": "filter_grok_17", - "match": { - "source.address": [ - "(?:%{IP:source.ip}|%{GREEDYDATA:source.domain})" - ] - } - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_if_102", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[destination.address]", - "plugins": [ - { - "id": "filter_grok_103", - "type": "filter", - "plugin": "grok", - "config": { - "id": "filter_grok_18", - "match": { - "destination.address": [ - "(?:%{IP:destination.ip}|%{GREEDYDATA:destination.domain})" - ] - } - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_if_104", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[client.address]", - "plugins": [ - { - "id": "filter_grok_105", - "type": "filter", - "plugin": "grok", - "config": { - "id": "filter_grok_19", - "match": { - "client.address": [ - "(?:%{IP:client.ip}|%{GREEDYDATA:client.domain})" - ] - } - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_if_106", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[server.address]", - "plugins": [ - { - "id": "filter_grok_107", - "type": "filter", - "plugin": "grok", - "config": { - "id": "filter_grok_20", - "match": { - "server.address": [ - "(?:%{IP:server.ip}|%{GREEDYDATA:server.domain})" - ] - } - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_mutate_108", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_36", - "lowercase": [ - "network.transport", - "network.protocol", - "network.direction", - "event.outcome" - ] - }, - "comments": [] - }, - { - "id": "filter_if_109", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[event.outcome] == \"est-allowed\"", - "plugins": [ - { - "id": "filter_mutate_110", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_37", - "update": { - "event.outcome": "allow" - } - }, - "comments": [] - } - ], - "else_ifs": [ - { - "condition": "[event.outcome] == \"permitted\"", - "plugins": [ - { - "id": "filter_mutate_111", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_38", - "update": { - "event.outcome": "allow" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.outcome] == \"denied\"", - "plugins": [ - { - "id": "filter_mutate_112", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_39", - "update": { - "event.outcome": "deny" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.outcome] == \"dropped\"", - "plugins": [ - { - "id": "filter_mutate_113", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_40", - "update": { - "event.outcome": "deny" - } - }, - "comments": [] - } - ] - } - ], - "else": null - } - }, - { - "id": "filter_if_114", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[network.transport] == \"icmpv6\"", - "plugins": [ - { - "id": "filter_mutate_115", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_41", - "update": { - "network.transport": "ipv6-icmp" - } - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_translate_116", - "type": "filter", - "plugin": "translate", - "config": { - "id": "filter_translate_1", - "field": "network.transport", - "destination": "network.iana_number", - "dictionary": { - "icmp": "1", - "igmp": "2", - "ipv4": "4", - "tcp": "6", - "egp": "8", - "igp": "9", - "pup": "12", - "udp": "17", - "rdp": "27", - "irtp": "28", - "dccp": "33", - "idpr": "35", - "ipv6": "41", - "ipv6-route": "43", - "ipv6-frag": "44", - "rsvp": "46", - "gre": "47", - "esp": "50", - "ipv6-icmp": "58", - "ipv6-nonxt": "59", - "ipv6-opts": "60" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_117", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_42", - "remove_field": [ - "ciscotag", - "timestamp" - ] - }, - "comments": [] - }, - { - "id": "filter_mutate_118", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_43", - "add_field": { - "event.module": "cisco", - "event.dataset": "asa" - } - }, - "comments": [] - }, - { - "id": "filter_translate_119", - "type": "filter", - "plugin": "translate", - "config": { - "id": "filter_translate_2", - "field": "[event.severity]", - "destination": "[log.level]", - "dictionary": { - "0": "emergency", - "1": "alert", - "2": "critical", - "3": "error", - "4": "warning", - "5": "notification", - "6": "informational", - "7": "debug" - } - }, - "comments": [] - } - ], - "output": [ - { - "id": "output_elasticsearch_120", - "type": "output", - "plugin": "elasticsearch", - "config": { - "id": "output_elasticsearch_1", - "api_key": "${es_api_key}", - "hosts": "https://homedc-90e54c.es.us-east-2.aws.elastic-cloud.com:443", - "index": "asa-1.2", - "pipeline": "asa" - }, - "comments": [] - } - ] -} diff --git a/src/logstashui/Common/tests/conversion_data/components/test-asa.json b/src/logstashui/Common/tests/conversion_data/components/test-asa.json deleted file mode 100644 index 3945c20..0000000 --- a/src/logstashui/Common/tests/conversion_data/components/test-asa.json +++ /dev/null @@ -1,1846 +0,0 @@ -{ - "input": [ - { - "id": "input_udp_0", - "type": "input", - "plugin": "udp", - "config": { - "id": "input_udp_1", - "port": "5119" - }, - "comments": [] - }, - { - "id": "input_cloudwatch_1", - "type": "input", - "plugin": "cloudwatch", - "config": {}, - "comments": [] - } - ], - "filter": [ - { - "id": "filter_mutate_2", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_1", - "rename": { - "message": "log.original", - "host": "observer.ip" - }, - "copy": { - "host": "sysloghost" - } - }, - "comments": [] - }, - { - "id": "filter_grok_3", - "type": "filter", - "plugin": "grok", - "config": { - "id": "filter_grok_1", - "match": { - "log.original": [ - "%{CISCO_TAGGED_SYSLOG} %{GREEDYDATA:message}", - "^<%{POSINT:syslog_pri}>%{DATA}: %%{DATA:ciscotag}: %{GREEDYDATA:message}", - "^<%{POSINT:syslog_pri}>%%{DATA:ciscotag}: %{GREEDYDATA:message}" - ] - } - }, - "comments": [] - }, - { - "id": "filter_grok_4", - "type": "filter", - "plugin": "grok", - "config": { - "id": "filter_grok_2", - "match": { - "ciscotag": [ - "%{WORD}-%{INT:event.severity}-%{INT:event.code}", - "%{WORD}-%{WORD}-%{INT:event.severity}-%{INT:event.code}" - ] - } - }, - "comments": [] - }, - { - "id": "filter_mutate_5", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_2", - "add_field": { - "event.action": "firewall-rule" - } - }, - "comments": [] - }, - { - "id": "filter_if_6", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[event.code] == \"105012\"", - "plugins": [ - { - "id": "filter_grok_7", - "type": "filter", - "plugin": "grok", - "config": { - "id": "filter_grok_3", - "match": { - "message": [ - "Teardown dynamic %{WORD:network.transport} translation from %{DATA:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} to %{DATA:cisco.destination_interface}:%{IP:destination.address}/%{INT:destination.port} duration %{DATA:cisco.duration_hms}$" - ] - } - }, - "comments": [] - } - ], - "else_ifs": [ - { - "condition": "[event.code] == \"106001\"", - "plugins": [ - { - "id": "filter_dissect_8", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_1", - "mapping": { - "message": "%{network.direction} %{network.transport} connection %{event.outcome} from %{source.address}/%{source.port} to %{destination.address}/%{destination.port} flags %{} on interface %{source_interface}" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_9", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_3", - "add_field": { - "event.category": "network_traffic" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"106002\"", - "plugins": [ - { - "id": "filter_dissect_10", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_2", - "mapping": { - "message": "%{network.transport} Connection %{event.outcome} by %{network.direction} list %{cisco.list_id} src %{source.address} dest %{destination.address}" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_11", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_4", - "add_field": { - "event.category": "network_traffic" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"106006\"", - "plugins": [ - { - "id": "filter_dissect_12", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_3", - "mapping": { - "message": "%{event.outcome} %{network.direction} %{network.transport} from %{source.address}/%{source.port} to %{destination.address}/%{destination.port} on interface %{cisco.source_interface}" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_13", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_5", - "add_field": { - "event.category": "network_traffic" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"106007\"", - "plugins": [ - { - "id": "filter_dissect_14", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_4", - "mapping": { - "message": "%{event.outcome} %{network.direction} %{network.transport} from %{source.address}/%{source.port} to %{destination.address}/%{destination.port} due to %{network.protocol} %{}" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_15", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_6", - "add_field": { - "event.category": "network_traffic" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"106010\"", - "plugins": [ - { - "id": "filter_dissect_16", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_5", - "mapping": { - "message": "%{event.outcome} %{network.direction} %{network.transport} src %{cisco.source_interface}:%{source.address}/%{source.port} %{} dst %{cisco.destination_interface}:%{destination.address}/%{destination.port} %{}" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_17", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_7", - "add_field": { - "event.category": "network_traffic" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"106013\"", - "plugins": [ - { - "id": "filter_dissect_18", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_6", - "mapping": { - "message": "Dropping echo request from %{source.address} to PAT address %{destination.address}" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_19", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_8", - "add_field": { - "network.transport": "icmp", - "network.direction": "inbound" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"106014\"", - "plugins": [ - { - "id": "filter_dissect_20", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_7", - "mapping": { - "message": "%{event.outcome} %{network.direction} %{network.transport} src %{cisco.source_interface}:%{source.address} %{}dst %{cisco.destination_interface}:%{destination.address} %{}" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_21", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_9", - "add_field": { - "event.category": "network_traffic" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"106015\"", - "plugins": [ - { - "id": "filter_dissect_22", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_8", - "mapping": { - "message": "%{event.outcome} %{network.transport} (no connection) from %{source.address}/%{source.port} to %{destination.address}/%{destination.port} flags %{} on interface %{cisco.source_interface}" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_23", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_10", - "add_field": { - "event.category": "nat_translation" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"106016\"", - "plugins": [ - { - "id": "filter_dissect_24", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_9", - "mapping": { - "message": "%{event.outcome} IP spoof from (%{source.address}) to %{destination.address} on interface %{cisco.source_interface}" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"106017\"", - "plugins": [ - { - "id": "filter_dissect_25", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_10", - "mapping": { - "message": "%{event.outcome} IP due to Land Attack from %{source.address} to %{destination.address}" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"106018\"", - "plugins": [ - { - "id": "filter_dissect_26", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_11", - "mapping": { - "message": "%{network.transport} packet type %{cisco.icmp_type} %{event.outcome} by %{network.direction} list %{cisco.list_id} src %{source.address} dest %{destination.address}" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"106020\"", - "plugins": [ - { - "id": "filter_dissect_27", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_12", - "mapping": { - "message": "%{event.outcome} IP teardrop fragment (size = %{}, offset = %{}) from %{source.address} to %{destination.address}" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"106021\"", - "plugins": [ - { - "id": "filter_dissect_28", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_13", - "mapping": { - "message": "%{event.outcome} %{network.transport} reverse path check from %{source.address} to %{destination.address} on interface %{cisco.source_interface}" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"106022\"", - "plugins": [ - { - "id": "filter_dissect_29", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_14", - "mapping": { - "message": "%{event.outcome} %{network.transport} connection spoof from %{source.address} to %{destination.address} on interface %{cisco.source_interface}" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"106023\"", - "plugins": [ - { - "id": "filter_grok_30", - "type": "filter", - "plugin": "grok", - "config": { - "id": "filter_grok_4", - "match": { - "message": [ - "%{WORD:event.outcome} %{WORD:network.transport} src %{WORD:cisco.source_interface}:%{IPORHOST:source.address}?(/%{INT:source.port}) dst %{WORD:cisco.destination_interface}:%{IPORHOST:destination.address}?(/%{INT:destination.port}) by access-group \"%{DATA:cisco.list_id}\"", - "%{WORD:event.outcome} %{WORD:network.transport} src %{WORD:cisco.source_interface}:%{IPORHOST:source.address} dst %{WORD:destination.direction}:%{IPORHOST:destination.address} \\(%{DATA}\\) by access-group \"%{DATA:cisco.list_id}\"", - "%{WORD:event.outcome} %{WORD:network.transport} src %{WORD:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} dst %{WORD:cisco.destination.interface}:%{IPORHOST:destination.address}/%{INT:destination.port} by access-group \"%{DATA:cisco.list_id}\"" - ] - } - }, - "comments": [] - }, - { - "id": "filter_mutate_31", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_11", - "add_field": { - "event.category": "network_traffic" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"106027\"", - "plugins": [ - { - "id": "filter_dissect_32", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_15", - "mapping": { - "message": "%{} %{event.outcome} src %{source.address} dst %{destination.address} by access-group \"%{cisco.list_id}\"%{}" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"106100\"", - "plugins": [ - { - "id": "filter_dissect_33", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_16", - "mapping": { - "message": "access-list %{cisco.list_id} %{event.outcome} %{network.transport} %{cisco.source_interface}/%{source.address}(%{source.port}) -> %{cisco.destination_interface}/%{destination.address}(%{destination.port}) %{}" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"106102\"", - "plugins": [ - { - "id": "filter_dissect_34", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_17", - "mapping": { - "message": "access-list %{cisco.list_id} %{event.outcome} %{network.transport} for user %{cisco.username} %{cisco.source_interface}/%{source.address} %{source.port} %{cisco.destination_interface}/%{destination.address} %{destination.port} %{}" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"106103\"", - "plugins": [ - { - "id": "filter_dissect_35", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_18", - "mapping": { - "message": "access-list %{cisco.list_id} %{event.outcome} %{network.transport} for user %{cisco.username} %{cisco.source_interface}/%{source.address} %{source.port} %{cisco.destination_interface}/%{destination.address} %{destination.port} %{}" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"113004\"", - "plugins": [ - { - "id": "filter_grok_36", - "type": "filter", - "plugin": "grok", - "config": { - "id": "filter_grok_5", - "match": { - "message": [ - "AAA user accounting %{WORD:cisco.auth_outcome} : server =%{SPACE}%{IP:source.address} : user =%{SPACE}%{DATA:source.user.name}$" - ] - } - }, - "comments": [] - }, - { - "id": "filter_mutate_37", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_12", - "add_field": { - "event.category": "authentication" - } - }, - "comments": [] - }, - { - "id": "filter_if_38", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[cisco.auth_outcome] == \"Successful\"", - "plugins": [ - { - "id": "filter_mutate_39", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_13", - "add_field": { - "event.action": "authentication_success" - } - }, - "comments": [] - } - ], - "else_ifs": [], - "else": { - "plugins": [ - { - "id": "filter_mutate_40", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_14", - "add_field": { - "event.action": "authentication_failure" - } - }, - "comments": [] - } - ] - } - } - } - ] - }, - { - "condition": "[event.code] == \"302015\" or [event.code] == \"302013\"", - "plugins": [ - { - "id": "filter_grok_41", - "type": "filter", - "plugin": "grok", - "config": { - "id": "filter_grok_6", - "match": { - "message": [ - "Built %{WORD:network.direction} %{WORD:network.transport} connection %{INT} for %{DATA:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} \\(%{IP}|\\) to %{DATA:cisco.destination_interface}:%{IPORHOST:destination.address}/%{INT:destination.port} \\(%{DATA}\\)", - "Built %{WORD:network.direction} %{WORD:network.transport} connection %{INT} for %{DATA:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} \\(%{DATA}\\)?(\\(%{DATA:cisco.source_username}\\)) to %{DATA:cisco.destination_interface}:%{IPORHOST:destination.address}/%{INT:destination.port} \\(%{DATA}\\) ?(\\(%{DATA:cisco.username}\\))", - "Built %{WORD:network.direction} %{WORD:network.transport} connection %{INT:cisco.connection_id} for %{WORD:cisco.source_interface}:%{IPORHOST:source.address}\\/%{INT:source.port} \\(%{DATA}\\) to %{WORD:cisco.destination_interface}:%{IPORHOST:destination.address}/%{INT:destination.port}" - ] - } - }, - "comments": [] - }, - { - "id": "filter_mutate_42", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_15", - "add_field": { - "event.category": "nat_translation" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"110003\"", - "plugins": [ - { - "id": "filter_grok_43", - "type": "filter", - "plugin": "grok", - "config": { - "id": "filter_grok_7", - "match": { - "message": [ - "%{DATA:cisco.event_error} for %{WORD:network.transport} from %{DATA:cisco.source_interface}:%{IP:source.address}\\/%{INT:source.port} to %{DATA:cisco.destination_interface}:%{IP:destination.address}\\/%{INT:destination.port}" - ] - } - }, - "comments": [] - }, - { - "id": "filter_mutate_44", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_16", - "add_field": { - "event.category": "error" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"113019\"", - "plugins": [ - { - "id": "filter_grok_45", - "type": "filter", - "plugin": "grok", - "config": { - "id": "filter_grok_8", - "match": { - "message": [ - "Group = %{DATA:cisco.group}, Username = %{DATA:user.name}, IP = %{IP:cisco.client_vpn_ip}, %{DATA:cisco.client_vpn_action}\\. Session Type: %{DATA:cisco.session_type}, Duration: %{DATA:cisco.duration}, Bytes xmt: %{INT:cisco.vpn_transmit_byte_summary}, Bytes rcv: %{INT:cisco.vpn_receive_byte_summary}, Reason: %{DATA:cisco.client_vpn_outcome}$" - ] - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"304001\"", - "plugins": [ - { - "id": "filter_dissect_46", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_19", - "mapping": { - "message": "%{source.address} %{}ccessed URL %{destination.address}:%{url.original}" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_47", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_17", - "add_field": { - "event.outcome": "allow" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"304002\"", - "plugins": [ - { - "id": "filter_dissect_48", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_20", - "mapping": { - "message": "Access %{event.outcome} URL %{url.original} SRC %{source.address} %{}EST %{destination.address} on interface %{cisco.source_interface}" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"305011\"", - "plugins": [ - { - "id": "filter_grok_49", - "type": "filter", - "plugin": "grok", - "config": { - "id": "filter_grok_9", - "match": { - "message": [ - "Built dynamic %{WORD:network.transport} translation from %{DATA:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} to %{DATA:cisco.destination_interface}:%{IP:destination.address}/%{INT:destination.port}" - ] - } - }, - "comments": [] - }, - { - "id": "filter_mutate_50", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_18", - "add_field": { - "event.category": "nat_translation" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"305012\"", - "plugins": [ - { - "id": "filter_grok_51", - "type": "filter", - "plugin": "grok", - "config": { - "id": "filter_grok_10", - "match": { - "message": [ - "Teardown dynamic %{WORD:network.transport} translation from %{DATA:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} to %{DATA:cisco.destination_interface}:%{IP:destination.address}/%{INT:destination.port}" - ] - } - }, - "comments": [] - }, - { - "id": "filter_mutate_52", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_19", - "add_field": { - "event.category": "nat_translation" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"313001\"", - "plugins": [ - { - "id": "filter_dissect_53", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_21", - "mapping": { - "message": "%{event.outcome} %{network.transport} type=%{cisco.icmp_type}, code=%{cisco.icmp_code} from %{source.address} on interface %{cisco.source_interface}" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"313004\"", - "plugins": [ - { - "id": "filter_dissect_54", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_22", - "mapping": { - "message": "%{event.outcome} %{network.transport} type=%{cisco.icmp_type}, from%{}addr %{source.address} on interface %{cisco.source_interface} to %{destination.address}: no matching session" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"313005\"", - "plugins": [ - { - "id": "filter_dissect_55", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_23", - "mapping": { - "message": "No matching connection for %{network.transport} error message: %{} on %{cisco.source_interface} interface.%{}riginal IP payload: %{}" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"313008\"", - "plugins": [ - { - "id": "filter_dissect_56", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_24", - "mapping": { - "message": "%{event.outcome} %{network.transport} type=%{cisco.icmp_type} , code=%{cisco.icmp_code} from %{source.address} on interface %{cisco.source_interface}" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"313009\"", - "plugins": [ - { - "id": "filter_dissect_57", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_25", - "mapping": { - "message": "%{event.outcome} invalid %{network.transport} code %{cisco.icmp_code} , for %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"322001\"", - "plugins": [ - { - "id": "filter_dissect_58", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_26", - "mapping": { - "message": "%{event.outcome} MAC address %{source.mac}, possible spoof attempt on interface %{cisco.source_interface}" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"338001\"", - "plugins": [ - { - "id": "filter_dissect_59", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_27", - "mapping": { - "message": "Dynamic filter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{source.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"338002\"", - "plugins": [ - { - "id": "filter_dissect_60", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_28", - "mapping": { - "message": "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{destination.domain}" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_61", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_20", - "add_field": { - "server.domain": "[destination.domain]" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"338003\"", - "plugins": [ - { - "id": "filter_dissect_62", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_29", - "mapping": { - "message": "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"338004\"", - "plugins": [ - { - "id": "filter_dissect_63", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_30", - "mapping": { - "message": "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"338005\"", - "plugins": [ - { - "id": "filter_dissect_64", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_31", - "mapping": { - "message": "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{source.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_65", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_21", - "add_field": { - "server.domain": "[source.domain]" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"338006\"", - "plugins": [ - { - "id": "filter_dissect_66", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_32", - "mapping": { - "message": "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{destination.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_67", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_22", - "add_field": { - "server.domain": "server.domain" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"338007\"", - "plugins": [ - { - "id": "filter_dissect_68", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_33", - "mapping": { - "message": "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"338008\"", - "plugins": [ - { - "id": "filter_dissect_69", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_34", - "mapping": { - "message": "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"338101\"", - "plugins": [ - { - "id": "filter_dissect_70", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_35", - "mapping": { - "message": "Dynamic %{}ilter %{event.outcome} white%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{source.domain}" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_71", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_23", - "add_field": { - "server.domain": "server.domain" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"338102\"", - "plugins": [ - { - "id": "filter_dissect_72", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_36", - "mapping": { - "message": "Dynamic %{}ilter %{event.outcome} white%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{destination.domain}" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_73", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_24", - "add_field": { - "server.domain": "server.domain" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"338103\"", - "plugins": [ - { - "id": "filter_dissect_74", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_37", - "mapping": { - "message": "Dynamic %{}ilter %{event.outcome} white%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{}" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"338104\"", - "plugins": [ - { - "id": "filter_dissect_75", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_38", - "mapping": { - "message": "Dynamic %{}ilter %{event.outcome} white%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{}" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"338201\"", - "plugins": [ - { - "id": "filter_dissect_76", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_39", - "mapping": { - "message": "Dynamic %{}ilter %{event.outcome} grey%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{source.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_77", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_25", - "add_field": { - "server.domain": "server.domain" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"338202\"", - "plugins": [ - { - "id": "filter_dissect_78", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_40", - "mapping": { - "message": "Dynamic %{}ilter %{event.outcome} grey%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{destination.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_79", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_26", - "add_field": { - "server.domain": "server.domain" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"338203\"", - "plugins": [ - { - "id": "filter_dissect_80", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_41", - "mapping": { - "message": "Dynamic %{}ilter %{event.outcome} grey%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{source.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_81", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_27", - "add_field": { - "server.domain": "server.domain" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"338204\"", - "plugins": [ - { - "id": "filter_dissect_82", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_42", - "mapping": { - "message": "Dynamic %{}ilter %{event.outcome} grey%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{destination.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_83", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_28", - "add_field": { - "server.domain": "server.domain" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"338301\"", - "plugins": [ - { - "id": "filter_dissect_84", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "filter_dissect_43", - "mapping": { - "message": "Intercepted DNS reply for domain %{source.domain} from %{cisco.source_interface}:%{source.address}/%{source.port} to %{cisco.destination_interface}:%{destination.address}/%{destination.port}, matched %{cisco.list_id}" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_85", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_29", - "add_field": { - "client.address": "client.address" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_86", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_30", - "add_field": { - "client.port": "client.port" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_87", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_31", - "add_field": { - "server.address": "server.address" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_88", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_32", - "add_field": { - "server.port": "server.port" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] in [\"302014\", \"302016\", \"302018\", \"302021\", \"302036\", \"302304\", \"302306\", \"302020\"]", - "plugins": [ - { - "id": "filter_grok_89", - "type": "filter", - "plugin": "grok", - "config": { - "id": "filter_grok_11", - "pattern_definitions": { - "NOTCOLON": "[^:]*", - "ECSSOURCEIPORHOST": "(?:%{IP:source.address}|%{HOSTNAME:source.domain})", - "ECSDESTIPORHOST": "(?:%{IP:destination.address}|%{HOSTNAME:destination.domain})", - "MAPPEDSRC": "(?:%{DATA:cisco.mapped_source_ip}|%{HOSTNAME})" - }, - "match": { - "message": [ - "Teardown %{NOTSPACE:network.transport} (?:state-bypass )?connection %{NOTSPACE:cisco.connection_id} (?:for|from) %{NOTCOLON}:%{DATA:source.address}/%{NUMBER:source.port:int}?(\\(%{DATA:cisco.source_username}\\)|) ?to %{NOTCOLON:cisco.destination_interface}:%{DATA:destination.address}/%{NUMBER:destination.port:int}?(\\(%{DATA:cisco.source_username}\\)|) ?(?:duration %{TIME:cisco.duration_hms} bytes %{NUMBER:network.bytes:int})%{GREEDYDATA}", - "Teardown %{NOTSPACE:network.transport} (?:state-bypass )?connection %{NOTSPACE:cisco.connection_id} (?:for|from) %{NOTCOLON}:%{DATA:source.address}/%{NUMBER:source.port:int} (?:%{NOTSPACE:cisco.source_username} )?to %{NOTCOLON:cisco.destination_interface}:%{DATA:destination.address}/%{NUMBER:destination.port:int} (?:%{NOTSPACE:cisco.destination_username} )?(?:duration %{TIME:cisco.duration_hms} bytes %{NUMBER:network.bytes:int})%{GREEDYDATA}", - "Teardown %{NOTSPACE:network.transport} connection for faddr (?:%{NOTCOLON:cisco.source_interface}:)?%{ECSDESTIPORHOST}/%{NUMBER} (?:%{NOTSPACE:cisco.destination_username} )?gaddr (?:%{NOTCOLON}:)?%{MAPPEDSRC}/%{NUMBER} laddr (?:%{NOTCOLON:cisco.source_interface}:)?%{ECSSOURCEIPORHOST}/%{NUMBER}(?: %{NOTSPACE:cisco.source_username})?%{GREEDYDATA}", - "Built %{WORD:network.direction} %{NOTSPACE:network.transport} connection for faddr (?:%{NOTCOLON:cisco.source_interface}:)?%{ECSDESTIPORHOST}/%{NUMBER} (?:%{NOTSPACE:cisco.destination_username} )?gaddr (?:%{NOTCOLON}:)?%{MAPPEDSRC}/%{NUMBER} laddr (?:%{NOTCOLON:cisco.source_interface}:)?%{ECSSOURCEIPORHOST}/%{NUMBER}(?: %{NOTSPACE:cisco.source_username})?%{GREEDYDATA}" - ] - } - }, - "comments": [] - }, - { - "id": "filter_mutate_90", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_33", - "add_field": { - "event.category": "nat_translation" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"419002\"", - "plugins": [ - { - "id": "filter_grok_91", - "type": "filter", - "plugin": "grok", - "config": { - "id": "filter_grok_12", - "match": { - "message": [ - "%{DATA:cisco.event_error} from %{WORD:cisco.source_interface}:%{IPORHOST:source.address}\\/%{INT:source.port} to %{WORD:cisco.destination_interface}:%{IPORHOST:destination.address}\\/%{INT:destination.port}" - ] - } - }, - "comments": [] - }, - { - "id": "filter_mutate_92", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_34", - "add_field": { - "event.category": "error" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] in [\"733100\", \"752015\", \"752012\"]", - "plugins": [ - { - "id": "filter_grok_93", - "type": "filter", - "plugin": "grok", - "config": { - "id": "filter_grok_13", - "match": { - "message": [ - "%{GREEDYDATA:cisco.event_error}" - ] - } - }, - "comments": [] - }, - { - "id": "filter_mutate_94", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_35", - "add_field": { - "event.category": "error" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] == \"716002\"", - "plugins": [ - { - "id": "filter_grok_95", - "type": "filter", - "plugin": "grok", - "config": { - "id": "filter_grok_14", - "match": { - "message": [ - "Group \\<%{DATA:cisco.group} User \\<%{DATA:user.name}\\> IP \\<%{IP:cisco.client_vpn_ip}\\> WebVPN session %{WORD:cisco.client_vpn_session_outcome}\\: %{DATA:cisco.web_vpn_action}\\." - ] - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.code] in ['722022', '722033', '722055', '722051', '113039', '722023', '722037']", - "plugins": [ - { - "id": "filter_grok_96", - "type": "filter", - "plugin": "grok", - "config": { - "id": "filter_grok_15", - "match": { - "message": [ - "Group \\<%{DATA:cisco.group} User \\<%{DATA:user.name}\\> IP \\<%{IP:cisco.client_vpn_ip}\\> %{GREEDYDATA:cisco.message}" - ] - } - }, - "comments": [] - } - ] - } - ], - "else": { - "plugins": [ - { - "id": "filter_grok_97", - "type": "filter", - "plugin": "grok", - "config": { - "id": "filter_grok_16", - "match": { - "message": [ - "forced_failure" - ] - } - }, - "comments": [] - } - ] - } - } - }, - { - "id": "filter_if_98", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[event.category] == \"nat_translation\"", - "plugins": [ - { - "id": "filter_drop_99", - "type": "filter", - "plugin": "drop", - "config": { - "id": "filter_drop_1" - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_if_100", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[source.address]", - "plugins": [ - { - "id": "filter_grok_101", - "type": "filter", - "plugin": "grok", - "config": { - "id": "filter_grok_17", - "match": { - "source.address": [ - "(?:%{IP:source.ip}|%{GREEDYDATA:source.domain})" - ] - } - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_if_102", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[destination.address]", - "plugins": [ - { - "id": "filter_grok_103", - "type": "filter", - "plugin": "grok", - "config": { - "id": "filter_grok_18", - "match": { - "destination.address": [ - "(?:%{IP:destination.ip}|%{GREEDYDATA:destination.domain})" - ] - } - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_if_104", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[client.address]", - "plugins": [ - { - "id": "filter_grok_105", - "type": "filter", - "plugin": "grok", - "config": { - "id": "filter_grok_19", - "match": { - "client.address": [ - "(?:%{IP:client.ip}|%{GREEDYDATA:client.domain})" - ] - } - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_if_106", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[server.address]", - "plugins": [ - { - "id": "filter_grok_107", - "type": "filter", - "plugin": "grok", - "config": { - "id": "filter_grok_20", - "match": { - "server.address": [ - "(?:%{IP:server.ip}|%{GREEDYDATA:server.domain})" - ] - } - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_mutate_108", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_36", - "lowercase": [ - "network.transport", - "network.protocol", - "network.direction", - "event.outcome" - ] - }, - "comments": [] - }, - { - "id": "filter_if_109", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[event.outcome] == \"est-allowed\"", - "plugins": [ - { - "id": "filter_mutate_110", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_37", - "update": { - "event.outcome": "allow" - } - }, - "comments": [] - } - ], - "else_ifs": [ - { - "condition": "[event.outcome] == \"permitted\"", - "plugins": [ - { - "id": "filter_mutate_111", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_38", - "update": { - "event.outcome": "allow" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.outcome] == \"denied\"", - "plugins": [ - { - "id": "filter_mutate_112", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_39", - "update": { - "event.outcome": "deny" - } - }, - "comments": [] - } - ] - }, - { - "condition": "[event.outcome] == \"dropped\"", - "plugins": [ - { - "id": "filter_mutate_113", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_40", - "update": { - "event.outcome": "deny" - } - }, - "comments": [] - } - ] - } - ], - "else": null - } - }, - { - "id": "filter_if_114", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[network.transport] == \"icmpv6\"", - "plugins": [ - { - "id": "filter_mutate_115", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_41", - "update": { - "network.transport": "ipv6-icmp" - } - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_translate_116", - "type": "filter", - "plugin": "translate", - "config": { - "id": "filter_translate_1", - "field": "network.transport", - "destination": "network.iana_number", - "dictionary": { - "icmp": "1", - "igmp": "2", - "ipv4": "4", - "tcp": "6", - "egp": "8", - "igp": "9", - "pup": "12", - "udp": "17", - "rdp": "27", - "irtp": "28", - "dccp": "33", - "idpr": "35", - "ipv6": "41", - "ipv6-route": "43", - "ipv6-frag": "44", - "rsvp": "46", - "gre": "47", - "esp": "50", - "ipv6-icmp": "58", - "ipv6-nonxt": "59", - "ipv6-opts": "60" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_117", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_42", - "remove_field": [ - "ciscotag", - "timestamp" - ] - }, - "comments": [] - }, - { - "id": "filter_mutate_118", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_43", - "add_field": { - "event.module": "cisco", - "event.dataset": "asa" - } - }, - "comments": [] - }, - { - "id": "filter_translate_119", - "type": "filter", - "plugin": "translate", - "config": { - "id": "filter_translate_2", - "field": "[event.severity]", - "destination": "[log.level]", - "dictionary": { - "0": "emergency", - "1": "alert", - "2": "critical", - "3": "error", - "4": "warning", - "5": "notification", - "6": "informational", - "7": "debug" - } - }, - "comments": [] - } - ], - "output": [ - { - "id": "output_elasticsearch_120", - "type": "output", - "plugin": "elasticsearch", - "config": { - "id": "output_elasticsearch_1", - "api_key": "${es_api_key}", - "hosts": "https://homedc-90e54c.es.us-east-2.aws.elastic-cloud.com:443", - "index": "asa-1.2", - "pipeline": "asa" - }, - "comments": [] - } - ] -} diff --git a/src/logstashui/Common/tests/conversion_data/components/test-boolean-numeric.json b/src/logstashui/Common/tests/conversion_data/components/test-boolean-numeric.json deleted file mode 100644 index d708725..0000000 --- a/src/logstashui/Common/tests/conversion_data/components/test-boolean-numeric.json +++ /dev/null @@ -1,134 +0,0 @@ -{ - "input": [ - { - "id": "input_tcp_0", - "type": "input", - "plugin": "tcp", - "config": { - "port": 5000, - "ssl_enable": "false", - "buffer_size": 65536 - }, - "comments": [] - }, - { - "id": "input_udp_1", - "type": "input", - "plugin": "udp", - "config": { - "port": 514, - "queue_size": 2000, - "workers": 4 - }, - "comments": [] - } - ], - "filter": [ - { - "id": "filter_grok_2", - "type": "filter", - "plugin": "grok", - "config": { - "match": { - "message": "%{NUMBER:duration:float} %{NUMBER:status:int}" - }, - "keep_empty_captures": "false", - "tag_on_failure": [ - "_grokfailure" - ], - "timeout_millis": 30000, - "break_on_match": "true" - }, - "comments": [] - }, - { - "id": "filter_mutate_3", - "type": "filter", - "plugin": "mutate", - "config": { - "convert": { - "duration": "float", - "status": "integer", - "bytes": "integer" - } - }, - "comments": [] - }, - { - "id": "filter_if_4", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[duration] > 1.5", - "plugins": [ - { - "id": "filter_mutate_5", - "type": "filter", - "plugin": "mutate", - "config": { - "add_field": { - "slow_request": "true" - } - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_if_6", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[status] >= 500", - "plugins": [ - { - "id": "filter_mutate_7", - "type": "filter", - "plugin": "mutate", - "config": { - "add_field": { - "is_error": "true", - "error_code": 500, - "threshold_pct": 0.99 - } - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_throttle_8", - "type": "filter", - "plugin": "throttle", - "config": { - "before_count": 3, - "after_count": 1, - "period": 60, - "key": "%{host}", - "add_tag": [ - "throttled" - ] - }, - "comments": [] - } - ], - "output": [ - { - "id": "output_stdout_9", - "type": "output", - "plugin": "stdout", - "config": { - "codec": { - "rubydebug": {} - } - }, - "comments": [] - } - ] -} diff --git a/src/logstashui/Common/tests/conversion_data/components/test-comments-brace-in-comment.json b/src/logstashui/Common/tests/conversion_data/components/test-comments-brace-in-comment.json deleted file mode 100644 index c136b37..0000000 --- a/src/logstashui/Common/tests/conversion_data/components/test-comments-brace-in-comment.json +++ /dev/null @@ -1,72 +0,0 @@ -{ - "input": [], - "filter": [ - { - "id": "filter_mutate_0", - "type": "filter", - "plugin": "mutate", - "config": { - "add_field": { - "real_field": "real_value" - }, - "remove_field": [ - "unwanted" - ] - }, - "comments": [ - "add_field => {\"this_key\" => \"this_value\"}", - "rename => {\"old_field\" => \"new_field\"}", - "remove_field => [\"field1\", \"field2\"]", - "replace => {\"message\" => \"override: %{message}\"}" - ] - }, - { - "id": "filter_grok_1", - "type": "filter", - "plugin": "grok", - "config": { - "match": { - "message": "%{GREEDYDATA:raw_message}" - } - }, - "comments": [ - "match => { \"message\" => \"%{COMBINEDAPACHELOG}\" }", - "pattern_definitions => { \"MY_PATTERN\" => \"\\\\w+\" }" - ] - }, - { - "id": "filter_date_2", - "type": "filter", - "plugin": "date", - "config": { - "match": [ - "timestamp", - "ISO8601" - ], - "target": "@timestamp" - }, - "comments": [ - "match => [\"timestamp\", \"dd/MMM/yyyy:HH:mm:ss Z\", \"ISO8601\"]", - "target => \"@timestamp\"" - ] - }, - { - "id": "filter_translate_3", - "type": "filter", - "plugin": "translate", - "config": { - "field": "status_code", - "destination": "status_label", - "dictionary": { - "200": "OK", - "404": "Not Found" - }, - "fallback": "Unknown" - }, - "comments": [ - "dictionary => { \"200\" => \"OK\", \"404\" => \"Not Found\", \"500\" => \"Error\" }" - ] - } - ], - "output": [] -} diff --git a/src/logstashui/Common/tests/conversion_data/components/test-comments-mixed.json b/src/logstashui/Common/tests/conversion_data/components/test-comments-mixed.json deleted file mode 100644 index 651f5f2..0000000 --- a/src/logstashui/Common/tests/conversion_data/components/test-comments-mixed.json +++ /dev/null @@ -1,154 +0,0 @@ -{ - "input": [ - { - "id": "input_comment_0", - "type": "input", - "plugin": "comment", - "config": { - "text": "inline on section opener" - } - }, - { - "id": "input_udp_1", - "type": "input", - "plugin": "udp", - "config": { - "port": 5140, - "buffer_size": 65536, - "tags": [ - "syslog" - ] - }, - "comments": [ - "standalone inside plugin", - "add_field => {\"commented_out\" => \"value\"} standalone with braces", - "inline on plugin opener", - "inline on scalar value", - "inline on array" - ] - }, - { - "id": "input_comment_2", - "type": "input", - "plugin": "comment", - "config": { - "text": "inline on plugin closer -> section comment" - } - } - ], - "filter": [ - { - "id": "filter_comment_3", - "type": "filter", - "plugin": "comment", - "config": { - "text": "inline on filter opener\nstandalone at section level" - } - }, - { - "id": "filter_mutate_4", - "type": "filter", - "plugin": "mutate", - "config": { - "add_field": { - "key1": "value1", - "key2": "value2" - }, - "remove_field": [ - "old" - ] - }, - "comments": [ - "standalone before first pair", - "standalone between pairs", - "standalone at end of plugin", - "inline on mutate opener", - "inline on hash opener", - "inline on hash pair", - "inline on hash closer", - "inline on array pair" - ] - }, - { - "id": "filter_comment_5", - "type": "filter", - "plugin": "comment", - "config": { - "text": "inline on plugin closer -> section comment\nstandalone between plugins" - } - }, - { - "id": "filter_if_6", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[type] == \"web\"", - "plugins": [ - { - "id": "filter_comment_7", - "type": "filter", - "plugin": "comment", - "config": { - "text": "standalone inside conditional" - } - }, - { - "id": "filter_grok_8", - "type": "filter", - "plugin": "grok", - "config": { - "match": { - "message": "%{COMBINEDAPACHELOG}" - } - }, - "comments": [ - "standalone inside nested plugin", - "match => {\"message\" => \"%{GREEDYDATA}\"} standalone with braces in conditional", - "inline on nested plugin opener", - "inline inside nested hash", - "inline on nested hash closer" - ] - }, - { - "id": "filter_comment_9", - "type": "filter", - "plugin": "comment", - "config": { - "text": "inline on nested plugin closer" - } - }, - { - "id": "filter_comment_10", - "type": "filter", - "plugin": "comment", - "config": { - "text": "standalone at end of conditional block" - } - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_drop_11", - "type": "filter", - "plugin": "drop", - "config": {}, - "comments": [] - } - ], - "output": [ - { - "id": "output_stdout_12", - "type": "output", - "plugin": "stdout", - "config": { - "codec": { - "rubydebug": {} - } - }, - "comments": [] - } - ] -} diff --git a/src/logstashui/Common/tests/conversion_data/components/test-comments-plugin-inline.json b/src/logstashui/Common/tests/conversion_data/components/test-comments-plugin-inline.json deleted file mode 100644 index 7d04adf..0000000 --- a/src/logstashui/Common/tests/conversion_data/components/test-comments-plugin-inline.json +++ /dev/null @@ -1,111 +0,0 @@ -{ - "input": [ - { - "id": "input_udp_0", - "type": "input", - "plugin": "udp", - "config": { - "port": 5140, - "buffer_size": 65536, - "tags": [ - "udp", - "syslog" - ] - }, - "comments": [ - "inline comment on plugin opener", - "inline comment on a scalar value", - "another scalar inline comment", - "inline comment after an array" - ] - }, - { - "id": "input_comment_1", - "type": "input", - "plugin": "comment", - "config": { - "text": "inline comment on plugin closer \u2014 becomes section-level comment" - } - } - ], - "filter": [ - { - "id": "filter_mutate_2", - "type": "filter", - "plugin": "mutate", - "config": { - "add_field": { - "first": "value1", - "second": "value2" - }, - "remove_field": [ - "unwanted", - "junk" - ], - "rename": { - "old_name": "new_name" - } - }, - "comments": [ - "opener comment", - "inline comment on hash opener", - "inline comment on hash pair", - "another hash pair comment", - "inline comment on hash closer", - "inline on array", - "another hash opener with inline", - "pair comment", - "hash closer comment" - ] - }, - { - "id": "filter_comment_3", - "type": "filter", - "plugin": "comment", - "config": { - "text": "plugin closer \u2014 section-level" - } - }, - { - "id": "filter_drop_4", - "type": "filter", - "plugin": "drop", - "config": {}, - "comments": [ - "opener comment on empty plugin" - ] - }, - { - "id": "filter_comment_5", - "type": "filter", - "plugin": "comment", - "config": { - "text": "closer comment on empty plugin" - } - } - ], - "output": [ - { - "id": "output_stdout_6", - "type": "output", - "plugin": "stdout", - "config": { - "codec": { - "rubydebug": {} - } - }, - "comments": [ - "output plugin with inline", - "inline on codec line" - ] - }, - { - "id": "output_comment_7", - "type": "output", - "plugin": "comment", - "config": { - "text": "closer" - } - } - ] -} diff --git a/src/logstashui/Common/tests/conversion_data/components/test-comments-section-opener.json b/src/logstashui/Common/tests/conversion_data/components/test-comments-section-opener.json deleted file mode 100644 index b580988..0000000 --- a/src/logstashui/Common/tests/conversion_data/components/test-comments-section-opener.json +++ /dev/null @@ -1,100 +0,0 @@ -{ - "input": [ - { - "id": "input_comment_0", - "type": "input", - "plugin": "comment", - "config": { - "text": "inline comment on input opener" - } - }, - { - "id": "input_beats_1", - "type": "input", - "plugin": "beats", - "config": { - "port": 5044 - }, - "comments": [ - "inline on beats opener" - ] - } - ], - "filter": [ - { - "id": "filter_comment_2", - "type": "filter", - "plugin": "comment", - "config": { - "text": "inline comment on filter opener" - } - }, - { - "id": "filter_mutate_3", - "type": "filter", - "plugin": "mutate", - "config": { - "add_field": { - "processed": "true" - } - }, - "comments": [] - }, - { - "id": "filter_comment_4", - "type": "filter", - "plugin": "comment", - "config": { - "text": "standalone comment between plugins at section level" - } - }, - { - "id": "filter_drop_5", - "type": "filter", - "plugin": "drop", - "config": {}, - "comments": [] - } - ], - "output": [ - { - "id": "output_comment_6", - "type": "output", - "plugin": "comment", - "config": { - "text": "inline comment on output opener" - } - }, - { - "id": "output_elasticsearch_7", - "type": "output", - "plugin": "elasticsearch", - "config": { - "hosts": [ - "localhost:9200" - ], - "index": "logs-%{+YYYY.MM.dd}" - }, - "comments": [] - }, - { - "id": "output_comment_8", - "type": "output", - "plugin": "comment", - "config": { - "text": "standalone at section level before second output plugin" - } - }, - { - "id": "output_stdout_9", - "type": "output", - "plugin": "stdout", - "config": { - "codec": { - "rubydebug": {} - } - }, - "comments": [] - } - ] -} diff --git a/src/logstashui/Common/tests/conversion_data/components/test-comments-standalone-in-plugin.json b/src/logstashui/Common/tests/conversion_data/components/test-comments-standalone-in-plugin.json deleted file mode 100644 index d7d2a80..0000000 --- a/src/logstashui/Common/tests/conversion_data/components/test-comments-standalone-in-plugin.json +++ /dev/null @@ -1,61 +0,0 @@ -{ - "input": [], - "filter": [ - { - "id": "filter_mutate_0", - "type": "filter", - "plugin": "mutate", - "config": { - "add_field": { - "field1": "value1" - }, - "remove_field": [ - "junk" - ] - }, - "comments": [ - "This is a standalone comment at the top of a plugin block", - "Standalone comment in the middle of a plugin block", - "Another standalone at the bottom" - ] - }, - { - "id": "filter_grok_1", - "type": "filter", - "plugin": "grok", - "config": { - "match": { - "message": "%{COMBINEDAPACHELOG}" - } - }, - "comments": [ - "Standalone comment before the only config key", - "Standalone at the end of plugin" - ] - }, - { - "id": "filter_comment_2", - "type": "filter", - "plugin": "comment", - "config": { - "text": "Section-level comment between plugins" - } - }, - { - "id": "filter_mutate_3", - "type": "filter", - "plugin": "mutate", - "config": { - "uppercase": [ - "log_level" - ] - }, - "comments": [ - "Leading standalone", - "Second leading standalone", - "Trailing standalone" - ] - } - ], - "output": [] -} diff --git a/src/logstashui/Common/tests/conversion_data/components/test-complex2.json b/src/logstashui/Common/tests/conversion_data/components/test-complex2.json deleted file mode 100644 index 5756543..0000000 --- a/src/logstashui/Common/tests/conversion_data/components/test-complex2.json +++ /dev/null @@ -1,952 +0,0 @@ -{ - "input": [ - { - "id": "input_comment_0", - "type": "input", - "plugin": "comment", - "config": { - "text": "\n\"LogstashUI kitchen sink\" pipeline\nGoal: be extremely feature-rich while staying within known-valid plugin options.\n\nBeats / Elastic Agent style shippers" - } - }, - { - "id": "input_beats_1", - "type": "input", - "plugin": "beats", - "config": { - "id": "in_beats_5044", - "port": "5044", - "add_field": { - "ingest_transport": "beats" - }, - "tags": [ - "from_beats" - ] - }, - "comments": [] - }, - { - "id": "input_comment_2", - "type": "input", - "plugin": "comment", - "config": { - "text": "JSON-over-TCP (common for app logs)" - } - }, - { - "id": "input_tcp_3", - "type": "input", - "plugin": "tcp", - "config": { - "id": "in_tcp_json_5514", - "port": 5514, - "mode": "server", - "codec": { - "json": {} - }, - "add_field": { - "ingest_transport": "tcp" - }, - "tags": [ - "from_tcp" - ] - }, - "comments": [] - }, - { - "id": "input_comment_4", - "type": "input", - "plugin": "comment", - "config": { - "text": "Syslog-ish UDP" - } - }, - { - "id": "input_udp_5", - "type": "input", - "plugin": "udp", - "config": { - "id": "in_udp_5515", - "port": 5515, - "codec": { - "plain": {} - }, - "add_field": { - "ingest_transport": "udp" - }, - "tags": [ - "from_udp" - ] - }, - "comments": [] - }, - { - "id": "input_comment_6", - "type": "input", - "plugin": "comment", - "config": { - "text": "HTTP event intake (webhooks, apps posting JSON, etc.)" - } - }, - { - "id": "input_http_7", - "type": "input", - "plugin": "http", - "config": { - "id": "in_http_8080", - "port": 8080, - "codec": { - "json": {} - }, - "add_field": { - "ingest_transport": "http" - }, - "tags": [ - "from_http" - ] - }, - "comments": [] - }, - { - "id": "input_comment_8", - "type": "input", - "plugin": "comment", - "config": { - "text": "Local dev/testing input" - } - }, - { - "id": "input_stdin_9", - "type": "input", - "plugin": "stdin", - "config": { - "id": "in_stdin", - "codec": { - "line": {} - }, - "add_field": { - "ingest_transport": "stdin" - }, - "tags": [ - "from_stdin" - ] - }, - "comments": [] - }, - { - "id": "input_comment_10", - "type": "input", - "plugin": "comment", - "config": { - "text": "Synthetic test data (makes it easy to validate end-to-end quickly)" - } - }, - { - "id": "input_generator_11", - "type": "input", - "plugin": "generator", - "config": { - "id": "in_generator", - "lines": [ - "Feb 21 09:12:01 host1 sshd[123]: Failed password for invalid user admin from 10.1.2.3 port 51234 ssh2", - "{\"@timestamp\":\"2026-02-21T14:12:02Z\",\"message\":\"GET /health 200\",\"source_ip\":\"8.8.8.8\",\"user_agent\":\"Mozilla/5.0\"}", - "level=info service=api latency_ms=42 source_ip=192.168.1.50 msg=\"request completed\"" - ], - "count": 1, - "add_field": { - "ingest_transport": "generator" - }, - "tags": [ - "from_generator" - ] - }, - "comments": [] - } - ], - "filter": [ - { - "id": "filter_comment_12", - "type": "filter", - "plugin": "comment", - "config": { - "text": "\nNormalize a few shared fields\n" - } - }, - { - "id": "filter_mutate_13", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "f_mutate_bootstrap", - "add_field": { - "[@metadata][pipeline]": "logstashui_kitchen_sink", - "event.module": "logstashui" - } - }, - "comments": [] - }, - { - "id": "filter_comment_14", - "type": "filter", - "plugin": "comment", - "config": { - "text": "Keep a canonical message field" - } - }, - { - "id": "filter_if_15", - "type": "filter", - "plugin": "if", - "config": { - "condition": "![message] and [event][original]", - "plugins": [ - { - "id": "filter_mutate_16", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "f_mutate_event_original_to_message", - "copy": { - "[event][original]": "message" - } - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_comment_17", - "type": "filter", - "plugin": "comment", - "config": { - "text": "\nTry to parse JSON *if* message looks like JSON (common when tcp/udp/plain feed JSON strings)\n" - } - }, - { - "id": "filter_if_18", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[message] =~ \"^[[:space:]]*\\\\{\"", - "plugins": [ - { - "id": "filter_json_19", - "type": "filter", - "plugin": "json", - "config": { - "id": "f_json_from_message", - "source": "message", - "target": "json", - "tag_on_failure": [ - "_jsonparsefailure_message" - ] - }, - "comments": [] - }, - { - "id": "filter_comment_20", - "type": "filter", - "plugin": "comment", - "config": { - "text": "If json parsed, promote a few expected keys (only if present)" - } - }, - { - "id": "filter_if_21", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[json][@timestamp]", - "plugins": [ - { - "id": "filter_mutate_22", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "f_promote_json_ts", - "copy": { - "[json][@timestamp]": "@timestamp" - } - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_if_23", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[json][source_ip]", - "plugins": [ - { - "id": "filter_mutate_24", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "f_promote_json_source_ip", - "copy": { - "[json][source_ip]": "source_ip" - } - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_if_25", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[json][user_agent]", - "plugins": [ - { - "id": "filter_mutate_26", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "f_promote_json_ua", - "copy": { - "[json][user_agent]": "user_agent" - } - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_comment_27", - "type": "filter", - "plugin": "comment", - "config": { - "text": "\nSyslog-ish parsing (UDP and some TCP)\n" - } - }, - { - "id": "filter_if_28", - "type": "filter", - "plugin": "if", - "config": { - "condition": "\"from_udp\" in [tags] or \"from_tcp\" in [tags]", - "plugins": [ - { - "id": "filter_comment_29", - "type": "filter", - "plugin": "comment", - "config": { - "text": "Try dissect first (fast) and fall back to grok" - } - }, - { - "id": "filter_dissect_30", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "f_dissect_syslogish", - "mapping": { - "message": "%{syslog_timestamp} %{syslog_host} %{syslog_program}[%{syslog_pid}]: %{syslog_message}" - }, - "tag_on_failure": [ - "_dissectfailure_syslogish" - ] - }, - "comments": [] - }, - { - "id": "filter_if_31", - "type": "filter", - "plugin": "if", - "config": { - "condition": "\"_dissectfailure_syslogish\" in [tags]", - "plugins": [ - { - "id": "filter_grok_32", - "type": "filter", - "plugin": "grok", - "config": { - "id": "f_grok_syslogish", - "match": { - "message": [ - "%{SYSLOGTIMESTAMP:syslog_timestamp} %{HOSTNAME:syslog_host} %{DATA:syslog_program}(?:\\[%{POSINT:syslog_pid}\\])?: %{GREEDYDATA:syslog_message}" - ] - }, - "tag_on_failure": [ - "_grokparsefailure_syslogish" - ] - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_comment_33", - "type": "filter", - "plugin": "comment", - "config": { - "text": "If we extracted a syslog timestamp, use it" - } - }, - { - "id": "filter_if_34", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[syslog_timestamp]", - "plugins": [ - { - "id": "filter_date_35", - "type": "filter", - "plugin": "date", - "config": { - "id": "f_date_syslog", - "match": [ - "syslog_timestamp", - "MMM d HH:mm:ss", - "MMM dd HH:mm:ss" - ], - "tag_on_failure": [ - "_dateparsefailure_syslog" - ] - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_comment_36", - "type": "filter", - "plugin": "comment", - "config": { - "text": "\nkey=value parsing for \u201cflat\u201d log lines\n" - } - }, - { - "id": "filter_if_37", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[message] =~ \"([A-Za-z0-9_.-]+)=([^\\\"]\\\\S+|\\\"[^\\\"]*\\\")\"", - "plugins": [ - { - "id": "filter_kv_38", - "type": "filter", - "plugin": "kv", - "config": { - "id": "f_kv_message", - "source": "message", - "trim_key": " ", - "trim_value": " ", - "value_split": "=", - "field_split_pattern": "\\s+", - "tag_on_failure": [ - "_kvfailure_message" - ] - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_comment_39", - "type": "filter", - "plugin": "comment", - "config": { - "text": "\nBasic typing / normalization\n" - } - }, - { - "id": "filter_mutate_40", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "f_mutate_normalize", - "rename": { - "msg": "message_short" - }, - "convert": { - "latency_ms": "integer" - }, - "lowercase": [ - "level" - ] - }, - "comments": [] - }, - { - "id": "filter_comment_41", - "type": "filter", - "plugin": "comment", - "config": { - "text": "\nEnrichments: useragent, geoip, cidr, dns\n" - } - }, - { - "id": "filter_if_42", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[user_agent]", - "plugins": [ - { - "id": "filter_useragent_43", - "type": "filter", - "plugin": "useragent", - "config": { - "id": "f_useragent", - "source": "user_agent", - "target": "user_agent_parsed" - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_comment_44", - "type": "filter", - "plugin": "comment", - "config": { - "text": "Canonicalize IP into source_ip if it exists elsewhere" - } - }, - { - "id": "filter_if_45", - "type": "filter", - "plugin": "if", - "config": { - "condition": "![source_ip] and [source][ip]", - "plugins": [ - { - "id": "filter_mutate_46", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "f_copy_source_ip", - "copy": { - "[source][ip]": "source_ip" - } - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_if_47", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[source_ip]", - "plugins": [ - { - "id": "filter_comment_48", - "type": "filter", - "plugin": "comment", - "config": { - "text": "Tag private vs public" - } - }, - { - "id": "filter_cidr_49", - "type": "filter", - "plugin": "cidr", - "config": { - "id": "f_cidr_private", - "address": [ - "%{source_ip}" - ], - "network": [ - "10.0.0.0/8", - "172.16.0.0/12", - "192.168.0.0/16" - ], - "add_tag": [ - "src_private" - ] - }, - "comments": [] - }, - { - "id": "filter_comment_50", - "type": "filter", - "plugin": "comment", - "config": { - "text": "GeoIP typically only makes sense for public IPs, so do it only if not private-tagged" - } - }, - { - "id": "filter_if_51", - "type": "filter", - "plugin": "if", - "config": { - "condition": "\"src_private\" not in [tags]", - "plugins": [ - { - "id": "filter_geoip_52", - "type": "filter", - "plugin": "geoip", - "config": { - "id": "f_geoip", - "source": "source_ip", - "target": "source_geo" - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_comment_53", - "type": "filter", - "plugin": "comment", - "config": { - "text": "Reverse DNS lookup; replace source_ip with hostname when possible (or leave as-is)" - } - }, - { - "id": "filter_dns_54", - "type": "filter", - "plugin": "dns", - "config": { - "id": "f_dns_reverse", - "reverse": [ - "source_ip" - ], - "action": "replace" - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_comment_55", - "type": "filter", - "plugin": "comment", - "config": { - "text": "\nTranslate severity/level into a normalized numeric\n" - } - }, - { - "id": "filter_translate_56", - "type": "filter", - "plugin": "translate", - "config": { - "id": "f_translate_level_to_severity", - "source": "level", - "target": "severity", - "dictionary": { - "trace": "0", - "debug": "1", - "info": "2", - "warn": "3", - "error": "4", - "fatal": "5" - }, - "fallback": "2" - }, - "comments": [] - }, - { - "id": "filter_mutate_57", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "f_convert_severity_int", - "convert": { - "severity": "integer" - } - }, - "comments": [] - }, - { - "id": "filter_comment_58", - "type": "filter", - "plugin": "comment", - "config": { - "text": "\nStable fingerprint for dedup / correlation\n" - } - }, - { - "id": "filter_fingerprint_59", - "type": "filter", - "plugin": "fingerprint", - "config": { - "id": "f_fingerprint_message", - "source": [ - "message" - ], - "method": "MURMUR3", - "target": "[@metadata][fingerprint]" - }, - "comments": [] - }, - { - "id": "filter_comment_60", - "type": "filter", - "plugin": "comment", - "config": { - "text": "\nExample branching: treat auth-ish messages specially\n" - } - }, - { - "id": "filter_if_61", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[syslog_program] == \"sshd\" or [message] =~ \"(?i)failed password|authentication failure|invalid user\"", - "plugins": [ - { - "id": "filter_mutate_62", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "f_tag_auth", - "add_tag": [ - "category_auth" - ], - "add_field": { - "event.category": "authentication" - } - }, - "comments": [] - } - ], - "else_ifs": [ - { - "condition": "[message] =~ \"(?i)GET\\\\s+/health|/ready|/live\"", - "plugins": [ - { - "id": "filter_mutate_63", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "f_tag_health", - "add_tag": [ - "category_healthcheck" - ], - "add_field": { - "event.category": "availability" - } - }, - "comments": [] - } - ] - } - ], - "else": { - "plugins": [ - { - "id": "filter_mutate_64", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "f_tag_generic", - "add_tag": [ - "category_generic" - ] - }, - "comments": [] - } - ] - } - } - }, - { - "id": "filter_comment_65", - "type": "filter", - "plugin": "comment", - "config": { - "text": "\nPrune down noisy fields (keeps top-level essentials)\n" - } - }, - { - "id": "filter_prune_66", - "type": "filter", - "plugin": "prune", - "config": { - "id": "f_prune", - "whitelist_names": [ - "^@timestamp$", - "^message$", - "^message_short$", - "^host$", - "^source_ip$", - "^source_geo$", - "^severity$", - "^level$", - "^tags$", - "^event\\..*$", - "^user_agent.*$", - "^syslog_.*$", - "^ingest_transport$" - ] - }, - "comments": [] - } - ], - "output": [ - { - "id": "output_comment_67", - "type": "output", - "plugin": "comment", - "config": { - "text": "Always see something in console during dev" - } - }, - { - "id": "output_stdout_68", - "type": "output", - "plugin": "stdout", - "config": { - "id": "out_stdout_rubydebug", - "codec": { - "rubydebug": { - "metadata": "true" - } - } - }, - "comments": [] - }, - { - "id": "output_comment_69", - "type": "output", - "plugin": "comment", - "config": { - "text": "Write to disk (great for debugging replay)" - } - }, - { - "id": "output_file_70", - "type": "output", - "plugin": "file", - "config": { - "id": "out_file_jsonl", - "path": "/tmp/logstashui-%{+YYYY.MM.dd}.jsonl", - "codec": { - "json_lines": {} - } - }, - "comments": [] - }, - { - "id": "output_comment_71", - "type": "output", - "plugin": "comment", - "config": { - "text": "Elasticsearch (local default)" - } - }, - { - "id": "output_elasticsearch_72", - "type": "output", - "plugin": "elasticsearch", - "config": { - "id": "out_es_local", - "hosts": [ - "http://localhost:9200" - ], - "index": "logstashui-%{+YYYY.MM.dd}", - "ilm_enabled": "false" - }, - "comments": [] - }, - { - "id": "output_comment_73", - "type": "output", - "plugin": "comment", - "config": { - "text": "Webhook back to your UI/API (example)" - } - }, - { - "id": "output_http_74", - "type": "output", - "plugin": "http", - "config": { - "id": "out_http_callback", - "url": "http://localhost:9000/logstash/callback", - "http_method": "post", - "format": "json" - }, - "comments": [] - }, - { - "id": "output_comment_75", - "type": "output", - "plugin": "comment", - "config": { - "text": "Kafka (example)" - } - }, - { - "id": "output_kafka_76", - "type": "output", - "plugin": "kafka", - "config": { - "id": "out_kafka", - "bootstrap_servers": "localhost:9092", - "topic_id": "logstashui-events" - }, - "comments": [] - }, - { - "id": "output_comment_77", - "type": "output", - "plugin": "comment", - "config": { - "text": "Pipeline-to-pipeline (requires another pipeline with pipeline input address => \"downstream\")" - } - }, - { - "id": "output_pipeline_78", - "type": "output", - "plugin": "pipeline", - "config": { - "id": "out_pipeline_downstream", - "send_to": [ - "downstream" - ] - }, - "comments": [] - } - ] -} diff --git a/src/logstashui/Common/tests/conversion_data/components/test-complex3.json b/src/logstashui/Common/tests/conversion_data/components/test-complex3.json deleted file mode 100644 index 1771e5a..0000000 --- a/src/logstashui/Common/tests/conversion_data/components/test-complex3.json +++ /dev/null @@ -1,1497 +0,0 @@ -{ - "input": [ - { - "id": "input_beats_0", - "type": "input", - "plugin": "beats", - "config": { - "port": "5044", - "ssl": "true", - "ssl_certificate": "/etc/logstash/certs/server.crt", - "ssl_key": "/etc/logstash/certs/server.key", - "ssl_verify_mode": "force_peer", - "ssl_certificate_authorities": [ - "/etc/logstash/certs/ca.crt" - ], - "codec": { - "json": {} - }, - "type": "beats", - "tags": [ - "beats_ssl" - ] - }, - "comments": [] - }, - { - "id": "input_http_1", - "type": "input", - "plugin": "http", - "config": { - "port": "8080", - "codec": { - "json": {} - }, - "ssl": "true", - "ssl_certificate": "/etc/logstash/certs/http.crt", - "ssl_key": "/etc/logstash/certs/http.key", - "threads": "4", - "max_pending_requests": "100", - "response_headers": { - "Content-Type": "application/json" - }, - "type": "webhook", - "tags": [ - "http_api" - ] - }, - "comments": [] - }, - { - "id": "input_kafka_2", - "type": "input", - "plugin": "kafka", - "config": { - "bootstrap_servers": "kafka1:9092,kafka2:9092,kafka3:9092", - "topics": [ - "app-logs", - "security-events", - "metrics" - ], - "group_id": "logstash-consumer", - "consumer_threads": "3", - "codec": { - "avro": { - "schema_uri": "http://schema-registry:8081/schemas/ids/1" - } - }, - "decorate_events": "true", - "security_protocol": "SASL_SSL", - "sasl_mechanism": "SCRAM-SHA-512", - "sasl_jaas_config": "org.apache.kafka.common.security.scram.ScramLoginModule required username='logstash' password='${KAFKA_PASS}';", - "type": "kafka", - "tags": [ - "kafka_stream" - ] - }, - "comments": [] - }, - { - "id": "input_jdbc_3", - "type": "input", - "plugin": "jdbc", - "config": { - "jdbc_driver_library": "/usr/share/logstash/vendor/jar/jdbc/postgresql.jar", - "jdbc_driver_class": "org.postgresql.Driver", - "jdbc_connection_string": "jdbc:postgresql://db:5432/prod", - "jdbc_user": "${DB_USER}", - "jdbc_password": "${DB_PASS}", - "schedule": "*/5 * * * *", - "statement": "SELECT * FROM events WHERE created_at > :sql_last_value", - "use_column_value": "true", - "tracking_column": "created_at", - "tracking_column_type": "timestamp", - "type": "database", - "tags": [ - "jdbc_poll" - ] - }, - "comments": [] - }, - { - "id": "input_file_4", - "type": "input", - "plugin": "file", - "config": { - "path": [ - "/var/log/nginx/*.log", - "/var/log/app/**/*.log" - ], - "start_position": "beginning", - "sincedb_path": "/var/lib/logstash/sincedb", - "codec": { - "multiline": { - "pattern": "^%{TIMESTAMP_ISO8601}", - "negate": "true", - "what": "previous", - "max_lines": 500 - } - }, - "type": "file", - "tags": [ - "file_input" - ] - }, - "comments": [] - }, - { - "id": "input_tcp_5", - "type": "input", - "plugin": "tcp", - "config": { - "port": "5000", - "codec": { - "json_lines": {} - }, - "ssl_enable": "true", - "ssl_cert": "/etc/logstash/certs/tcp.crt", - "ssl_key": "/etc/logstash/certs/tcp.key", - "type": "tcp_json", - "tags": [ - "tcp_secure" - ] - }, - "comments": [] - }, - { - "id": "input_udp_6", - "type": "input", - "plugin": "udp", - "config": { - "port": "514", - "codec": { - "cef": {} - }, - "type": "syslog", - "tags": [ - "syslog_udp" - ] - }, - "comments": [] - }, - { - "id": "input_rabbitmq_7", - "type": "input", - "plugin": "rabbitmq", - "config": { - "host": "rabbitmq", - "port": "5672", - "user": "${RABBIT_USER}", - "password": "${RABBIT_PASS}", - "queue": "logs", - "exchange": "logs-exchange", - "exchange_type": "topic", - "key": "logs.#", - "durable": "true", - "codec": { - "json": {} - }, - "type": "rabbitmq", - "tags": [ - "amqp" - ] - }, - "comments": [] - }, - { - "id": "input_redis_8", - "type": "input", - "plugin": "redis", - "config": { - "host": "redis", - "port": "6379", - "password": "${REDIS_PASS}", - "data_type": "list", - "key": "logstash:queue", - "codec": { - "json": {} - }, - "type": "redis", - "tags": [ - "redis_queue" - ] - }, - "comments": [] - }, - { - "id": "input_s3_9", - "type": "input", - "plugin": "s3", - "config": { - "bucket": "logs-archive", - "region": "us-east-1", - "access_key_id": "${AWS_KEY}", - "secret_access_key": "${AWS_SECRET}", - "interval": "300", - "codec": { - "json_lines": {} - }, - "type": "s3", - "tags": [ - "s3_archive" - ] - }, - "comments": [] - }, - { - "id": "input_kinesis_10", - "type": "input", - "plugin": "kinesis", - "config": { - "kinesis_stream_name": "app-stream", - "region": "us-west-2", - "codec": { - "json": {} - }, - "type": "kinesis", - "tags": [ - "aws_kinesis" - ] - }, - "comments": [] - } - ], - "filter": [ - { - "id": "filter_if_11", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[type] == \"beats\"", - "plugins": [ - { - "id": "filter_if_12", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[agent][type] == \"filebeat\"", - "plugins": [ - { - "id": "filter_if_13", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[log][file][path] =~ /nginx/", - "plugins": [ - { - "id": "filter_grok_14", - "type": "filter", - "plugin": "grok", - "config": { - "match": { - "message": "%{IPORHOST:client_ip} - %{DATA:user} \\[%{HTTPDATE:timestamp}\\] \"%{WORD:method} %{DATA:path} HTTP/%{NUMBER:version}\" %{NUMBER:status:int} %{NUMBER:bytes:int} \"%{DATA:referrer}\" \"%{DATA:agent}\"" - } - }, - "comments": [] - }, - { - "id": "filter_date_15", - "type": "filter", - "plugin": "date", - "config": { - "match": [ - "timestamp", - "dd/MMM/yyyy:HH:mm:ss Z" - ], - "target": "@timestamp" - }, - "comments": [] - }, - { - "id": "filter_useragent_16", - "type": "filter", - "plugin": "useragent", - "config": { - "source": "agent", - "target": "ua" - }, - "comments": [] - }, - { - "id": "filter_geoip_17", - "type": "filter", - "plugin": "geoip", - "config": { - "source": "client_ip", - "target": "geo", - "database": "/usr/share/GeoIP/GeoLite2-City.mmdb" - }, - "comments": [] - }, - { - "id": "filter_if_18", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[status] >= 500", - "plugins": [ - { - "id": "filter_mutate_19", - "type": "filter", - "plugin": "mutate", - "config": { - "add_tag": [ - "error", - "server_error" - ], - "add_field": { - "severity": "critical" - } - }, - "comments": [] - } - ], - "else_ifs": [ - { - "condition": "[status] >= 400", - "plugins": [ - { - "id": "filter_mutate_20", - "type": "filter", - "plugin": "mutate", - "config": { - "add_tag": [ - "error", - "client_error" - ], - "add_field": { - "severity": "warning" - } - }, - "comments": [] - } - ] - } - ], - "else": null - } - }, - { - "id": "filter_ruby_21", - "type": "filter", - "plugin": "ruby", - "config": { - "code": "\n bytes = event.get(\"bytes\").to_i\n if bytes > 10485760\n event.set(\"size_class\", \"large\")\n elsif bytes > 1048576\n event.set(\"size_class\", \"medium\")\n else\n event.set(\"size_class\", \"small\")\n end\n " - }, - "comments": [] - }, - { - "id": "filter_fingerprint_22", - "type": "filter", - "plugin": "fingerprint", - "config": { - "source": [ - "client_ip", - "path", - "timestamp" - ], - "target": "[@metadata][fingerprint]", - "method": "SHA256" - }, - "comments": [] - } - ], - "else_ifs": [ - { - "condition": "[log][file][path] =~ /application/", - "plugins": [ - { - "id": "filter_json_23", - "type": "filter", - "plugin": "json", - "config": { - "source": "message", - "target": "app" - }, - "comments": [] - }, - { - "id": "filter_if_24", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[app][level]", - "plugins": [ - { - "id": "filter_translate_25", - "type": "filter", - "plugin": "translate", - "config": { - "field": "[app][level]", - "destination": "severity_num", - "dictionary": { - "DEBUG": "1", - "INFO": "2", - "WARN": "3", - "ERROR": "4", - "FATAL": "5" - }, - "fallback": "2" - }, - "comments": [] - }, - { - "id": "filter_mutate_26", - "type": "filter", - "plugin": "mutate", - "config": { - "convert": { - "severity_num": "integer" - } - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_if_27", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[app][exception]", - "plugins": [ - { - "id": "filter_mutate_28", - "type": "filter", - "plugin": "mutate", - "config": { - "add_tag": [ - "exception" - ] - }, - "comments": [] - }, - { - "id": "filter_ruby_29", - "type": "filter", - "plugin": "ruby", - "config": { - "code": "\n exc = event.get(\"[app][exception]\")\n if exc.is_a?(Hash)\n event.set(\"exception_class\", exc[\"class\"])\n event.set(\"exception_msg\", exc[\"message\"])\n end\n " - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - } - ] - } - ], - "else": null - } - } - ], - "else_ifs": [ - { - "condition": "[agent][type] == \"metricbeat\"", - "plugins": [ - { - "id": "filter_if_30", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[system][cpu]", - "plugins": [ - { - "id": "filter_ruby_31", - "type": "filter", - "plugin": "ruby", - "config": { - "code": "\n cpu = event.get(\"[system][cpu]\")\n if cpu && cpu[\"cores\"]\n total = 0.0\n cpu[\"cores\"].each { |c| total += c[\"user\"][\"pct\"].to_f if c[\"user\"] }\n avg = total / cpu[\"cores\"].length\n event.set(\"[system][cpu][avg_pct]\", avg.round(2))\n event.tag(\"cpu_warning\") if avg > 75\n event.tag(\"cpu_critical\") if avg > 90\n end\n " - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_if_32", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[system][memory][actual][used][pct]", - "plugins": [ - { - "id": "filter_ruby_33", - "type": "filter", - "plugin": "ruby", - "config": { - "code": "\n pct = event.get(\"[system][memory][actual][used][pct]\").to_f * 100\n event.set(\"mem_used_pct\", pct.round(2))\n event.tag(\"memory_warning\") if pct > 85\n event.tag(\"memory_critical\") if pct > 95\n " - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - } - ] - } - ], - "else": null - } - } - ], - "else_ifs": [ - { - "condition": "[type] == \"kafka\"", - "plugins": [ - { - "id": "filter_if_34", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[kafka][topic] == \"security-events\"", - "plugins": [ - { - "id": "filter_json_35", - "type": "filter", - "plugin": "json", - "config": { - "source": "message", - "target": "security" - }, - "comments": [] - }, - { - "id": "filter_if_36", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[security][ip]", - "plugins": [ - { - "id": "filter_cidr_37", - "type": "filter", - "plugin": "cidr", - "config": { - "address": [ - "%{[security][ip]}" - ], - "network": [ - "10.0.0.0/8", - "172.16.0.0/12", - "192.168.0.0/16" - ], - "add_tag": [ - "internal_ip" - ] - }, - "comments": [] - }, - { - "id": "filter_if_38", - "type": "filter", - "plugin": "if", - "config": { - "condition": "\"internal_ip\" not in [tags]", - "plugins": [ - { - "id": "filter_geoip_39", - "type": "filter", - "plugin": "geoip", - "config": { - "source": "[security][ip]", - "target": "threat_geo", - "database": "/usr/share/GeoIP/GeoLite2-City.mmdb" - }, - "comments": [] - }, - { - "id": "filter_geoip_40", - "type": "filter", - "plugin": "geoip", - "config": { - "source": "[security][ip]", - "target": "threat_asn", - "database": "/usr/share/GeoIP/GeoLite2-ASN.mmdb", - "default_database_type": "ASN" - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_if_41", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[security][event_type]", - "plugins": [ - { - "id": "filter_translate_42", - "type": "filter", - "plugin": "translate", - "config": { - "field": "[security][event_type]", - "destination": "threat_score", - "dictionary": { - "brute_force": "75", - "sql_injection": "90", - "xss": "85", - "unauthorized": "80", - "privilege_escalation": "95", - "malware": "100" - }, - "fallback": "50" - }, - "comments": [] - }, - { - "id": "filter_mutate_43", - "type": "filter", - "plugin": "mutate", - "config": { - "convert": { - "threat_score": "integer" - } - }, - "comments": [] - }, - { - "id": "filter_if_44", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[threat_score] >= 90", - "plugins": [ - { - "id": "filter_mutate_45", - "type": "filter", - "plugin": "mutate", - "config": { - "add_tag": [ - "critical_threat" - ] - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_ruby_46", - "type": "filter", - "plugin": "ruby", - "config": { - "code": "\n score = event.get(\"threat_score\").to_i\n is_ext = event.get(\"tags\").include?(\"internal_ip\") ? 0 : 20\n composite = score + is_ext\n event.set(\"composite_risk\", [composite, 100].min)\n\t\t\t\t\t\t\n if composite >= 100\n event.set(\"risk\", \"critical\")\n elsif composite >= 80\n event.set(\"risk\", \"high\")\n elsif composite >= 60\n event.set(\"risk\", \"medium\")\n else\n event.set(\"risk\", \"low\")\n end\n " - }, - "comments": [] - } - ], - "else_ifs": [ - { - "condition": "[kafka][topic] == \"metrics\"", - "plugins": [ - { - "id": "filter_dissect_47", - "type": "filter", - "plugin": "dissect", - "config": { - "mapping": { - "metric": "%{env}.%{dc}.%{host}.%{service}.%{type}.%{name}" - } - }, - "comments": [] - }, - { - "id": "filter_if_48", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[type] == \"response_time\"", - "plugins": [ - { - "id": "filter_ruby_49", - "type": "filter", - "plugin": "ruby", - "config": { - "code": "\n val = event.get(\"value\").to_f\n if val > 5000\n event.set(\"perf_status\", \"critical\")\n elsif val > 2000\n event.set(\"perf_status\", \"slow\")\n else\n event.set(\"perf_status\", \"normal\")\n end\n " - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_aggregate_50", - "type": "filter", - "plugin": "aggregate", - "config": { - "task_id": "%{service}_%{name}", - "code": "\n map[\"count\"] ||= 0\n map[\"sum\"] ||= 0.0\n map[\"min\"] ||= Float::INFINITY\n map[\"max\"] ||= -Float::INFINITY\n\t\t\t\t\t\t\n val = event.get(\"value\").to_f\n map[\"count\"] += 1\n map[\"sum\"] += val\n map[\"min\"] = [map[\"min\"], val].min\n map[\"max\"] = [map[\"max\"], val].max\n\t\t\t\t\t\t\n avg = map[\"sum\"] / map[\"count\"]\n event.set(\"rolling_avg\", avg.round(2))\n event.set(\"rolling_min\", map[\"min\"])\n event.set(\"rolling_max\", map[\"max\"])\n ", - "timeout": "300" - }, - "comments": [] - } - ] - } - ], - "else": null - } - } - ] - }, - { - "condition": "[type] == \"database\"", - "plugins": [ - { - "id": "filter_if_51", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[event_data]", - "plugins": [ - { - "id": "filter_json_52", - "type": "filter", - "plugin": "json", - "config": { - "source": "event_data", - "target": "evt" - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_date_53", - "type": "filter", - "plugin": "date", - "config": { - "match": [ - "created_at", - "ISO8601", - "yyyy-MM-dd HH:mm:ss" - ], - "target": "@timestamp" - }, - "comments": [] - }, - { - "id": "filter_elasticsearch_54", - "type": "filter", - "plugin": "elasticsearch", - "config": { - "hosts": [ - "http://elasticsearch:9200" - ], - "index": "user-profiles", - "query_template": "user_lookup.json", - "fields": { - "department": "user_dept", - "role": "user_role" - } - }, - "comments": [] - }, - { - "id": "filter_if_55", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[event_type] =~ /^(login|logout|password_change)$/", - "plugins": [ - { - "id": "filter_mutate_56", - "type": "filter", - "plugin": "mutate", - "config": { - "add_tag": [ - "auth_event" - ] - }, - "comments": [] - } - ], - "else_ifs": [ - { - "condition": "[event_type] =~ /^(create|update|delete)$/", - "plugins": [ - { - "id": "filter_mutate_57", - "type": "filter", - "plugin": "mutate", - "config": { - "add_tag": [ - "data_operation" - ] - }, - "comments": [] - } - ] - } - ], - "else": null - } - } - ] - }, - { - "condition": "[type] == \"webhook\"", - "plugins": [ - { - "id": "filter_if_58", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[headers][user-agent]", - "plugins": [ - { - "id": "filter_useragent_59", - "type": "filter", - "plugin": "useragent", - "config": { - "source": "[headers][user-agent]", - "target": "webhook_ua" - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_if_60", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[headers][x-signature]", - "plugins": [ - { - "id": "filter_ruby_61", - "type": "filter", - "plugin": "ruby", - "config": { - "init": "require \"openssl\"", - "code": "\n sig = event.get(\"[headers][x-signature]\")\n payload = event.get(\"message\").to_s\n secret = ENV[\"WEBHOOK_SECRET\"]\n expected = \"sha256=\" + OpenSSL::HMAC.hexdigest(\"SHA256\", secret, payload)\n\t\t\t\t\t\t\n if sig == expected\n event.set(\"sig_valid\", true)\n else\n event.set(\"sig_valid\", false)\n event.tag(\"invalid_signature\")\n end\n " - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_aggregate_62", - "type": "filter", - "plugin": "aggregate", - "config": { - "task_id": "%{[headers][x-forwarded-for]}", - "code": "\n map[\"count\"] ||= 0\n map[\"count\"] += 1\n event.set(\"request_count\", map[\"count\"])\n ", - "timeout": "60" - }, - "comments": [] - }, - { - "id": "filter_if_63", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[request_count] and [request_count] > 100", - "plugins": [ - { - "id": "filter_mutate_64", - "type": "filter", - "plugin": "mutate", - "config": { - "add_tag": [ - "rate_limit_exceeded" - ] - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - } - ] - } - ], - "else": null - } - }, - { - "id": "filter_if_65", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[message] =~ /=/", - "plugins": [ - { - "id": "filter_kv_66", - "type": "filter", - "plugin": "kv", - "config": { - "source": "message", - "field_split": "&", - "value_split": "=", - "target": "parsed" - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_if_67", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[user_agent] and ![ua]", - "plugins": [ - { - "id": "filter_useragent_68", - "type": "filter", - "plugin": "useragent", - "config": { - "source": "user_agent", - "target": "ua" - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_if_69", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[source_ip]", - "plugins": [ - { - "id": "filter_dns_70", - "type": "filter", - "plugin": "dns", - "config": { - "reverse": [ - "source_ip" - ], - "action": "append", - "nameserver": [ - "8.8.8.8" - ], - "hit_cache_size": "10000", - "hit_cache_ttl": "3600" - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_prune_71", - "type": "filter", - "plugin": "prune", - "config": { - "whitelist_names": [ - "^@", - "^_", - "type", - "tags", - "message" - ] - }, - "comments": [] - }, - { - "id": "filter_mutate_72", - "type": "filter", - "plugin": "mutate", - "config": { - "add_field": { - "env": "${ENVIRONMENT:prod}", - "cluster": "${CLUSTER:default}" - }, - "remove_field": [ - "@version" - ] - }, - "comments": [] - }, - { - "id": "filter_if_73", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[env] == \"production\" and ([tags] and \"debug\" in [tags])", - "plugins": [ - { - "id": "filter_drop_74", - "type": "filter", - "plugin": "drop", - "config": {}, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_throttle_75", - "type": "filter", - "plugin": "throttle", - "config": { - "before_count": "3", - "after_count": "1", - "period": "60", - "key": "%{fingerprint}", - "add_tag": [ - "throttled" - ] - }, - "comments": [] - }, - { - "id": "filter_if_76", - "type": "filter", - "plugin": "if", - "config": { - "condition": "\"critical_threat\" in [tags]", - "plugins": [ - { - "id": "filter_clone_77", - "type": "filter", - "plugin": "clone", - "config": { - "clones": [ - "siem" - ], - "add_field": { - "cloned": "true" - } - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_metrics_78", - "type": "filter", - "plugin": "metrics", - "config": { - "meter": [ - "events" - ], - "add_tag": [ - "metric" - ], - "flush_interval": "30", - "rates": [ - 1, - 5, - 15 - ] - }, - "comments": [] - }, - { - "id": "filter_if_79", - "type": "filter", - "plugin": "if", - "config": { - "condition": "([severity] == \"critical\" or [severity] == \"error\") and ([status] >= 500 or [threat_score] >= 90)", - "plugins": [ - { - "id": "filter_mutate_80", - "type": "filter", - "plugin": "mutate", - "config": { - "add_field": { - "priority": "P1", - "oncall": "true" - }, - "add_tag": [ - "p1" - ] - }, - "comments": [] - } - ], - "else_ifs": [ - { - "condition": "([severity] == \"warning\") and ([status] >= 400 or [threat_score] >= 70)", - "plugins": [ - { - "id": "filter_mutate_81", - "type": "filter", - "plugin": "mutate", - "config": { - "add_field": { - "priority": "P2" - }, - "add_tag": [ - "p2" - ] - }, - "comments": [] - } - ] - } - ], - "else": null - } - }, - { - "id": "filter_fingerprint_82", - "type": "filter", - "plugin": "fingerprint", - "config": { - "source": "message", - "target": "event_hash", - "method": "MURMUR3" - }, - "comments": [] - }, - { - "id": "filter_uuid_83", - "type": "filter", - "plugin": "uuid", - "config": { - "target": "event_id" - }, - "comments": [] - }, - { - "id": "filter_ruby_84", - "type": "filter", - "plugin": "ruby", - "config": { - "code": "\n event.set(\"processed_at\", Time.now.utc.iso8601)\n event.set(\"pipeline_v\", \"2.0\")\n " - }, - "comments": [] - } - ], - "output": [ - { - "id": "output_if_85", - "type": "output", - "plugin": "if", - "config": { - "condition": "\"throttled\" not in [tags] and \"metric\" not in [tags]", - "plugins": [ - { - "id": "output_elasticsearch_86", - "type": "output", - "plugin": "elasticsearch", - "config": { - "hosts": [ - "https://es1:9200", - "https://es2:9200" - ], - "user": "${ES_USER}", - "password": "${ES_PASS}", - "ssl": "true", - "cacert": "/etc/logstash/certs/ca.crt", - "index": "%{type}-%{+YYYY.MM.dd}", - "document_id": "%{event_id}", - "pipeline": "enrich", - "ilm_enabled": "true", - "ilm_rollover_alias": "%{type}", - "ilm_pattern": "{now/d}-000001", - "ilm_policy": "logs-policy", - "http_compression": "true" - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "output_if_87", - "type": "output", - "plugin": "if", - "config": { - "condition": "\"siem\" in [tags]", - "plugins": [ - { - "id": "output_elasticsearch_88", - "type": "output", - "plugin": "elasticsearch", - "config": { - "hosts": [ - "https://siem-es:9200" - ], - "user": "${SIEM_USER}", - "password": "${SIEM_PASS}", - "ssl": "true", - "cacert": "/etc/logstash/certs/siem-ca.crt", - "index": "security-%{+YYYY.MM.dd}" - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "output_if_89", - "type": "output", - "plugin": "if", - "config": { - "condition": "[env] == \"production\"", - "plugins": [ - { - "id": "output_s3_90", - "type": "output", - "plugin": "s3", - "config": { - "access_key_id": "${AWS_KEY}", - "secret_access_key": "${AWS_SECRET}", - "region": "us-east-1", - "bucket": "logs-archive", - "size_file": "104857600", - "time_file": "15", - "codec": { - "json_lines": {} - }, - "prefix": "logs/%{type}/year=%{+YYYY}/month=%{+MM}/day=%{+dd}", - "encoding": "gzip", - "server_side_encryption": "true" - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "output_kafka_91", - "type": "output", - "plugin": "kafka", - "config": { - "bootstrap_servers": "kafka1:9092,kafka2:9092", - "topic_id": "processed-%{type}", - "codec": { - "json": {} - }, - "compression_type": "snappy", - "acks": "all", - "security_protocol": "SASL_SSL", - "sasl_mechanism": "SCRAM-SHA-512", - "sasl_jaas_config": "org.apache.kafka.common.security.scram.ScramLoginModule required username='${KAFKA_USER}' password='${KAFKA_PASS}';" - }, - "comments": [] - }, - { - "id": "output_if_92", - "type": "output", - "plugin": "if", - "config": { - "condition": "\"p1\" in [tags] or \"critical\" in [tags]", - "plugins": [ - { - "id": "output_redis_93", - "type": "output", - "plugin": "redis", - "config": { - "host": [ - "redis1", - "redis2" - ], - "port": "26379", - "password": "${REDIS_PASS}", - "data_type": "list", - "key": "alerts:critical" - }, - "comments": [] - }, - { - "id": "output_http_94", - "type": "output", - "plugin": "http", - "config": { - "url": "https://alerts.example.com/api/events", - "http_method": "post", - "format": "json", - "headers": { - "Authorization": "Bearer ${ALERT_TOKEN}", - "Content-Type": "application/json" - }, - "automatic_retries": "3" - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "output_if_95", - "type": "output", - "plugin": "if", - "config": { - "condition": "[env] != \"production\"", - "plugins": [ - { - "id": "output_file_96", - "type": "output", - "plugin": "file", - "config": { - "path": "/var/log/logstash/debug-%{type}.log", - "codec": { - "json_lines": {} - } - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "output_if_97", - "type": "output", - "plugin": "if", - "config": { - "condition": "\"metric\" in [tags]", - "plugins": [ - { - "id": "output_graphite_98", - "type": "output", - "plugin": "graphite", - "config": { - "host": "graphite", - "port": "2003", - "metrics_format": "logstash.%{env}.%{type}.count", - "fields_are_metrics": "true" - }, - "comments": [] - }, - { - "id": "output_influxdb_99", - "type": "output", - "plugin": "influxdb", - "config": { - "host": "influxdb", - "port": "8086", - "db": "metrics", - "user": "${INFLUX_USER}", - "password": "${INFLUX_PASS}", - "measurement": "%{type}_metrics", - "use_event_fields_for_data_points": "true" - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "output_if_100", - "type": "output", - "plugin": "if", - "config": { - "condition": "[type] == \"rabbitmq\"", - "plugins": [ - { - "id": "output_mongodb_101", - "type": "output", - "plugin": "mongodb", - "config": { - "uri": "mongodb://${MONGO_USER}:${MONGO_PASS}@mongo:27017/logs", - "database": "logs", - "collection": "%{type}", - "isodate": "true", - "bulk": "true", - "bulk_size": "100" - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "output_if_102", - "type": "output", - "plugin": "if", - "config": { - "condition": "\"p1\" in [tags]", - "plugins": [ - { - "id": "output_email_103", - "type": "output", - "plugin": "email", - "config": { - "to": "oncall@example.com", - "from": "alerts@example.com", - "subject": "P1 Alert: %{type}", - "body": "Event: %{event_id}\nTime: %{@timestamp}\nSeverity: %{severity}\nMessage: %{message}", - "address": "smtp.example.com", - "port": "587", - "use_tls": "true", - "username": "${SMTP_USER}", - "password": "${SMTP_PASS}" - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "output_tcp_104", - "type": "output", - "plugin": "tcp", - "config": { - "host": "logstash-secondary", - "port": "5005", - "codec": { - "json_lines": {} - } - }, - "comments": [] - }, - { - "id": "output_stdout_105", - "type": "output", - "plugin": "stdout", - "config": { - "codec": { - "rubydebug": {} - } - }, - "comments": [] - } - ] -} diff --git a/src/logstashui/Common/tests/conversion_data/components/test-data-types.json b/src/logstashui/Common/tests/conversion_data/components/test-data-types.json deleted file mode 100644 index f83671f..0000000 --- a/src/logstashui/Common/tests/conversion_data/components/test-data-types.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "input": [], - "filter": [ - { - "id": "filter_grok_0", - "type": "filter", - "plugin": "grok", - "config": { - "match": { - "test": "test" - }, - "pattern_definitions": { - "1": "2", - "test": "test", - "asd": "asf" - }, - "patterns_dir": [ - "test" - ], - "tag_on_failure": [] - }, - "comments": [] - }, - { - "id": "filter_grok_1", - "type": "filter", - "plugin": "grok", - "config": { - "match": { - "test": [ - "test", - "test2" - ] - }, - "pattern_definitions": { - "test": "test" - }, - "patterns_dir": "test" - }, - "comments": [] - }, - { - "id": "filter_comment_2", - "type": "filter", - "plugin": "comment", - "config": { - "text": "test\nmulti\nrow" - } - } - ], - "output": [] -} diff --git a/src/logstashui/Common/tests/conversion_data/components/test-datatypes.json b/src/logstashui/Common/tests/conversion_data/components/test-datatypes.json deleted file mode 100644 index f83671f..0000000 --- a/src/logstashui/Common/tests/conversion_data/components/test-datatypes.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "input": [], - "filter": [ - { - "id": "filter_grok_0", - "type": "filter", - "plugin": "grok", - "config": { - "match": { - "test": "test" - }, - "pattern_definitions": { - "1": "2", - "test": "test", - "asd": "asf" - }, - "patterns_dir": [ - "test" - ], - "tag_on_failure": [] - }, - "comments": [] - }, - { - "id": "filter_grok_1", - "type": "filter", - "plugin": "grok", - "config": { - "match": { - "test": [ - "test", - "test2" - ] - }, - "pattern_definitions": { - "test": "test" - }, - "patterns_dir": "test" - }, - "comments": [] - }, - { - "id": "filter_comment_2", - "type": "filter", - "plugin": "comment", - "config": { - "text": "test\nmulti\nrow" - } - } - ], - "output": [] -} diff --git a/src/logstashui/Common/tests/conversion_data/components/test-devopsschool-1.json b/src/logstashui/Common/tests/conversion_data/components/test-devopsschool-1.json deleted file mode 100644 index b879d77..0000000 --- a/src/logstashui/Common/tests/conversion_data/components/test-devopsschool-1.json +++ /dev/null @@ -1,51 +0,0 @@ -{ - "input": [ - { - "id": "input_beats_0", - "type": "input", - "plugin": "beats", - "config": { - "port": "5044" - }, - "comments": [] - } - ], - "filter": [ - { - "id": "filter_grok_1", - "type": "filter", - "plugin": "grok", - "config": { - "match": { - "message": "%{COMBINEDAPACHELOG}" - } - }, - "comments": [] - } - ], - "output": [ - { - "id": "output_elasticsearch_2", - "type": "output", - "plugin": "elasticsearch", - "config": { - "hosts": [ - "http://elasticsearch:9200" - ], - "index": "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" - }, - "comments": [] - }, - { - "id": "output_stdout_3", - "type": "output", - "plugin": "stdout", - "config": { - "codec": { - "rubydebug": {} - } - }, - "comments": [] - } - ] -} diff --git a/src/logstashui/Common/tests/conversion_data/components/test-devopsschool-2.json b/src/logstashui/Common/tests/conversion_data/components/test-devopsschool-2.json deleted file mode 100644 index 9a82ca7..0000000 --- a/src/logstashui/Common/tests/conversion_data/components/test-devopsschool-2.json +++ /dev/null @@ -1,39 +0,0 @@ -{ - "input": [ - { - "id": "input_beats_0", - "type": "input", - "plugin": "beats", - "config": { - "port": "5044" - }, - "comments": [] - } - ], - "filter": [ - { - "id": "filter_grok_1", - "type": "filter", - "plugin": "grok", - "config": { - "match": { - "message": "%{SYSLOGLINE}" - } - }, - "comments": [] - } - ], - "output": [ - { - "id": "output_stdout_2", - "type": "output", - "plugin": "stdout", - "config": { - "codec": { - "rubydebug": {} - } - }, - "comments": [] - } - ] -} diff --git a/src/logstashui/Common/tests/conversion_data/components/test-devopsschool-4.json b/src/logstashui/Common/tests/conversion_data/components/test-devopsschool-4.json deleted file mode 100644 index 6ad22f2..0000000 --- a/src/logstashui/Common/tests/conversion_data/components/test-devopsschool-4.json +++ /dev/null @@ -1,62 +0,0 @@ -{ - "input": [ - { - "id": "input_file_0", - "type": "input", - "plugin": "file", - "config": { - "path": "/var/log/apache2/access.log", - "start_position": "beginning", - "sincedb_path": "/dev/null" - }, - "comments": [] - } - ], - "filter": [ - { - "id": "filter_grok_1", - "type": "filter", - "plugin": "grok", - "config": { - "match": { - "message": "%{COMBINEDAPACHELOG}" - } - }, - "comments": [] - }, - { - "id": "filter_date_2", - "type": "filter", - "plugin": "date", - "config": { - "match": [ - "timestamp", - "dd/MMM/yyyy:HH:mm:ss Z" - ] - }, - "comments": [] - }, - { - "id": "filter_geoip_3", - "type": "filter", - "plugin": "geoip", - "config": { - "source": "clientip" - }, - "comments": [] - } - ], - "output": [ - { - "id": "output_elasticsearch_4", - "type": "output", - "plugin": "elasticsearch", - "config": { - "hosts": [ - "localhost:9200" - ] - }, - "comments": [] - } - ] -} diff --git a/src/logstashui/Common/tests/conversion_data/components/test-devopsschool-5.json b/src/logstashui/Common/tests/conversion_data/components/test-devopsschool-5.json deleted file mode 100644 index e16f5da..0000000 --- a/src/logstashui/Common/tests/conversion_data/components/test-devopsschool-5.json +++ /dev/null @@ -1,48 +0,0 @@ -{ - "input": [ - { - "id": "input_beats_0", - "type": "input", - "plugin": "beats", - "config": { - "port": "5044" - }, - "comments": [] - } - ], - "filter": [ - { - "id": "filter_grok_1", - "type": "filter", - "plugin": "grok", - "config": { - "match": { - "message": "%{COMBINEDAPACHELOG}" - } - }, - "comments": [] - }, - { - "id": "filter_geoip_2", - "type": "filter", - "plugin": "geoip", - "config": { - "source": "clientip" - }, - "comments": [] - } - ], - "output": [ - { - "id": "output_elasticsearch_3", - "type": "output", - "plugin": "elasticsearch", - "config": { - "hosts": [ - "localhost:9200" - ] - }, - "comments": [] - } - ] -} diff --git a/src/logstashui/Common/tests/conversion_data/components/test-elasticdocs-apache.json b/src/logstashui/Common/tests/conversion_data/components/test-elasticdocs-apache.json deleted file mode 100644 index d373e91..0000000 --- a/src/logstashui/Common/tests/conversion_data/components/test-elasticdocs-apache.json +++ /dev/null @@ -1,86 +0,0 @@ -{ - "input": [ - { - "id": "input_file_0", - "type": "input", - "plugin": "file", - "config": { - "path": "/tmp/access_log", - "start_position": "beginning" - }, - "comments": [] - } - ], - "filter": [ - { - "id": "filter_if_1", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[path] =~ \"access\"", - "plugins": [ - { - "id": "filter_mutate_2", - "type": "filter", - "plugin": "mutate", - "config": { - "replace": { - "type": "apache_access" - } - }, - "comments": [] - }, - { - "id": "filter_grok_3", - "type": "filter", - "plugin": "grok", - "config": { - "match": { - "message": "%{COMBINEDAPACHELOG}" - } - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_date_4", - "type": "filter", - "plugin": "date", - "config": { - "match": [ - "timestamp", - "dd/MMM/yyyy:HH:mm:ss Z" - ] - }, - "comments": [] - } - ], - "output": [ - { - "id": "output_elasticsearch_5", - "type": "output", - "plugin": "elasticsearch", - "config": { - "hosts": [ - "localhost:9200" - ] - }, - "comments": [] - }, - { - "id": "output_stdout_6", - "type": "output", - "plugin": "stdout", - "config": { - "codec": { - "rubydebug": {} - } - }, - "comments": [] - } - ] -} diff --git a/src/logstashui/Common/tests/conversion_data/components/test-elasticdocs-configuring_filters.json b/src/logstashui/Common/tests/conversion_data/components/test-elasticdocs-configuring_filters.json deleted file mode 100644 index ce29631..0000000 --- a/src/logstashui/Common/tests/conversion_data/components/test-elasticdocs-configuring_filters.json +++ /dev/null @@ -1,60 +0,0 @@ -{ - "input": [ - { - "id": "input_stdin_0", - "type": "input", - "plugin": "stdin", - "config": {}, - "comments": [] - } - ], - "filter": [ - { - "id": "filter_grok_1", - "type": "filter", - "plugin": "grok", - "config": { - "match": { - "message": "%{COMBINEDAPACHELOG}" - } - }, - "comments": [] - }, - { - "id": "filter_date_2", - "type": "filter", - "plugin": "date", - "config": { - "match": [ - "timestamp", - "dd/MMM/yyyy:HH:mm:ss Z" - ] - }, - "comments": [] - } - ], - "output": [ - { - "id": "output_elasticsearch_3", - "type": "output", - "plugin": "elasticsearch", - "config": { - "hosts": [ - "localhost:9200" - ] - }, - "comments": [] - }, - { - "id": "output_stdout_4", - "type": "output", - "plugin": "stdout", - "config": { - "codec": { - "rubydebug": {} - } - }, - "comments": [] - } - ] -} diff --git a/src/logstashui/Common/tests/conversion_data/components/test-elasticdocs-syslog.json b/src/logstashui/Common/tests/conversion_data/components/test-elasticdocs-syslog.json deleted file mode 100644 index bfa06fe..0000000 --- a/src/logstashui/Common/tests/conversion_data/components/test-elasticdocs-syslog.json +++ /dev/null @@ -1,92 +0,0 @@ -{ - "input": [ - { - "id": "input_tcp_0", - "type": "input", - "plugin": "tcp", - "config": { - "port": "5000", - "type": "syslog" - }, - "comments": [] - }, - { - "id": "input_udp_1", - "type": "input", - "plugin": "udp", - "config": { - "port": "5000", - "type": "syslog" - }, - "comments": [] - } - ], - "filter": [ - { - "id": "filter_if_2", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[type] == \"syslog\"", - "plugins": [ - { - "id": "filter_grok_3", - "type": "filter", - "plugin": "grok", - "config": { - "match": { - "message": "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\\[%{POSINT:syslog_pid}\\])?: %{GREEDYDATA:syslog_message}" - }, - "add_field": [ - "received_at", - "%{@timestamp}", - "received_from", - "%{host}" - ] - }, - "comments": [] - }, - { - "id": "filter_date_4", - "type": "filter", - "plugin": "date", - "config": { - "match": [ - "syslog_timestamp", - "MMM d HH:mm:ss", - "MMM dd HH:mm:ss" - ] - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - } - ], - "output": [ - { - "id": "output_elasticsearch_5", - "type": "output", - "plugin": "elasticsearch", - "config": { - "hosts": [ - "localhost:9200" - ] - }, - "comments": [] - }, - { - "id": "output_stdout_6", - "type": "output", - "plugin": "stdout", - "config": { - "codec": { - "rubydebug": {} - } - }, - "comments": [] - } - ] -} diff --git a/src/logstashui/Common/tests/conversion_data/components/test-es-input.json b/src/logstashui/Common/tests/conversion_data/components/test-es-input.json deleted file mode 100644 index c3c26fe..0000000 --- a/src/logstashui/Common/tests/conversion_data/components/test-es-input.json +++ /dev/null @@ -1,48 +0,0 @@ -{ - "input": [ - { - "id": "input_elasticsearch_0", - "type": "input", - "plugin": "elasticsearch", - "config": { - "api_key": "test", - "cloud_id": "test", - "index": "kibana_sample_data_ecommerce", - "query": "{\"query\":{\"match_all\":{}}}", - "slices": "6", - "ssl_enabled": "true", - "connect_timeout_seconds": "120", - "request_timeout_seconds": "600", - "socket_timeout_seconds": "600" - }, - "comments": [] - } - ], - "filter": [ - { - "id": "filter_mutate_1", - "type": "filter", - "plugin": "mutate", - "config": { - "add_field": { - "test": "test" - } - }, - "comments": [] - } - ], - "output": [ - { - "id": "output_csv_2", - "type": "output", - "plugin": "csv", - "config": { - "fields": [ - "test" - ], - "path": "/home/ubuntu/test.csv" - }, - "comments": [] - } - ] -} diff --git a/src/logstashui/Common/tests/conversion_data/components/test-ls-repo-mysql.json b/src/logstashui/Common/tests/conversion_data/components/test-ls-repo-mysql.json deleted file mode 100644 index 2a09593..0000000 --- a/src/logstashui/Common/tests/conversion_data/components/test-ls-repo-mysql.json +++ /dev/null @@ -1,156 +0,0 @@ -{ - "input": [ - { - "id": "input_beats_0", - "type": "input", - "plugin": "beats", - "config": { - "port": 5044, - "host": "0.0.0.0" - }, - "comments": [] - } - ], - "filter": [ - { - "id": "filter_if_1", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[fileset][module] == \"mysql\"", - "plugins": [ - { - "id": "filter_if_2", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[fileset][name] == \"error\"", - "plugins": [ - { - "id": "filter_grok_3", - "type": "filter", - "plugin": "grok", - "config": { - "match": { - "message": [ - "%{LOCALDATETIME:[mysql][error][timestamp]} (\\[%{DATA:[mysql][error][level]}\\] )?%{GREEDYDATA:[mysql][error][message]}", - "%{TIMESTAMP_ISO8601:[mysql][error][timestamp]} %{NUMBER:[mysql][error][thread_id]} \\[%{DATA:[mysql][error][level]}\\] %{GREEDYDATA:[mysql][error][message1]}", - "%{GREEDYDATA:[mysql][error][message2]}" - ] - }, - "pattern_definitions": { - "LOCALDATETIME": "[0-9]+ %{TIME}" - }, - "remove_field": "message" - }, - "comments": [] - }, - { - "id": "filter_mutate_4", - "type": "filter", - "plugin": "mutate", - "config": { - "rename": { - "[mysql][error][message1]": "[mysql][error][message]" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_5", - "type": "filter", - "plugin": "mutate", - "config": { - "rename": { - "[mysql][error][message2]": "[mysql][error][message]" - } - }, - "comments": [] - }, - { - "id": "filter_date_6", - "type": "filter", - "plugin": "date", - "config": { - "match": [ - "[mysql][error][timestamp]", - "ISO8601", - "YYMMdd H:m:s" - ], - "remove_field": "[mysql][error][time]" - }, - "comments": [] - } - ], - "else_ifs": [ - { - "condition": "[fileset][name] == \"slowlog\"", - "plugins": [ - { - "id": "filter_grok_7", - "type": "filter", - "plugin": "grok", - "config": { - "match": { - "message": [ - "^# User@Host: %{USER:[mysql][slowlog][user]}(\\[[^\\]]+\\])? @ %{HOSTNAME:[mysql][slowlog][host]} \\[(IP:[mysql][slowlog][ip])?\\](\\s*Id:\\s* %{NUMBER:[mysql][slowlog][id]})?\n# Query_time: %{NUMBER:[mysql][slowlog][query_time][sec]}\\s* Lock_time: %{NUMBER:[mysql][slowlog][lock_time][sec]}\\s* Rows_sent: %{NUMBER:[mysql][slowlog][rows_sent]}\\s* Rows_examined: %{NUMBER:[mysql][slowlog][rows_examined]}\n(SET timestamp=%{NUMBER:[mysql][slowlog][timestamp]};\n)?%{GREEDYMULTILINE:[mysql][slowlog][query]}" - ] - }, - "pattern_definitions": { - "GREEDYMULTILINE": "(.|\n)*" - }, - "remove_field": "message" - }, - "comments": [] - }, - { - "id": "filter_date_8", - "type": "filter", - "plugin": "date", - "config": { - "match": [ - "[mysql][slowlog][timestamp]", - "UNIX" - ] - }, - "comments": [] - }, - { - "id": "filter_mutate_9", - "type": "filter", - "plugin": "mutate", - "config": { - "gsub": [ - "[mysql][slowlog][query]", - "\n# Time: [0-9]+ [0-9][0-9]:[0-9][0-9]:[0-9][0-9](\\.[0-9]+)?$", - "" - ] - }, - "comments": [] - } - ] - } - ], - "else": null - } - } - ], - "else_ifs": [], - "else": null - } - } - ], - "output": [ - { - "id": "output_elasticsearch_10", - "type": "output", - "plugin": "elasticsearch", - "config": { - "hosts": "localhost", - "manage_template": "false", - "index": "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" - }, - "comments": [] - } - ] -} diff --git a/src/logstashui/Common/tests/conversion_data/components/test-ls-repo-nginx.json b/src/logstashui/Common/tests/conversion_data/components/test-ls-repo-nginx.json deleted file mode 100644 index 3493de0..0000000 --- a/src/logstashui/Common/tests/conversion_data/components/test-ls-repo-nginx.json +++ /dev/null @@ -1,156 +0,0 @@ -{ - "input": [ - { - "id": "input_beats_0", - "type": "input", - "plugin": "beats", - "config": { - "port": 5044, - "host": "0.0.0.0" - }, - "comments": [] - } - ], - "filter": [ - { - "id": "filter_if_1", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[fileset][module] == \"nginx\"", - "plugins": [ - { - "id": "filter_if_2", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[fileset][name] == \"access\"", - "plugins": [ - { - "id": "filter_grok_3", - "type": "filter", - "plugin": "grok", - "config": { - "match": { - "message": [ - "%{IPORHOST:[nginx][access][remote_ip]} - %{DATA:[nginx][access][user_name]} \\[%{HTTPDATE:[nginx][access][time]}\\] \"%{WORD:[nginx][access][method]} %{DATA:[nginx][access][url]} HTTP/%{NUMBER:[nginx][access][http_version]}\" %{NUMBER:[nginx][access][response_code]} %{NUMBER:[nginx][access][body_sent][bytes]} \"%{DATA:[nginx][access][referrer]}\" \"%{DATA:[nginx][access][agent]}\"" - ] - }, - "remove_field": "message" - }, - "comments": [] - }, - { - "id": "filter_mutate_4", - "type": "filter", - "plugin": "mutate", - "config": { - "add_field": { - "read_timestamp": "%{@timestamp}" - } - }, - "comments": [] - }, - { - "id": "filter_date_5", - "type": "filter", - "plugin": "date", - "config": { - "match": [ - "[nginx][access][time]", - "dd/MMM/YYYY:H:m:s Z" - ], - "remove_field": "[nginx][access][time]" - }, - "comments": [] - }, - { - "id": "filter_useragent_6", - "type": "filter", - "plugin": "useragent", - "config": { - "source": "[nginx][access][agent]", - "target": "[nginx][access][user_agent]", - "remove_field": "[nginx][access][agent]" - }, - "comments": [] - }, - { - "id": "filter_geoip_7", - "type": "filter", - "plugin": "geoip", - "config": { - "source": "[nginx][access][remote_ip]", - "target": "[nginx][access][geoip]" - }, - "comments": [] - } - ], - "else_ifs": [ - { - "condition": "[fileset][name] == \"error\"", - "plugins": [ - { - "id": "filter_grok_8", - "type": "filter", - "plugin": "grok", - "config": { - "match": { - "message": [ - "%{DATA:[nginx][error][time]} \\[%{DATA:[nginx][error][level]}\\] %{NUMBER:[nginx][error][pid]}#%{NUMBER:[nginx][error][tid]}: (\\*%{NUMBER:[nginx][error][connection_id]} )?%{GREEDYDATA:[nginx][error][message]}" - ] - }, - "remove_field": "message" - }, - "comments": [] - }, - { - "id": "filter_mutate_9", - "type": "filter", - "plugin": "mutate", - "config": { - "rename": { - "@timestamp": "read_timestamp" - } - }, - "comments": [] - }, - { - "id": "filter_date_10", - "type": "filter", - "plugin": "date", - "config": { - "match": [ - "[nginx][error][time]", - "YYYY/MM/dd H:m:s" - ], - "remove_field": "[nginx][error][time]" - }, - "comments": [] - } - ] - } - ], - "else": null - } - } - ], - "else_ifs": [], - "else": null - } - } - ], - "output": [ - { - "id": "output_elasticsearch_11", - "type": "output", - "plugin": "elasticsearch", - "config": { - "hosts": "localhost", - "manage_template": "false", - "index": "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" - }, - "comments": [] - } - ] -} diff --git a/src/logstashui/Common/tests/conversion_data/components/test-ls-repo-system.json b/src/logstashui/Common/tests/conversion_data/components/test-ls-repo-system.json deleted file mode 100644 index 94114af..0000000 --- a/src/logstashui/Common/tests/conversion_data/components/test-ls-repo-system.json +++ /dev/null @@ -1,135 +0,0 @@ -{ - "input": [ - { - "id": "input_beats_0", - "type": "input", - "plugin": "beats", - "config": { - "port": 5044, - "host": "0.0.0.0" - }, - "comments": [] - } - ], - "filter": [ - { - "id": "filter_if_1", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[fileset][module] == \"system\"", - "plugins": [ - { - "id": "filter_if_2", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[fileset][name] == \"auth\"", - "plugins": [ - { - "id": "filter_grok_3", - "type": "filter", - "plugin": "grok", - "config": { - "match": { - "message": [ - "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\\[%{POSINT:[system][auth][pid]}\\])?: %{DATA:[system][auth][ssh][event]} %{DATA:[system][auth][ssh][method]} for (invalid user )?%{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]} port %{NUMBER:[system][auth][ssh][port]} ssh2(: %{GREEDYDATA:[system][auth][ssh][signature]})?", - "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\\[%{POSINT:[system][auth][pid]}\\])?: %{DATA:[system][auth][ssh][event]} user %{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]}", - "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\\[%{POSINT:[system][auth][pid]}\\])?: Did not receive identification string from %{IPORHOST:[system][auth][ssh][dropped_ip]}", - "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sudo(?:\\[%{POSINT:[system][auth][pid]}\\])?: \\s*%{DATA:[system][auth][user]} :( %{DATA:[system][auth][sudo][error]} ;)? TTY=%{DATA:[system][auth][sudo][tty]} ; PWD=%{DATA:[system][auth][sudo][pwd]} ; USER=%{DATA:[system][auth][sudo][user]} ; COMMAND=%{GREEDYDATA:[system][auth][sudo][command]}", - "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} groupadd(?:\\[%{POSINT:[system][auth][pid]}\\])?: new group: name=%{DATA:system.auth.groupadd.name}, GID=%{NUMBER:system.auth.groupadd.gid}", - "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} useradd(?:\\[%{POSINT:[system][auth][pid]}\\])?: new user: name=%{DATA:[system][auth][useradd][name]}, UID=%{NUMBER:[system][auth][useradd][uid]}, GID=%{NUMBER:[system][auth][useradd][gid]}, home=%{DATA:[system][auth][useradd][home]}, shell=%{DATA:[system][auth][useradd][shell]}$", - "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} %{DATA:[system][auth][program]}(?:\\[%{POSINT:[system][auth][pid]}\\])?: %{GREEDYMULTILINE:[system][auth][message]}" - ] - }, - "pattern_definitions": { - "GREEDYMULTILINE": "(.|\n)*" - }, - "remove_field": "message" - }, - "comments": [] - }, - { - "id": "filter_date_4", - "type": "filter", - "plugin": "date", - "config": { - "match": [ - "[system][auth][timestamp]", - "MMM d HH:mm:ss", - "MMM dd HH:mm:ss" - ] - }, - "comments": [] - }, - { - "id": "filter_geoip_5", - "type": "filter", - "plugin": "geoip", - "config": { - "source": "[system][auth][ssh][ip]", - "target": "[system][auth][ssh][geoip]" - }, - "comments": [] - } - ], - "else_ifs": [ - { - "condition": "[fileset][name] == \"syslog\"", - "plugins": [ - { - "id": "filter_grok_6", - "type": "filter", - "plugin": "grok", - "config": { - "match": { - "message": [ - "%{SYSLOGTIMESTAMP:[system][syslog][timestamp]} %{SYSLOGHOST:[system][syslog][hostname]} %{DATA:[system][syslog][program]}(?:\\[%{POSINT:[system][syslog][pid]}\\])?: %{GREEDYMULTILINE:[system][syslog][message]}" - ] - }, - "pattern_definitions": { - "GREEDYMULTILINE": "(.|\n)*" - }, - "remove_field": "message" - }, - "comments": [] - }, - { - "id": "filter_date_7", - "type": "filter", - "plugin": "date", - "config": { - "match": [ - "[system][syslog][timestamp]", - "MMM d HH:mm:ss", - "MMM dd HH:mm:ss" - ] - }, - "comments": [] - } - ] - } - ], - "else": null - } - } - ], - "else_ifs": [], - "else": null - } - } - ], - "output": [ - { - "id": "output_elasticsearch_8", - "type": "output", - "plugin": "elasticsearch", - "config": { - "hosts": "localhost", - "manage_template": "false", - "index": "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" - }, - "comments": [] - } - ] -} diff --git a/src/logstashui/Common/tests/conversion_data/components/test-multiline-ruby-with-hash.json b/src/logstashui/Common/tests/conversion_data/components/test-multiline-ruby-with-hash.json deleted file mode 100644 index 3d0be39..0000000 --- a/src/logstashui/Common/tests/conversion_data/components/test-multiline-ruby-with-hash.json +++ /dev/null @@ -1,60 +0,0 @@ -{ - "input": [ - { - "id": "input_stdin_0", - "type": "input", - "plugin": "stdin", - "config": { - "codec": { - "line": {} - } - }, - "comments": [] - } - ], - "filter": [ - { - "id": "filter_ruby_1", - "type": "filter", - "plugin": "ruby", - "config": { - "code": "\nrequire \"digest\"\nmsg = event.get(\"message\").to_s\nevent.set(\"hash\", Digest::MD5.hexdigest(msg))\n# this # is NOT a comment \u2014 it is inside a single-quoted string\nif event.get(\"level\") == \"ERROR\"\n\tevent.set(\"alert\", true)\n\tevent.set(\"severity\", \"high\")\nelsif event.get(\"level\") == \"WARN\"\n\tevent.set(\"severity\", \"medium\")\nelse\n\tevent.set(\"severity\", \"low\")\nend\n# another hash # mark inside the string \u2014 still not a comment\n\t\t" - }, - "comments": [] - }, - { - "id": "filter_ruby_2", - "type": "filter", - "plugin": "ruby", - "config": { - "init": "\nrequire \"openssl\"\nrequire \"base64\"\n# init comment inside single-quoted string\n@secret = ENV[\"SIGNING_SECRET\"] || \"default\"\n\t\t", - "code": "\npayload = event.get(\"message\").to_s\nsig = Base64.strict_encode64(\n\tOpenSSL::HMAC.digest(\"SHA256\", @secret, payload)\n)\nevent.set(\"signature\", sig)\n\t\t" - }, - "comments": [] - }, - { - "id": "filter_mutate_3", - "type": "filter", - "plugin": "mutate", - "config": { - "add_field": { - "pipeline": "ruby-test" - } - }, - "comments": [] - } - ], - "output": [ - { - "id": "output_stdout_4", - "type": "output", - "plugin": "stdout", - "config": { - "codec": { - "rubydebug": {} - } - }, - "comments": [] - } - ] -} diff --git a/src/logstashui/Common/tests/conversion_data/components/test-nested-conditionals-comments.json b/src/logstashui/Common/tests/conversion_data/components/test-nested-conditionals-comments.json deleted file mode 100644 index 20eaa18..0000000 --- a/src/logstashui/Common/tests/conversion_data/components/test-nested-conditionals-comments.json +++ /dev/null @@ -1,266 +0,0 @@ -{ - "input": [ - { - "id": "input_stdin_0", - "type": "input", - "plugin": "stdin", - "config": {}, - "comments": [] - } - ], - "filter": [ - { - "id": "filter_comment_1", - "type": "filter", - "plugin": "comment", - "config": { - "text": "Top-level comment before any conditional" - } - }, - { - "id": "filter_if_2", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[type] == \"web\"", - "plugins": [ - { - "id": "filter_comment_3", - "type": "filter", - "plugin": "comment", - "config": { - "text": "Comment inside first if branch" - } - }, - { - "id": "filter_if_4", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[status] >= 500", - "plugins": [ - { - "id": "filter_comment_5", - "type": "filter", - "plugin": "comment", - "config": { - "text": "Comment inside nested if" - } - }, - { - "id": "filter_mutate_6", - "type": "filter", - "plugin": "mutate", - "config": { - "add_tag": [ - "server_error" - ], - "add_field": { - "severity": "high" - } - }, - "comments": [] - }, - { - "id": "filter_comment_7", - "type": "filter", - "plugin": "comment", - "config": { - "text": "Comment after plugin inside nested if" - } - }, - { - "id": "filter_if_8", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[status] == 503", - "plugins": [ - { - "id": "filter_mutate_9", - "type": "filter", - "plugin": "mutate", - "config": { - "add_tag": [ - "service_unavailable" - ] - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_comment_10", - "type": "filter", - "plugin": "comment", - "config": { - "text": "Trailing comment inside nested if" - } - } - ], - "else_ifs": [ - { - "condition": "[status] >= 400", - "plugins": [ - { - "id": "filter_comment_11", - "type": "filter", - "plugin": "comment", - "config": { - "text": "Comment inside else-if" - } - }, - { - "id": "filter_mutate_12", - "type": "filter", - "plugin": "mutate", - "config": { - "add_tag": [ - "client_error" - ], - "add_field": { - "severity": "medium" - } - }, - "comments": [] - }, - { - "id": "filter_comment_13", - "type": "filter", - "plugin": "comment", - "config": { - "text": "Trailing comment inside else-if" - } - } - ] - } - ], - "else": { - "plugins": [ - { - "id": "filter_comment_14", - "type": "filter", - "plugin": "comment", - "config": { - "text": "Comment inside else" - } - }, - { - "id": "filter_mutate_15", - "type": "filter", - "plugin": "mutate", - "config": { - "add_tag": [ - "success" - ], - "add_field": { - "severity": "low" - } - }, - "comments": [] - } - ] - } - } - }, - { - "id": "filter_comment_16", - "type": "filter", - "plugin": "comment", - "config": { - "text": "Comment at end of outer if block" - } - } - ], - "else_ifs": [ - { - "condition": "[type] == \"db\"", - "plugins": [ - { - "id": "filter_comment_17", - "type": "filter", - "plugin": "comment", - "config": { - "text": "Comment at start of else-if block" - } - }, - { - "id": "filter_mutate_18", - "type": "filter", - "plugin": "mutate", - "config": { - "add_field": { - "source": "database" - } - }, - "comments": [] - } - ] - } - ], - "else": { - "plugins": [ - { - "id": "filter_comment_19", - "type": "filter", - "plugin": "comment", - "config": { - "text": "Comment in final else" - } - }, - { - "id": "filter_drop_20", - "type": "filter", - "plugin": "drop", - "config": {}, - "comments": [] - } - ] - } - } - }, - { - "id": "filter_comment_21", - "type": "filter", - "plugin": "comment", - "config": { - "text": "Comment between conditional and next plugin at section level" - } - }, - { - "id": "filter_mutate_22", - "type": "filter", - "plugin": "mutate", - "config": { - "add_field": { - "processed_by": "logstash" - } - }, - "comments": [] - }, - { - "id": "filter_comment_23", - "type": "filter", - "plugin": "comment", - "config": { - "text": "Trailing section-level comment" - } - } - ], - "output": [ - { - "id": "output_stdout_24", - "type": "output", - "plugin": "stdout", - "config": { - "codec": { - "rubydebug": {} - } - }, - "comments": [] - } - ] -} diff --git a/src/logstashui/Common/tests/conversion_data/components/test-regex-conditions.json b/src/logstashui/Common/tests/conversion_data/components/test-regex-conditions.json deleted file mode 100644 index e86286f..0000000 --- a/src/logstashui/Common/tests/conversion_data/components/test-regex-conditions.json +++ /dev/null @@ -1,228 +0,0 @@ -{ - "input": [ - { - "id": "input_syslog_0", - "type": "input", - "plugin": "syslog", - "config": { - "port": 514 - }, - "comments": [] - } - ], - "filter": [ - { - "id": "filter_if_1", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[message] =~ /^ERROR/", - "plugins": [ - { - "id": "filter_mutate_2", - "type": "filter", - "plugin": "mutate", - "config": { - "add_tag": [ - "error" - ] - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_if_3", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[message] =~ /\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}/", - "plugins": [ - { - "id": "filter_date_4", - "type": "filter", - "plugin": "date", - "config": { - "match": [ - "message", - "ISO8601" - ], - "target": "@timestamp" - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_if_5", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[host] =~ /^(web|app|db)-\\d+\\.example\\.com$/", - "plugins": [ - { - "id": "filter_mutate_6", - "type": "filter", - "plugin": "mutate", - "config": { - "add_field": { - "internal": "true" - } - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_if_7", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[message] !~ /^$/", - "plugins": [ - { - "id": "filter_grok_8", - "type": "filter", - "plugin": "grok", - "config": { - "match": { - "message": "%{GREEDYDATA:content}" - } - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_if_9", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[path] =~ /\\/api\\/v[12]\\// and [method] == \"POST\"", - "plugins": [ - { - "id": "filter_mutate_10", - "type": "filter", - "plugin": "mutate", - "config": { - "add_tag": [ - "api_write" - ] - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_if_11", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[status] =~ /^5\\d\\d$/", - "plugins": [ - { - "id": "filter_mutate_12", - "type": "filter", - "plugin": "mutate", - "config": { - "add_tag": [ - "server_error" - ] - }, - "comments": [] - } - ], - "else_ifs": [ - { - "condition": "[status] =~ /^4\\d\\d$/", - "plugins": [ - { - "id": "filter_mutate_13", - "type": "filter", - "plugin": "mutate", - "config": { - "add_tag": [ - "client_error" - ] - }, - "comments": [] - } - ] - } - ], - "else": null - } - }, - { - "id": "filter_if_14", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[user_agent] =~ /(?i)bot|crawler|spider/", - "plugins": [ - { - "id": "filter_drop_15", - "type": "filter", - "plugin": "drop", - "config": {}, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - } - ], - "output": [ - { - "id": "output_if_16", - "type": "output", - "plugin": "if", - "config": { - "condition": "\"error\" in [tags]", - "plugins": [ - { - "id": "output_file_17", - "type": "output", - "plugin": "file", - "config": { - "path": "/var/log/errors.log", - "codec": { - "json": {} - } - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "output_stdout_18", - "type": "output", - "plugin": "stdout", - "config": { - "codec": { - "rubydebug": {} - } - }, - "comments": [] - } - ] -} diff --git a/src/logstashui/Common/tests/conversion_data/components/test-sample-nginx.json b/src/logstashui/Common/tests/conversion_data/components/test-sample-nginx.json deleted file mode 100644 index a3b972d..0000000 --- a/src/logstashui/Common/tests/conversion_data/components/test-sample-nginx.json +++ /dev/null @@ -1,183 +0,0 @@ -{ - "input": [ - { - "id": "input_stdin_0", - "type": "input", - "plugin": "stdin", - "config": { - "codec": { - "line": {} - } - }, - "comments": [] - } - ], - "filter": [ - { - "id": "filter_mutate_1", - "type": "filter", - "plugin": "mutate", - "config": { - "add_field": { - "event.dataset": "nginx.access", - "service.name": "nginx" - } - }, - "comments": [] - }, - { - "id": "filter_grok_2", - "type": "filter", - "plugin": "grok", - "config": { - "match": { - "message": [ - "%{IPORHOST:source.address} %{DATA:nginx.ident} %{DATA:user.name} \\[%{HTTPDATE:nginx.access.time}\\] \"%{WORD:http.request.method} %{DATA:url.original} HTTP/%{NUMBER:http.version}\" %{NUMBER:http.response.status_code:int} (?:%{NUMBER:http.response.body.bytes:int}|-) \"%{DATA:http.request.referrer}\" \"%{DATA:user_agent.original}\" %{NUMBER:nginx.access.request_time:float}", - "%{IPORHOST:source.address} %{DATA:nginx.ident} %{DATA:user.name} \\[%{HTTPDATE:nginx.access.time}\\] \"%{WORD:http.request.method} %{DATA:url.original} HTTP/%{NUMBER:http.version}\" %{NUMBER:http.response.status_code:int} (?:%{NUMBER:http.response.body.bytes:int}|-) \"%{DATA:http.request.referrer}\" \"%{DATA:user_agent.original}\"", - "%{IPORHOST:source.address} %{DATA:nginx.ident} %{DATA:user.name} \\[%{HTTPDATE:nginx.access.time}\\] \"%{WORD:http.request.method} %{DATA:url.original} HTTP/%{NUMBER:http.version}\" %{NUMBER:http.response.status_code:int} (?:%{NUMBER:http.response.body.bytes:int}|-) \"%{DATA:http.request.referrer}\" \"%{DATA:user_agent.original}\" (?:rt=%{NUMBER:nginx.access.request_time:float}\\s+uct=%{NUMBER:nginx.access.upstream_connect_time:float}\\s+uht=%{NUMBER:nginx.access.upstream_header_time:float}\\s+urt=%{NUMBER:nginx.access.upstream_response_time:float})" - ] - }, - "tag_on_failure": [ - "_grok_nginx_access_fail" - ] - }, - "comments": [] - }, - { - "id": "filter_date_3", - "type": "filter", - "plugin": "date", - "config": { - "match": [ - "nginx.access.time", - "dd/MMM/yyyy:HH:mm:ss Z" - ], - "target": "@timestamp" - }, - "comments": [] - }, - { - "id": "filter_urldecode_4", - "type": "filter", - "plugin": "urldecode", - "config": { - "field": "url.original" - }, - "comments": [] - }, - { - "id": "filter_dissect_5", - "type": "filter", - "plugin": "dissect", - "config": { - "mapping": { - "url.original": "%{url.path}?%{url.query}" - } - }, - "comments": [] - }, - { - "id": "filter_useragent_6", - "type": "filter", - "plugin": "useragent", - "config": { - "source": "user_agent.original", - "target": "user_agent" - }, - "comments": [] - }, - { - "id": "filter_mutate_7", - "type": "filter", - "plugin": "mutate", - "config": { - "copy": { - "source.address": "source.ip" - } - }, - "comments": [] - }, - { - "id": "filter_geoip_8", - "type": "filter", - "plugin": "geoip", - "config": { - "source": "source.ip", - "target": "source.geo", - "tag_on_failure": [ - "_geoip_fail" - ] - }, - "comments": [] - }, - { - "id": "filter_mutate_9", - "type": "filter", - "plugin": "mutate", - "config": { - "gsub": [ - "http.request.referrer", - "^-$", - "", - "user.name", - "^-$", - "" - ] - }, - "comments": [] - }, - { - "id": "filter_if_10", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[http][response][status_code] and [http][response][status_code] >= 500", - "plugins": [ - { - "id": "filter_mutate_11", - "type": "filter", - "plugin": "mutate", - "config": { - "add_tag": [ - "nginx_server_error" - ] - }, - "comments": [] - } - ], - "else_ifs": [ - { - "condition": "[http][response][status_code] and [http][response][status_code] >= 400", - "plugins": [ - { - "id": "filter_mutate_12", - "type": "filter", - "plugin": "mutate", - "config": { - "add_tag": [ - "nginx_client_error" - ] - }, - "comments": [] - } - ] - } - ], - "else": null - } - } - ], - "output": [ - { - "id": "output_stdout_13", - "type": "output", - "plugin": "stdout", - "config": { - "codec": { - "rubydebug": {} - } - }, - "comments": [] - } - ] -} diff --git a/src/logstashui/Common/tests/conversion_data/components/test-snmp-v0.2.json b/src/logstashui/Common/tests/conversion_data/components/test-snmp-v0.2.json deleted file mode 100644 index 2d17fe9..0000000 --- a/src/logstashui/Common/tests/conversion_data/components/test-snmp-v0.2.json +++ /dev/null @@ -1,212 +0,0 @@ -{ - "input": [ - { - "id": "input_snmp_0", - "type": "input", - "plugin": "snmp", - "config": { - "hosts": [ - { - "host": "udp:1.2.3.4/161", - "version": "3", - "timeout": 1000, - "retries": 2 - } - ], - "interval": "30", - "security_name": "test", - "security_level": "authPriv", - "ecs_compatibility": "disabled", - "oid_mapping_format": "dotted_string", - "auth_protocol": "sha", - "auth_pass": "test", - "priv_protocol": "aes", - "priv_pass": "test", - "get": [ - "1.3.6.1.4.1.9.2.1.57.0", - "1.3.6.1.4.1.9.9.48.1.1.1.5.1", - "1.3.6.1.4.1.9.9.48.1.1.1.6.1", - "1.3.6.1.2.1.1.1.0", - "1.3.6.1.2.1.1.5.0", - "1.3.6.1.2.1.1.2.0", - "1.3.6.1.2.1.1.3.0" - ], - "tables": [ - { - "name": "cdpCacheTable", - "columns": [ - "1.3.6.1.4.1.9.9.23.1.2.1.1.1", - "1.3.6.1.4.1.9.9.23.1.2.1.1.6", - "1.3.6.1.4.1.9.9.23.1.2.1.1.7", - "1.3.6.1.4.1.9.9.23.1.2.1.1.8", - "1.3.6.1.4.1.9.9.23.1.2.1.1.9", - "1.3.6.1.4.1.9.9.23.1.2.1.1.5", - "1.3.6.1.4.1.9.9.23.1.2.1.1.4" - ] - }, - { - "name": "sensors", - "columns": [ - "1.3.6.1.4.1.9.9.13.1.3.1.2", - "1.3.6.1.4.1.9.9.13.1.3.1.3", - "1.3.6.1.4.1.9.9.13.1.3.1.4", - "1.3.6.1.4.1.9.9.13.1.3.1.5", - "1.3.6.1.4.1.9.9.13.1.3.1.6" - ] - }, - { - "name": "fans", - "columns": [ - "1.3.6.1.4.1.9.9.13.1.4.1.2", - "1.3.6.1.4.1.9.9.13.1.4.1.3" - ] - }, - { - "name": "interfaces", - "columns": [ - "1.3.6.1.2.1.2.2.1.1", - "1.3.6.1.2.1.2.2.1.2", - "1.3.6.1.2.1.2.2.1.3", - "1.3.6.1.2.1.2.2.1.7", - "1.3.6.1.2.1.2.2.1.8", - "1.3.6.1.2.1.31.1.1.1.1", - "1.3.6.1.2.1.31.1.1.1.18", - "1.3.6.1.2.1.31.1.1.1.15", - "1.3.6.1.2.1.2.2.1.5", - "1.3.6.1.2.1.2.2.1.6", - "1.3.6.1.2.1.2.2.1.4", - "1.3.6.1.2.1.31.1.1.1.6", - "1.3.6.1.2.1.31.1.1.1.10", - "1.3.6.1.2.1.31.1.1.1.9", - "1.3.6.1.2.1.31.1.1.1.13", - "1.3.6.1.2.1.31.1.1.1.7", - "1.3.6.1.2.1.31.1.1.1.11", - "1.3.6.1.2.1.31.1.1.1.8", - "1.3.6.1.2.1.31.1.1.1.12", - "1.3.6.1.2.1.2.2.1.9", - "1.3.6.1.2.1.17.7.1.4.5.1.1", - "1.3.6.1.2.1.2.2.1.14", - "1.3.6.1.2.1.2.2.1.20", - "1.3.6.1.2.1.2.2.1.13", - "1.3.6.1.2.1.2.2.1.19" - ] - } - ] - }, - "comments": [] - } - ], - "filter": [ - { - "id": "filter_mutate_1", - "type": "filter", - "plugin": "mutate", - "config": { - "rename": { - "host": "[host][hostname]" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_2", - "type": "filter", - "plugin": "mutate", - "config": { - "rename": { - "1.3.6.1.4.1.9.2.1.57.0": "[system][cpu][total][norm][pct]", - "1.3.6.1.4.1.9.9.48.1.1.1.5.1": "[system][memory][actual][used][bytes]", - "1.3.6.1.4.1.9.9.48.1.1.1.6.1": "[system][memory][actual][free][bytes]", - "1.3.6.1.2.1.1.1.0": "[host][description]", - "1.3.6.1.2.1.1.5.0": "[host][name]", - "1.3.6.1.2.1.1.2.0": "[host][id]", - "1.3.6.1.2.1.1.3.0": "[host][uptime]" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_3", - "type": "filter", - "plugin": "mutate", - "config": { - "add_field": { - "[network][name]": "home-segment-1 (192.168.4.0/24)", - "[metricset][module]": "system" - } - }, - "comments": [] - }, - { - "id": "filter_ruby_4", - "type": "filter", - "plugin": "ruby", - "config": { - "code": " v = event.get(\"[system][cpu][total][norm][pct]\")\n if v\n event.set(\"[system][cpu][total][norm][pct]\", v.to_f / 100.0)\n end" - }, - "comments": [] - }, - { - "id": "filter_ruby_5", - "type": "filter", - "plugin": "ruby", - "config": { - "code": "\n used = event.get(\"[system][memory][actual][used][bytes]\")\n free = event.get(\"[system][memory][actual][free][bytes]\")\n\t\t\n if used && free\n used_f = used.to_f\n free_f = free.to_f\n total_f = used_f + free_f\n\t\t\n if total_f > 0\n event.set(\"[system][memory][total]\", total_f)\n event.set(\"[system][memory][actual][used][pct]\", (used_f / total_f))\n event.set(\"[system][memory][actual][free][pct]\", (free_f / total_f))\n end\n end\n " - }, - "comments": [] - }, - { - "id": "filter_ruby_6", - "type": "filter", - "plugin": "ruby", - "config": { - "code": "rows = event.get('[cdpCacheTable]')\nif rows.is_a?(Array)\n host_name = event.get('[host][name]')\n host_hostname = event.get('[host][hostname]')\n network_name = event.get('[network][name]')\n timestamp = event.get('@timestamp')\n rows.each do |row|\n next unless row.is_a?(Hash)\n row['cdpCacheIfIndex'] = row.delete('1.3.6.1.4.1.9.9.23.1.2.1.1.1')\n row['cdpCacheDeviceId'] = row.delete('1.3.6.1.4.1.9.9.23.1.2.1.1.6')\n row['cdpCacheDevicePort'] = row.delete('1.3.6.1.4.1.9.9.23.1.2.1.1.7')\n row['cdpCachePlatform'] = row.delete('1.3.6.1.4.1.9.9.23.1.2.1.1.8')\n row['cdpCacheCapabilities'] = row.delete('1.3.6.1.4.1.9.9.23.1.2.1.1.9')\n row['cdpCacheVersion'] = row.delete('1.3.6.1.4.1.9.9.23.1.2.1.1.5')\n row['cdpCacheAddress'] = row.delete('1.3.6.1.4.1.9.9.23.1.2.1.1.4')\n new_event = LogStash::Event.new({\n '@timestamp' => timestamp,\n 'host' => { 'name' => host_name, 'hostname' => host_hostname },\n 'network' => { 'name' => network_name },\n 'table' => row,\n 'metricset' => { 'module' => 'snmp' },\n 'event' => { 'kind' => 'cdpcachetable' }\n })\n new_event_block.call(new_event)\n end\n event.remove('[cdpCacheTable]')\n event.set('[event][kind]', 'metrics')\nend" - }, - "comments": [] - }, - { - "id": "filter_ruby_7", - "type": "filter", - "plugin": "ruby", - "config": { - "code": "rows = event.get('[sensors]')\nif rows.is_a?(Array)\n host_name = event.get('[host][name]')\n host_hostname = event.get('[host][hostname]')\n network_name = event.get('[network][name]')\n timestamp = event.get('@timestamp')\n rows.each do |row|\n next unless row.is_a?(Hash)\n row['description'] = row.delete('1.3.6.1.4.1.9.9.13.1.3.1.2')\n row['temp_celsius'] = row.delete('1.3.6.1.4.1.9.9.13.1.3.1.3')\n row['temp_threshold'] = row.delete('1.3.6.1.4.1.9.9.13.1.3.1.4')\n row['temp_last_shutdown'] = row.delete('1.3.6.1.4.1.9.9.13.1.3.1.5')\n row['state'] = row.delete('1.3.6.1.4.1.9.9.13.1.3.1.6')\n new_event = LogStash::Event.new({\n '@timestamp' => timestamp,\n 'host' => { 'name' => host_name, 'hostname' => host_hostname },\n 'network' => { 'name' => network_name },\n 'table' => row,\n 'metricset' => { 'module' => 'snmp' },\n 'event' => { 'kind' => 'sensors' }\n })\n new_event_block.call(new_event)\n end\n event.remove('[sensors]')\n event.set('[event][kind]', 'metrics')\nend" - }, - "comments": [] - }, - { - "id": "filter_ruby_8", - "type": "filter", - "plugin": "ruby", - "config": { - "code": "rows = event.get('[fans]')\nif rows.is_a?(Array)\n host_name = event.get('[host][name]')\n host_hostname = event.get('[host][hostname]')\n network_name = event.get('[network][name]')\n timestamp = event.get('@timestamp')\n rows.each do |row|\n next unless row.is_a?(Hash)\n row['description'] = row.delete('1.3.6.1.4.1.9.9.13.1.4.1.2')\n row['state'] = row.delete('1.3.6.1.4.1.9.9.13.1.4.1.3')\n new_event = LogStash::Event.new({\n '@timestamp' => timestamp,\n 'host' => { 'name' => host_name, 'hostname' => host_hostname },\n 'network' => { 'name' => network_name },\n 'table' => row,\n 'metricset' => { 'module' => 'snmp' },\n 'event' => { 'kind' => 'fans' }\n })\n new_event_block.call(new_event)\n end\n event.remove('[fans]')\n event.set('[event][kind]', 'metrics')\nend" - }, - "comments": [] - }, - { - "id": "filter_ruby_9", - "type": "filter", - "plugin": "ruby", - "config": { - "code": "rows = event.get('[interfaces]')\nif rows.is_a?(Array)\n host_name = event.get('[host][name]')\n host_hostname = event.get('[host][hostname]')\n network_name = event.get('[network][name]')\n timestamp = event.get('@timestamp')\n rows.each do |row|\n next unless row.is_a?(Hash)\n row['ifIndex'] = row.delete('1.3.6.1.2.1.2.2.1.1')\n row['ifDescr'] = row.delete('1.3.6.1.2.1.2.2.1.2')\n row['ifType'] = row.delete('1.3.6.1.2.1.2.2.1.3')\n row['ifAdminStatus'] = row.delete('1.3.6.1.2.1.2.2.1.7')\n row['ifOperStatus'] = row.delete('1.3.6.1.2.1.2.2.1.8')\n row['ifName'] = row.delete('1.3.6.1.2.1.31.1.1.1.1')\n row['ifAlias'] = row.delete('1.3.6.1.2.1.31.1.1.1.18')\n row['ifHighSpeed'] = row.delete('1.3.6.1.2.1.31.1.1.1.15')\n row['ifSpeed'] = row.delete('1.3.6.1.2.1.2.2.1.5')\n row['ifPhysAddress'] = row.delete('1.3.6.1.2.1.2.2.1.6')\n row['ifMtu'] = row.delete('1.3.6.1.2.1.2.2.1.4')\n row['ifHCInOctets'] = row.delete('1.3.6.1.2.1.31.1.1.1.6')\n row['ifHCOutOctets'] = row.delete('1.3.6.1.2.1.31.1.1.1.10')\n row['ifHCInBroadcastPkts'] = row.delete('1.3.6.1.2.1.31.1.1.1.9')\n row['ifHCOutBroadcastPkts'] = row.delete('1.3.6.1.2.1.31.1.1.1.13')\n row['ifHCInUcastPkts'] = row.delete('1.3.6.1.2.1.31.1.1.1.7')\n row['ifHCOutUcastPkts'] = row.delete('1.3.6.1.2.1.31.1.1.1.11')\n row['ifHCInMulticastPkts'] = row.delete('1.3.6.1.2.1.31.1.1.1.8')\n row['ifHCOutMulticastPkts'] = row.delete('1.3.6.1.2.1.31.1.1.1.12')\n row['ifLastChange'] = row.delete('1.3.6.1.2.1.2.2.1.9')\n row['dot1qPvid'] = row.delete('1.3.6.1.2.1.17.7.1.4.5.1.1')\n row['ifInErrors'] = row.delete('1.3.6.1.2.1.2.2.1.14')\n row['ifOutErrors'] = row.delete('1.3.6.1.2.1.2.2.1.20')\n row['ifInDiscards'] = row.delete('1.3.6.1.2.1.2.2.1.13')\n row['ifOutDiscards'] = row.delete('1.3.6.1.2.1.2.2.1.19')\n new_event = LogStash::Event.new({\n '@timestamp' => timestamp,\n 'host' => { 'name' => host_name, 'hostname' => host_hostname },\n 'network' => { 'name' => network_name },\n 'table' => row,\n 'metricset' => { 'module' => 'snmp' },\n 'event' => { 'kind' => 'interfaces' }\n })\n new_event_block.call(new_event)\n end\n event.remove('[interfaces]')\n event.set('[event][kind]', 'metrics')\nend" - }, - "comments": [] - } - ], - "output": [ - { - "id": "output_elasticsearch_10", - "type": "output", - "plugin": "elasticsearch", - "config": { - "data_stream": "true", - "data_stream_type": "metrics", - "data_stream_namespace": "default", - "data_stream_dataset": "snmp.polling", - "cloud_id": "test", - "user": "test", - "password": "test" - }, - "comments": [] - } - ] -} diff --git a/src/logstashui/Common/tests/conversion_data/components/test-string-escaping.json b/src/logstashui/Common/tests/conversion_data/components/test-string-escaping.json deleted file mode 100644 index 1216586..0000000 --- a/src/logstashui/Common/tests/conversion_data/components/test-string-escaping.json +++ /dev/null @@ -1,60 +0,0 @@ -{ - "input": [], - "filter": [ - { - "id": "filter_mutate_0", - "type": "filter", - "plugin": "mutate", - "config": { - "add_field": { - "double_quoted_with_hash": "value # not a comment", - "with_brackets": "data [in] brackets {and} braces", - "with_arrow": "key => value pattern", - "env_ref": "${MY_VAR}", - "sprintf_ref": "prefix-%{field_name}" - } - }, - "comments": [] - }, - { - "id": "filter_grok_1", - "type": "filter", - "plugin": "grok", - "config": { - "match": { - "message": "%{IP:client} \\[%{HTTPDATE:ts}\\] \"%{WORD:method} %{URIPATHPARAM:path}" - }, - "pattern_definitions": { - "CUSTOM_IP": "\\b(?:\\d{1,3}\\.){3}\\d{1,3}\\b" - } - }, - "comments": [] - }, - { - "id": "filter_mutate_2", - "type": "filter", - "plugin": "mutate", - "config": { - "rename": { - "@timestamp": "event_time", - "host": "source_host" - } - }, - "comments": [] - } - ], - "output": [ - { - "id": "output_file_3", - "type": "output", - "plugin": "file", - "config": { - "path": "/var/log/output/%{type}/%{+YYYY}/%{+MM}/%{+dd}.log", - "codec": { - "json": {} - } - }, - "comments": [] - } - ] -} diff --git a/src/logstashui/Common/tests/conversion_data/components/test-twitter.json b/src/logstashui/Common/tests/conversion_data/components/test-twitter.json deleted file mode 100644 index e2dcd67..0000000 --- a/src/logstashui/Common/tests/conversion_data/components/test-twitter.json +++ /dev/null @@ -1,89 +0,0 @@ -{ - "input": [ - { - "id": "input_comment_0", - "type": "input", - "plugin": "comment", - "config": { - "text": "This is the sample pipeline whose screenshots are used in\nthe Pipeline Viewer documentation (../pipeline-viewer.asciidoc)\n\nWhenever the Pipeline Viewer UI changes, run this pipeline and\nopen in the new UI to take updated screenshots.\n\nNote: you will have to setup the environment variables used\nbelow. Refer to the Twitter Logstash Input plugin documentation\nfor their expected values" - } - }, - { - "id": "input_twitter_1", - "type": "input", - "plugin": "twitter", - "config": { - "id": "tweet harvester", - "consumer_key": "${TWITTER_API_CONSUMER_KEY}", - "consumer_secret": "${TWITTER_API_CONSUMER_SECRET}", - "keywords": [ - "rain", - "monsoon", - "shower", - "drizzle" - ], - "oauth_token": "${TWITTER_API_OAUTH_TOKEN}", - "oauth_token_secret": "${TWITTER_API_OAUTH_TOKEN_SECRET}" - }, - "comments": [] - } - ], - "filter": [ - { - "id": "filter_grok_2", - "type": "filter", - "plugin": "grok", - "config": { - "match": { - "message": "%{WORD:is_rt}" - } - }, - "comments": [] - }, - { - "id": "filter_if_3", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[is_rt] == \"RT\"", - "plugins": [ - { - "id": "filter_drop_4", - "type": "filter", - "plugin": "drop", - "config": { - "id": "drop_all_RTs" - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - } - ], - "output": [ - { - "id": "output_stdout_5", - "type": "output", - "plugin": "stdout", - "config": { - "codec": { - "dots": {} - } - }, - "comments": [] - }, - { - "id": "output_elasticsearch_6", - "type": "output", - "plugin": "elasticsearch", - "config": { - "user": "elastic", - "password": "changeme", - "index": "tweets" - }, - "comments": [] - } - ] -} diff --git a/src/logstashui/Common/tests/conversion_data/components/test_complex1.json b/src/logstashui/Common/tests/conversion_data/components/test_complex1.json deleted file mode 100644 index ccaef73..0000000 --- a/src/logstashui/Common/tests/conversion_data/components/test_complex1.json +++ /dev/null @@ -1,952 +0,0 @@ -{ - "input": [ - { - "id": "input_comment_0", - "type": "input", - "plugin": "comment", - "config": { - "text": "\n\"LogstashUI kitchen sink\" pipeline\nGoal: be extremely feature-rich while staying within known-valid plugin options.\n\nBeats / Elastic Agent style shippers" - } - }, - { - "id": "input_beats_1", - "type": "input", - "plugin": "beats", - "config": { - "id": "in_beats_5044", - "port": "5044", - "add_field": { - "ingest_transport": "beats" - }, - "tags": [ - "from_beats" - ] - }, - "comments": [] - }, - { - "id": "input_comment_2", - "type": "input", - "plugin": "comment", - "config": { - "text": "JSON-over-TCP (common for app logs)" - } - }, - { - "id": "input_tcp_3", - "type": "input", - "plugin": "tcp", - "config": { - "id": "in_tcp_json_5514", - "port": "5514", - "mode": "server", - "codec": { - "json": {} - }, - "add_field": { - "ingest_transport": "tcp" - }, - "tags": [ - "from_tcp" - ] - }, - "comments": [] - }, - { - "id": "input_comment_4", - "type": "input", - "plugin": "comment", - "config": { - "text": "Syslog-ish UDP" - } - }, - { - "id": "input_udp_5", - "type": "input", - "plugin": "udp", - "config": { - "id": "in_udp_5515", - "port": "5515", - "codec": { - "plain": {} - }, - "add_field": { - "ingest_transport": "udp" - }, - "tags": [ - "from_udp" - ] - }, - "comments": [] - }, - { - "id": "input_comment_6", - "type": "input", - "plugin": "comment", - "config": { - "text": "HTTP event intake (webhooks, apps posting JSON, etc.)" - } - }, - { - "id": "input_http_7", - "type": "input", - "plugin": "http", - "config": { - "id": "in_http_8080", - "port": "8080", - "codec": { - "json": {} - }, - "add_field": { - "ingest_transport": "http" - }, - "tags": [ - "from_http" - ] - }, - "comments": [] - }, - { - "id": "input_comment_8", - "type": "input", - "plugin": "comment", - "config": { - "text": "Local dev/testing input" - } - }, - { - "id": "input_stdin_9", - "type": "input", - "plugin": "stdin", - "config": { - "id": "in_stdin", - "codec": { - "line": {} - }, - "add_field": { - "ingest_transport": "stdin" - }, - "tags": [ - "from_stdin" - ] - }, - "comments": [] - }, - { - "id": "input_comment_10", - "type": "input", - "plugin": "comment", - "config": { - "text": "Synthetic test data (makes it easy to validate end-to-end quickly)" - } - }, - { - "id": "input_generator_11", - "type": "input", - "plugin": "generator", - "config": { - "id": "in_generator", - "lines": [ - "Feb 21 09:12:01 host1 sshd[123]: Failed password for invalid user admin from 10.1.2.3 port 51234 ssh2", - "{\"@timestamp\":\"2026-02-21T14:12:02Z\",\"message\":\"GET /health 200\",\"source_ip\":\"8.8.8.8\",\"user_agent\":\"Mozilla/5.0\"}", - "level=info service=api latency_ms=42 source_ip=192.168.1.50 msg=\"request completed\"" - ], - "count": "1", - "add_field": { - "ingest_transport": "generator" - }, - "tags": [ - "from_generator" - ] - }, - "comments": [] - } - ], - "filter": [ - { - "id": "filter_comment_12", - "type": "filter", - "plugin": "comment", - "config": { - "text": "\nNormalize a few shared fields\n" - } - }, - { - "id": "filter_mutate_13", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "f_mutate_bootstrap", - "add_field": { - "[@metadata][pipeline]": "logstashui_kitchen_sink", - "event.module": "logstashui" - } - }, - "comments": [] - }, - { - "id": "filter_comment_14", - "type": "filter", - "plugin": "comment", - "config": { - "text": "Keep a canonical message field" - } - }, - { - "id": "filter_if_15", - "type": "filter", - "plugin": "if", - "config": { - "condition": "![message] and [event][original]", - "plugins": [ - { - "id": "filter_mutate_16", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "f_mutate_event_original_to_message", - "copy": { - "[event][original]": "message" - } - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_comment_17", - "type": "filter", - "plugin": "comment", - "config": { - "text": "\nTry to parse JSON *if* message looks like JSON (common when tcp/udp/plain feed JSON strings)\n" - } - }, - { - "id": "filter_if_18", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[message] =~ \"^[[:space:]]*\\\\{\"", - "plugins": [ - { - "id": "filter_json_19", - "type": "filter", - "plugin": "json", - "config": { - "id": "f_json_from_message", - "source": "message", - "target": "json", - "tag_on_failure": [ - "_jsonparsefailure_message" - ] - }, - "comments": [] - }, - { - "id": "filter_comment_20", - "type": "filter", - "plugin": "comment", - "config": { - "text": "If json parsed, promote a few expected keys (only if present)" - } - }, - { - "id": "filter_if_21", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[json][@timestamp]", - "plugins": [ - { - "id": "filter_mutate_22", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "f_promote_json_ts", - "copy": { - "[json][@timestamp]": "@timestamp" - } - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_if_23", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[json][source_ip]", - "plugins": [ - { - "id": "filter_mutate_24", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "f_promote_json_source_ip", - "copy": { - "[json][source_ip]": "source_ip" - } - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_if_25", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[json][user_agent]", - "plugins": [ - { - "id": "filter_mutate_26", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "f_promote_json_ua", - "copy": { - "[json][user_agent]": "user_agent" - } - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_comment_27", - "type": "filter", - "plugin": "comment", - "config": { - "text": "\nSyslog-ish parsing (UDP and some TCP)\n" - } - }, - { - "id": "filter_if_28", - "type": "filter", - "plugin": "if", - "config": { - "condition": "\"from_udp\" in [tags] or \"from_tcp\" in [tags]", - "plugins": [ - { - "id": "filter_comment_29", - "type": "filter", - "plugin": "comment", - "config": { - "text": "Try dissect first (fast) and fall back to grok" - } - }, - { - "id": "filter_dissect_30", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "f_dissect_syslogish", - "mapping": { - "message": "%{syslog_timestamp} %{syslog_host} %{syslog_program}[%{syslog_pid}]: %{syslog_message}" - }, - "tag_on_failure": [ - "_dissectfailure_syslogish" - ] - }, - "comments": [] - }, - { - "id": "filter_if_31", - "type": "filter", - "plugin": "if", - "config": { - "condition": "\"_dissectfailure_syslogish\" in [tags]", - "plugins": [ - { - "id": "filter_grok_32", - "type": "filter", - "plugin": "grok", - "config": { - "id": "f_grok_syslogish", - "match": { - "message": [ - "%{SYSLOGTIMESTAMP:syslog_timestamp} %{HOSTNAME:syslog_host} %{DATA:syslog_program}(?:\\[%{POSINT:syslog_pid}\\])?: %{GREEDYDATA:syslog_message}" - ] - }, - "tag_on_failure": [ - "_grokparsefailure_syslogish" - ] - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_comment_33", - "type": "filter", - "plugin": "comment", - "config": { - "text": "If we extracted a syslog timestamp, use it" - } - }, - { - "id": "filter_if_34", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[syslog_timestamp]", - "plugins": [ - { - "id": "filter_date_35", - "type": "filter", - "plugin": "date", - "config": { - "id": "f_date_syslog", - "match": [ - "syslog_timestamp", - "MMM d HH:mm:ss", - "MMM dd HH:mm:ss" - ], - "tag_on_failure": [ - "_dateparsefailure_syslog" - ] - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_comment_36", - "type": "filter", - "plugin": "comment", - "config": { - "text": "\nkey=value parsing for \u201cflat\u201d log lines\n" - } - }, - { - "id": "filter_if_37", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[message] =~ \"([A-Za-z0-9_.-]+)=([^\\\"]\\\\S+|\\\"[^\\\"]*\\\")\"", - "plugins": [ - { - "id": "filter_kv_38", - "type": "filter", - "plugin": "kv", - "config": { - "id": "f_kv_message", - "source": "message", - "trim_key": " ", - "trim_value": " ", - "value_split": "=", - "field_split_pattern": "\\s+", - "tag_on_failure": [ - "_kvfailure_message" - ] - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_comment_39", - "type": "filter", - "plugin": "comment", - "config": { - "text": "\nBasic typing / normalization\n" - } - }, - { - "id": "filter_mutate_40", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "f_mutate_normalize", - "rename": { - "msg": "message_short" - }, - "convert": { - "latency_ms": "integer" - }, - "lowercase": [ - "level" - ] - }, - "comments": [] - }, - { - "id": "filter_comment_41", - "type": "filter", - "plugin": "comment", - "config": { - "text": "\nEnrichments: useragent, geoip, cidr, dns\n" - } - }, - { - "id": "filter_if_42", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[user_agent]", - "plugins": [ - { - "id": "filter_useragent_43", - "type": "filter", - "plugin": "useragent", - "config": { - "id": "f_useragent", - "source": "user_agent", - "target": "user_agent_parsed" - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_comment_44", - "type": "filter", - "plugin": "comment", - "config": { - "text": "Canonicalize IP into source_ip if it exists elsewhere" - } - }, - { - "id": "filter_if_45", - "type": "filter", - "plugin": "if", - "config": { - "condition": "![source_ip] and [source][ip]", - "plugins": [ - { - "id": "filter_mutate_46", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "f_copy_source_ip", - "copy": { - "[source][ip]": "source_ip" - } - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_if_47", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[source_ip]", - "plugins": [ - { - "id": "filter_comment_48", - "type": "filter", - "plugin": "comment", - "config": { - "text": "Tag private vs public" - } - }, - { - "id": "filter_cidr_49", - "type": "filter", - "plugin": "cidr", - "config": { - "id": "f_cidr_private", - "address": [ - "%{source_ip}" - ], - "network": [ - "10.0.0.0/8", - "172.16.0.0/12", - "192.168.0.0/16" - ], - "add_tag": [ - "src_private" - ] - }, - "comments": [] - }, - { - "id": "filter_comment_50", - "type": "filter", - "plugin": "comment", - "config": { - "text": "GeoIP typically only makes sense for public IPs, so do it only if not private-tagged" - } - }, - { - "id": "filter_if_51", - "type": "filter", - "plugin": "if", - "config": { - "condition": "\"src_private\" not in [tags]", - "plugins": [ - { - "id": "filter_geoip_52", - "type": "filter", - "plugin": "geoip", - "config": { - "id": "f_geoip", - "source": "source_ip", - "target": "source_geo" - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_comment_53", - "type": "filter", - "plugin": "comment", - "config": { - "text": "Reverse DNS lookup; replace source_ip with hostname when possible (or leave as-is)" - } - }, - { - "id": "filter_dns_54", - "type": "filter", - "plugin": "dns", - "config": { - "id": "f_dns_reverse", - "reverse": [ - "source_ip" - ], - "action": "replace" - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_comment_55", - "type": "filter", - "plugin": "comment", - "config": { - "text": "\nTranslate severity/level into a normalized numeric\n" - } - }, - { - "id": "filter_translate_56", - "type": "filter", - "plugin": "translate", - "config": { - "id": "f_translate_level_to_severity", - "source": "level", - "target": "severity", - "dictionary": { - "trace": "0", - "debug": "1", - "info": "2", - "warn": "3", - "error": "4", - "fatal": "5" - }, - "fallback": "2" - }, - "comments": [] - }, - { - "id": "filter_mutate_57", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "f_convert_severity_int", - "convert": { - "severity": "integer" - } - }, - "comments": [] - }, - { - "id": "filter_comment_58", - "type": "filter", - "plugin": "comment", - "config": { - "text": "\nStable fingerprint for dedup / correlation\n" - } - }, - { - "id": "filter_fingerprint_59", - "type": "filter", - "plugin": "fingerprint", - "config": { - "id": "f_fingerprint_message", - "source": [ - "message" - ], - "method": "MURMUR3", - "target": "[@metadata][fingerprint]" - }, - "comments": [] - }, - { - "id": "filter_comment_60", - "type": "filter", - "plugin": "comment", - "config": { - "text": "\nExample branching: treat auth-ish messages specially\n" - } - }, - { - "id": "filter_if_61", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[syslog_program] == \"sshd\" or [message] =~ \"(?i)failed password|authentication failure|invalid user\"", - "plugins": [ - { - "id": "filter_mutate_62", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "f_tag_auth", - "add_tag": [ - "category_auth" - ], - "add_field": { - "event.category": "authentication" - } - }, - "comments": [] - } - ], - "else_ifs": [ - { - "condition": "[message] =~ \"(?i)GET\\\\s+/health|/ready|/live\"", - "plugins": [ - { - "id": "filter_mutate_63", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "f_tag_health", - "add_tag": [ - "category_healthcheck" - ], - "add_field": { - "event.category": "availability" - } - }, - "comments": [] - } - ] - } - ], - "else": { - "plugins": [ - { - "id": "filter_mutate_64", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "f_tag_generic", - "add_tag": [ - "category_generic" - ] - }, - "comments": [] - } - ] - } - } - }, - { - "id": "filter_comment_65", - "type": "filter", - "plugin": "comment", - "config": { - "text": "\nPrune down noisy fields (keeps top-level essentials)\n" - } - }, - { - "id": "filter_prune_66", - "type": "filter", - "plugin": "prune", - "config": { - "id": "f_prune", - "whitelist_names": [ - "^@timestamp$", - "^message$", - "^message_short$", - "^host$", - "^source_ip$", - "^source_geo$", - "^severity$", - "^level$", - "^tags$", - "^event\\..*$", - "^user_agent.*$", - "^syslog_.*$", - "^ingest_transport$" - ] - }, - "comments": [] - } - ], - "output": [ - { - "id": "output_comment_67", - "type": "output", - "plugin": "comment", - "config": { - "text": "Always see something in console during dev" - } - }, - { - "id": "output_stdout_68", - "type": "output", - "plugin": "stdout", - "config": { - "id": "out_stdout_rubydebug", - "codec": { - "rubydebug": { - "metadata": "true" - } - } - }, - "comments": [] - }, - { - "id": "output_comment_69", - "type": "output", - "plugin": "comment", - "config": { - "text": "Write to disk (great for debugging replay)" - } - }, - { - "id": "output_file_70", - "type": "output", - "plugin": "file", - "config": { - "id": "out_file_jsonl", - "path": "/tmp/logstashui-%{+YYYY.MM.dd}.jsonl", - "codec": { - "json_lines": {} - } - }, - "comments": [] - }, - { - "id": "output_comment_71", - "type": "output", - "plugin": "comment", - "config": { - "text": "Elasticsearch (local default)" - } - }, - { - "id": "output_elasticsearch_72", - "type": "output", - "plugin": "elasticsearch", - "config": { - "id": "out_es_local", - "hosts": [ - "http://localhost:9200" - ], - "index": "logstashui-%{+YYYY.MM.dd}", - "ilm_enabled": "false" - }, - "comments": [] - }, - { - "id": "output_comment_73", - "type": "output", - "plugin": "comment", - "config": { - "text": "Webhook back to your UI/API (example)" - } - }, - { - "id": "output_http_74", - "type": "output", - "plugin": "http", - "config": { - "id": "out_http_callback", - "url": "http://localhost:9000/logstash/callback", - "http_method": "post", - "format": "json" - }, - "comments": [] - }, - { - "id": "output_comment_75", - "type": "output", - "plugin": "comment", - "config": { - "text": "Kafka (example)" - } - }, - { - "id": "output_kafka_76", - "type": "output", - "plugin": "kafka", - "config": { - "id": "out_kafka", - "bootstrap_servers": "localhost:9092", - "topic_id": "logstashui-events" - }, - "comments": [] - }, - { - "id": "output_comment_77", - "type": "output", - "plugin": "comment", - "config": { - "text": "Pipeline-to-pipeline (requires another pipeline with pipeline input address => \"downstream\")" - } - }, - { - "id": "output_pipeline_78", - "type": "output", - "plugin": "pipeline", - "config": { - "id": "out_pipeline_downstream", - "send_to": [ - "downstream" - ] - }, - "comments": [] - } - ] -} diff --git a/src/logstashui/Common/tests/conversion_data/components/test_elasticdocs-conditional.json b/src/logstashui/Common/tests/conversion_data/components/test_elasticdocs-conditional.json deleted file mode 100644 index 39c72eb..0000000 --- a/src/logstashui/Common/tests/conversion_data/components/test_elasticdocs-conditional.json +++ /dev/null @@ -1,116 +0,0 @@ -{ - "input": [ - { - "id": "input_file_0", - "type": "input", - "plugin": "file", - "config": { - "path": "/tmp/*_log" - }, - "comments": [] - } - ], - "filter": [ - { - "id": "filter_if_1", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[path] =~ \"access\"", - "plugins": [ - { - "id": "filter_mutate_2", - "type": "filter", - "plugin": "mutate", - "config": { - "replace": { - "type": "apache_access" - } - }, - "comments": [] - }, - { - "id": "filter_grok_3", - "type": "filter", - "plugin": "grok", - "config": { - "match": { - "message": "%{COMBINEDAPACHELOG}" - } - }, - "comments": [] - }, - { - "id": "filter_date_4", - "type": "filter", - "plugin": "date", - "config": { - "match": [ - "timestamp", - "dd/MMM/yyyy:HH:mm:ss Z" - ] - }, - "comments": [] - } - ], - "else_ifs": [ - { - "condition": "[path] =~ \"error\"", - "plugins": [ - { - "id": "filter_mutate_5", - "type": "filter", - "plugin": "mutate", - "config": { - "replace": { - "type": "apache_error" - } - }, - "comments": [] - } - ] - } - ], - "else": { - "plugins": [ - { - "id": "filter_mutate_6", - "type": "filter", - "plugin": "mutate", - "config": { - "replace": { - "type": "random_logs" - } - }, - "comments": [] - } - ] - } - } - } - ], - "output": [ - { - "id": "output_elasticsearch_7", - "type": "output", - "plugin": "elasticsearch", - "config": { - "hosts": [ - "localhost:9200" - ] - }, - "comments": [] - }, - { - "id": "output_stdout_8", - "type": "output", - "plugin": "stdout", - "config": { - "codec": { - "rubydebug": {} - } - }, - "comments": [] - } - ] -} diff --git a/src/logstashui/Common/tests/conversion_data/components/text-complex4.json b/src/logstashui/Common/tests/conversion_data/components/text-complex4.json deleted file mode 100644 index da47a4e..0000000 --- a/src/logstashui/Common/tests/conversion_data/components/text-complex4.json +++ /dev/null @@ -1,386 +0,0 @@ -{ - "input": [ - { - "id": "input_generator_0", - "type": "input", - "plugin": "generator", - "config": { - "id": "gen_edgeA", - "count": 1, - "lines": [ - "2026-02-22T01:23:45Z level=INFO service=api trace.id=abc123 method=GET path=\"/api/v2/items/42\" ip=8.8.8.8 ua=\"Mozilla/5.0 (Windows NT 10.0; Win64; x64)\" msg=\"hello\\world\"" - ], - "add_field": { - "[@metadata][source]": "generator", - "event.original": "%{message}" - } - }, - "comments": [] - } - ], - "filter": [ - { - "id": "filter_comment_1", - "type": "filter", - "plugin": "comment", - "config": { - "text": "Fast split: timestamp + remainder" - } - }, - { - "id": "filter_dissect_2", - "type": "filter", - "plugin": "dissect", - "config": { - "id": "dissect_ts_rest", - "mapping": { - "message": "%{ts} %{rest}" - }, - "tag_on_failure": [ - "_dissectfailure_ts_rest" - ] - }, - "comments": [] - }, - { - "id": "filter_date_3", - "type": "filter", - "plugin": "date", - "config": { - "id": "date_ts", - "match": [ - "ts", - "ISO8601" - ], - "tag_on_failure": [ - "_dateparsefailure_ts" - ] - }, - "comments": [] - }, - { - "id": "filter_comment_4", - "type": "filter", - "plugin": "comment", - "config": { - "text": "Parse key=value in rest" - } - }, - { - "id": "filter_kv_5", - "type": "filter", - "plugin": "kv", - "config": { - "id": "kv_rest", - "source": "rest", - "trim_key": " ", - "trim_value": " ", - "value_split": "=", - "field_split_pattern": "\\s+", - "include_brackets": "false", - "tag_on_failure": [ - "_kvfailure_rest" - ] - }, - "comments": [] - }, - { - "id": "filter_comment_6", - "type": "filter", - "plugin": "comment", - "config": { - "text": "Normalize: remove surrounding quotes on selected fields (common log format)" - } - }, - { - "id": "filter_mutate_7", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "mutate_strip_quotes", - "gsub": [ - "path", - "^\"|\"$", - "", - "ua", - "^\"|\"$", - "", - "msg", - "^\"|\"$", - "" - ] - }, - "comments": [] - }, - { - "id": "filter_comment_8", - "type": "filter", - "plugin": "comment", - "config": { - "text": "Promote a few fields into ECS-ish places" - } - }, - { - "id": "filter_mutate_9", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "mutate_promote", - "rename": { - "ip": "[source][ip]", - "ua": "[user_agent][original]", - "method": "[http][request][method]", - "path": "[url][path]", - "trace.id": "[trace][id]" - }, - "lowercase": [ - "level" - ] - }, - "comments": [] - }, - { - "id": "filter_comment_10", - "type": "filter", - "plugin": "comment", - "config": { - "text": "Route key uses nested refs in sprintf (great UI test)" - } - }, - { - "id": "filter_mutate_11", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "mutate_route_key", - "add_field": { - "route_key": "%{[@metadata][source]}::%{[service]}::%{[http][request][method]}::%{[url][path]}" - } - }, - "comments": [] - }, - { - "id": "filter_comment_12", - "type": "filter", - "plugin": "comment", - "config": { - "text": "Regex literals (escaped slashes)" - } - }, - { - "id": "filter_if_13", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[url][path] =~ /^\\/api\\/v2\\/items\\/[0-9]+$/", - "plugins": [ - { - "id": "filter_mutate_14", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "tag_items", - "add_tag": [ - "route_items" - ] - }, - "comments": [] - } - ], - "else_ifs": [ - { - "condition": "[url][path] =~ /^\\/api\\/v2\\/[A-Za-z0-9._-]+$/", - "plugins": [ - { - "id": "filter_mutate_15", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "tag_api_generic", - "add_tag": [ - "route_api_generic" - ] - }, - "comments": [] - } - ] - } - ], - "else": { - "plugins": [ - { - "id": "filter_mutate_16", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "tag_other", - "add_tag": [ - "route_other" - ] - }, - "comments": [] - } - ] - } - } - }, - { - "id": "filter_comment_17", - "type": "filter", - "plugin": "comment", - "config": { - "text": "useragent parsing" - } - }, - { - "id": "filter_if_18", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[user_agent][original]", - "plugins": [ - { - "id": "filter_useragent_19", - "type": "filter", - "plugin": "useragent", - "config": { - "id": "ua_parse", - "source": "[user_agent][original]", - "target": "[user_agent][parsed]" - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_comment_20", - "type": "filter", - "plugin": "comment", - "config": { - "text": "geoip on public source.ip" - } - }, - { - "id": "filter_if_21", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[source][ip]", - "plugins": [ - { - "id": "filter_geoip_22", - "type": "filter", - "plugin": "geoip", - "config": { - "id": "geoip_source", - "source": "[source][ip]", - "target": "[source][geo]" - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_comment_23", - "type": "filter", - "plugin": "comment", - "config": { - "text": "fingerprint based on nested refs + message" - } - }, - { - "id": "filter_fingerprint_24", - "type": "filter", - "plugin": "fingerprint", - "config": { - "id": "fp_event", - "source": [ - "route_key", - "message" - ], - "method": "MURMUR3", - "target": "[@metadata][fp]" - }, - "comments": [] - }, - { - "id": "filter_comment_25", - "type": "filter", - "plugin": "comment", - "config": { - "text": "Replace literal backslash with slash in msg (escape-heavy but valid)" - } - }, - { - "id": "filter_mutate_26", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "mutate_gsub_backslash", - "gsub": [ - "msg", - "\\\\", - "/" - ] - }, - "comments": [] - }, - { - "id": "filter_prune_27", - "type": "filter", - "plugin": "prune", - "config": { - "id": "prune_edgeA", - "whitelist_names": [ - "^@timestamp$", - "^message$", - "^tags$", - "^level$", - "^service$", - "^route_key$", - "^trace\\..*$", - "^http\\..*$", - "^url\\..*$", - "^source\\..*$", - "^user_agent\\..*$", - "^msg$", - "^@metadata\\..*$" - ] - }, - "comments": [] - } - ], - "output": [ - { - "id": "output_stdout_28", - "type": "output", - "plugin": "stdout", - "config": { - "codec": { - "rubydebug": { - "metadata": "true" - } - } - }, - "comments": [] - }, - { - "id": "output_file_29", - "type": "output", - "plugin": "file", - "config": { - "path": "/tmp/edgeA-%{+YYYY.MM.dd}.jsonl", - "codec": { - "json_lines": {} - } - }, - "comments": [] - } - ] -} diff --git a/src/logstashui/Common/tests/conversion_data/components/text-complex5.json b/src/logstashui/Common/tests/conversion_data/components/text-complex5.json deleted file mode 100644 index 1e75f10..0000000 --- a/src/logstashui/Common/tests/conversion_data/components/text-complex5.json +++ /dev/null @@ -1,94 +0,0 @@ -{ - "input": [ - { - "id": "input_generator_0", - "type": "input", - "plugin": "generator", - "config": { - "id": "gen_edgecase_3", - "count": 1, - "lines": [ - "path=\"C:\\Program Files\\App\\\" msg=\"quote:\" and backslash:\\\\\"" - ] - }, - "comments": [] - } - ], - "filter": [ - { - "id": "filter_comment_1", - "type": "filter", - "plugin": "comment", - "config": { - "text": "This ruby code contains both quote styles and backslashes." - } - }, - { - "id": "filter_ruby_2", - "type": "filter", - "plugin": "ruby", - "config": { - "id": "ruby_edgecase_3", - "code": "\n # Double quotes inside single-quoted LSCL string\n event.set(\"[edge][note]\", \"He said: \"hello\"\")\n # Single quote inside Ruby string\n event.set(\"[edge][apostrophe]\", \"it's fine\")\n # Trailing backslash in a field value (nasty for serializers)\n event.set(\"[edge][trail]\", \"C:\temp\")\n " - }, - "comments": [] - }, - { - "id": "filter_comment_3", - "type": "filter", - "plugin": "comment", - "config": { - "text": "Parse key=val" - } - }, - { - "id": "filter_kv_4", - "type": "filter", - "plugin": "kv", - "config": { - "id": "kv_edgecase_3", - "source": "message", - "value_split": "=", - "field_split_pattern": "\\s+" - }, - "comments": [] - }, - { - "id": "filter_comment_5", - "type": "filter", - "plugin": "comment", - "config": { - "text": "Replace literal backslash \"\\\" with \"/\"" - } - }, - { - "id": "filter_mutate_6", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "gsub_edgecase_3", - "gsub": [ - "path", - "\\\\", - "/" - ] - }, - "comments": [] - } - ], - "output": [ - { - "id": "output_stdout_7", - "type": "output", - "plugin": "stdout", - "config": { - "codec": { - "rubydebug": { - "metadata": "true" - } - } - }, - "comments": [] - } - ] -} diff --git a/src/logstashui/Common/tests/conversion_data/components/text-complex6.json b/src/logstashui/Common/tests/conversion_data/components/text-complex6.json deleted file mode 100644 index 429bb20..0000000 --- a/src/logstashui/Common/tests/conversion_data/components/text-complex6.json +++ /dev/null @@ -1,469 +0,0 @@ -{ - "input": [ - { - "id": "input_kafka_0", - "type": "input", - "plugin": "kafka", - "config": { - "id": "input_kafka_1", - "topics": [ - "critical_business_events", - "low_latency_metrics" - ], - "bootstrap_servers": "kafka1:9092,kafka2:9092", - "group_id": "logstash_critical_group", - "codec": { - "json_lines": {} - }, - "type": "business_event", - "tags": [ - "kafka_input", - "critical" - ], - "max_poll_records": "500" - }, - "comments": [] - }, - { - "id": "input_redis_1", - "type": "input", - "plugin": "redis", - "config": { - "id": "input_redis_1", - "host": "redis-cache.example.com", - "port": "6379", - "data_type": "list", - "key": "service_log_queue", - "type": "service_log", - "tags": [ - "redis_input", - "service_data" - ], - "codec": { - "plain": {} - } - }, - "comments": [] - }, - { - "id": "input_tcp_2", - "type": "input", - "plugin": "tcp", - "config": { - "id": "input_tcp_1", - "port": "9999", - "type": "audit_log", - "ssl_enable": "true", - "ssl_cert": "/etc/logstash/certs/logstash.crt", - "ssl_key": "/etc/logstash/certs/logstash.key", - "ssl_verify": "true", - "codec": { - "json": { - "delimiter": "\n" - } - }, - "tags": [ - "tcp_input", - "sensitive" - ] - }, - "comments": [] - } - ], - "filter": [ - { - "id": "filter_if_3", - "type": "filter", - "plugin": "if", - "config": { - "condition": "\"service_data\" in [tags]", - "plugins": [ - { - "id": "filter_json_4", - "type": "filter", - "plugin": "json", - "config": { - "id": "filter_json_1", - "source": "message", - "target": "parsed_service_log", - "remove_field": [ - "message" - ], - "add_tag": [ - "json_attempt" - ] - }, - "comments": [] - }, - { - "id": "filter_if_5", - "type": "filter", - "plugin": "if", - "config": { - "condition": "\"_jsonparsefailure\" in [tags]", - "plugins": [ - { - "id": "filter_grok_6", - "type": "filter", - "plugin": "grok", - "config": { - "id": "filter_grok_1", - "match": { - "message": "(?%{TIMESTAMP_ISO8601}) %{DATA:service_id} \\[%{LOGLEVEL:level}] %{NUMBER:req_id:int} - %{GREEDYDATA:log_msg}" - }, - "add_tag": [ - "grok_fallback_success" - ], - "remove_tag": [ - "_jsonparsefailure" - ] - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_if_7", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[parsed_service_log][sensitive_data] == true or [tags] =~ /_grokparsefailure/", - "plugins": [ - { - "id": "filter_mutate_8", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_1", - "gsub": [ - "message", - "(\\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Z|a-z]{2,}\\b)", - "email_masked" - ] - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_mutate_9", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_2", - "rename": { - "[parsed_service_log][level]": "log_level" - }, - "add_field": { - "correlation_id": "%{[parsed_service_log][request_id]}" - } - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_if_10", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[correlation_id] and [log_level]", - "plugins": [ - { - "id": "filter_aggregate_11", - "type": "filter", - "plugin": "aggregate", - "config": { - "id": "filter_aggregate_1", - "task_id": "%{correlation_id}", - "code": "\n if event.get('log_level') == 'START'\n map['start_time'] = event.get('@timestamp').time.to_f\n map['service'] = event.get('service_id')\n event.cancel\n elsif event.get('log_level') == 'END' and map['start_time']\n end_time = event.get('@timestamp').time.to_f\n duration = (end_time - map['start_time']) * 1000 # Duration in ms\n event.set('request_duration_ms', duration.round(3))\n event.set('service_name', map['service'])\n event.set('type', 'request_summary')\n end\n ", - "map_action": "create_or_update", - "push_map_as_event_on_timeout": "true", - "timeout": "60", - "timeout_code": "event.set('error_reason', 'Unmatched_START_Event')", - "timeout_task_id_field": "unmatched_correlation_id" - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_if_12", - "type": "filter", - "plugin": "if", - "config": { - "condition": "\"critical\" in [tags] or [type] == \"audit_log\"", - "plugins": [ - { - "id": "filter_translate_13", - "type": "filter", - "plugin": "translate", - "config": { - "id": "filter_translate_1", - "field": "tenant_id", - "destination": "tenant_name", - "dictionary_path": "/etc/logstash/dicts/tenant_map.yml", - "fallback": "Unknown_Tenant", - "refresh_interval": "600" - }, - "comments": [] - }, - { - "id": "filter_mutate_14", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_3", - "convert": { - "transaction_amount": "float" - }, - "remove_field": [ - "host", - "port" - ] - }, - "comments": [] - }, - { - "id": "filter_date_15", - "type": "filter", - "plugin": "date", - "config": { - "id": "filter_date_1", - "match": [ - "[event_time]", - "ISO8601", - "UNIX_MS" - ], - "target": "@timestamp", - "remove_tag": [ - "_dateparsefailure" - ] - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_if_16", - "type": "filter", - "plugin": "if", - "config": { - "condition": "(\"_grokparsefailure\" in [tags] or \"_jsonparsefailure\" in [tags]) and [type] != \"audit_log\"", - "plugins": [ - { - "id": "filter_mutate_17", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_4", - "add_tag": [ - "dlq_candidate", - "parsing_error" - ], - "add_field": { - "dlq_reason": "Parsing_Failed" - } - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_mutate_18", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_5", - "remove_tag": [ - "_jsonparsefailure", - "_grokparsefailure", - "_dateparsefailure" - ] - }, - "comments": [] - } - ], - "output": [ - { - "id": "output_if_19", - "type": "output", - "plugin": "if", - "config": { - "condition": "[tenant_name] =~ /^PRIORITY_/", - "plugins": [ - { - "id": "output_elasticsearch_20", - "type": "output", - "plugin": "elasticsearch", - "config": { - "id": "output_elasticsearch_1", - "hosts": [ - "https://es-priority:9200" - ], - "index": "tenant_priority-%{tenant_name}-%{+YYYY.MM}", - "workers": "1", - "manage_template": "false" - }, - "comments": [] - } - ], - "else_ifs": [ - { - "condition": "[tenant_name]", - "plugins": [ - { - "id": "output_elasticsearch_21", - "type": "output", - "plugin": "elasticsearch", - "config": { - "id": "output_elasticsearch_2", - "hosts": [ - "https://es-main:9200" - ], - "index": "tenant_general-%{+YYYY.MM.dd}", - "dlq_enabled": "true", - "dlq_path": "/var/lib/logstash/dlq" - }, - "comments": [] - } - ] - } - ], - "else": null - } - }, - { - "id": "output_if_22", - "type": "output", - "plugin": "if", - "config": { - "condition": "\"dlq_candidate\" in [tags]", - "plugins": [ - { - "id": "output_file_23", - "type": "output", - "plugin": "file", - "config": { - "id": "output_file_1", - "path": "/var/log/logstash/error_logs/dlq_parsing_failures.log", - "codec": { - "json_lines": { - "target": "original_event" - } - }, - "add_tag": [ - "s3_backup" - ] - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "output_if_24", - "type": "output", - "plugin": "if", - "config": { - "condition": "[type] == \"audit_log\" or [type] == \"request_summary\" or \"s3_backup\" in [tags]", - "plugins": [ - { - "id": "output_s3_25", - "type": "output", - "plugin": "s3", - "config": { - "id": "output_s3_1", - "bucket": "logstash-archive-bucket", - "region": "us-west-2", - "time_file": "15", - "size_file": "50", - "codec": { - "json_lines": {} - }, - "temporary_directory": "/tmp/logstash_s3_tmp" - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "output_if_26", - "type": "output", - "plugin": "if", - "config": { - "condition": "\"unmatched_correlation_id\" in [tags]", - "plugins": [ - { - "id": "output_tcp_27", - "type": "output", - "plugin": "tcp", - "config": { - "id": "output_tcp_1", - "host": "graylog-server.example.com", - "port": "12201", - "codec": { - "gelf": { - "level": 1, - "short_message": "Log Aggregation Timeout/Error: %{unmatched_correlation_id}" - } - } - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "output_if_28", - "type": "output", - "plugin": "if", - "config": { - "condition": "[log_level] =~ /(START|END|FATAL)/", - "plugins": [ - { - "id": "output_stdout_29", - "type": "output", - "plugin": "stdout", - "config": { - "id": "output_stdout_1", - "codec": { - "rubydebug": { - "metadata": "true" - } - } - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - } - ] -} diff --git a/src/logstashui/Common/tests/conversion_data/components/text-complex7.json b/src/logstashui/Common/tests/conversion_data/components/text-complex7.json deleted file mode 100644 index 60bdd43..0000000 --- a/src/logstashui/Common/tests/conversion_data/components/text-complex7.json +++ /dev/null @@ -1,558 +0,0 @@ -{ - "input": [ - { - "id": "input_beats_0", - "type": "input", - "plugin": "beats", - "config": { - "id": "input_beats_1", - "port": "5044", - "ssl": "true", - "ssl_certificate": "/etc/logstash/certs/logstash.crt", - "ssl_key": "/etc/logstash/certs/logstash.key", - "codec": { - "json": {} - }, - "tags": [ - "beats_input", - "app_log" - ] - }, - "comments": [] - }, - { - "id": "input_udp_1", - "type": "input", - "plugin": "udp", - "config": { - "id": "input_udp_1", - "port": "5140", - "buffer_size": "8192", - "codec": { - "plain": { - "charset": "UTF-8" - } - }, - "type": "network_flow", - "tags": [ - "udp_input", - "unstructured" - ] - }, - "comments": [] - }, - { - "id": "input_jdbc_2", - "type": "input", - "plugin": "jdbc", - "config": { - "id": "input_jdbc_1", - "jdbc_driver_library": "/usr/share/logstash/logstash-core/lib/jars/postgresql-42.2.8.jar", - "jdbc_driver_class": "org.postgresql.Driver", - "jdbc_connection_string": "jdbc:postgresql://db.example.com:5432/config_db", - "jdbc_user": "logstash_user", - "jdbc_password": "${JDBC_PASSWORD}", - "schedule": "0 * * * *", - "statement": "SELECT id, user_name, config_item, change_timestamp FROM config_changes WHERE change_timestamp > :sql_last_value ORDER BY change_timestamp ASC", - "use_column_value": "true", - "tracking_column": "change_timestamp", - "tracking_column_type": "timestamp", - "last_run_metadata_path": "/var/lib/logstash/.jdbc_last_run_config_db", - "type": "config_audit", - "tags": [ - "jdbc_input", - "audit" - ] - }, - "comments": [] - } - ], - "filter": [ - { - "id": "filter_mutate_3", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_1", - "rename": { - "@timestamp": "log_recv_time" - }, - "add_field": { - "severity": "INFO" - } - }, - "comments": [] - }, - { - "id": "filter_if_4", - "type": "filter", - "plugin": "if", - "config": { - "condition": "\"app_log\" in [tags]", - "plugins": [ - { - "id": "filter_if_5", - "type": "filter", - "plugin": "if", - "config": { - "condition": "\"_jsonparsefailure\" in [tags]", - "plugins": [ - { - "id": "filter_grok_6", - "type": "filter", - "plugin": "grok", - "config": { - "id": "filter_grok_1", - "match": { - "message": "(?\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}\\.\\d{3}) \\[%{DATA:thread}] %{LOGLEVEL:log_level} %{DATA:logger} - %{GREEDYDATA:log_message}" - }, - "add_tag": [ - "grok_fallback_success" - ], - "remove_tag": [ - "_jsonparsefailure" - ] - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_if_7", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[log_level]", - "plugins": [ - { - "id": "filter_mutate_8", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_2", - "uppercase": [ - "log_level" - ], - "copy": { - "log_level": "severity" - } - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_if_9", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[severity] =~ /(WARNING|ERROR|FATAL)/", - "plugins": [ - { - "id": "filter_geoip_10", - "type": "filter", - "plugin": "geoip", - "config": { - "id": "filter_geoip_1", - "source": "[fields][source_ip]", - "target": "geo", - "database": "/etc/logstash/geoip/GeoLite2-City.mmdb", - "remove_field": [ - "continent_code", - "location" - ] - }, - "comments": [] - }, - { - "id": "filter_translate_11", - "type": "filter", - "plugin": "translate", - "config": { - "id": "filter_translate_1", - "field": "[service_code]", - "destination": "service_name", - "dictionary_path": "/etc/logstash/dictionaries/service_codes.csv", - "fallback": "Unknown Service", - "refresh_interval": "300" - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "filter_mutate_12", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_3", - "remove_field": [ - "message", - "agent" - ] - }, - "comments": [] - } - ], - "else_ifs": [ - { - "condition": "[type] == \"network_flow\"", - "plugins": [ - { - "id": "filter_grok_13", - "type": "filter", - "plugin": "grok", - "config": { - "id": "filter_grok_2", - "match": { - "message": "%{NETFLOW_V9}" - }, - "on_failure": [ - "_netflowparsefailure" - ] - }, - "comments": [] - }, - { - "id": "filter_if_14", - "type": "filter", - "plugin": "if", - "config": { - "condition": "\"_netflowparsefailure\" in [tags]", - "plugins": [ - { - "id": "filter_mutate_15", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_4", - "add_tag": [ - "unparsed_flow" - ], - "remove_tag": [ - "_grokparsefailure", - "_netflowparsefailure" - ], - "copy": { - "message": "unparsed_data" - }, - "replace": { - "message": "Truncated unparsed flow data." - } - }, - "comments": [] - } - ], - "else_ifs": [], - "else": { - "plugins": [ - { - "id": "filter_aggregate_16", - "type": "filter", - "plugin": "aggregate", - "config": { - "id": "filter_aggregate_1", - "task_id": "%{source_ip}_%{destination_ip}_%{protocol}", - "code": "map['total_packets'] ||= 0; map['total_packets'] += event.get('packets').to_i; map['total_bytes'] ||= 0; map['total_bytes'] += event.get('bytes').to_i", - "map_action": "create_or_update", - "push_map_as_event_on_timeout": "true", - "timeout": "120", - "timeout_task_id_field": "aggregated_flow_id", - "timeout_tags": [ - "_aggregate_timeout" - ] - }, - "comments": [] - } - ] - } - } - } - ] - }, - { - "condition": "[type] == \"config_audit\"", - "plugins": [ - { - "id": "filter_if_17", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[change_timestamp]", - "plugins": [ - { - "id": "filter_date_18", - "type": "filter", - "plugin": "date", - "config": { - "id": "filter_date_1", - "match": [ - "change_timestamp", - "YYYY-MM-dd HH:mm:ss.SSSSSS" - ], - "target": "@timestamp", - "remove_field": [ - "change_timestamp" - ] - }, - "comments": [] - }, - { - "id": "filter_if_19", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[user_name] != \"system\"", - "plugins": [ - { - "id": "filter_ruby_20", - "type": "filter", - "plugin": "ruby", - "config": { - "id": "filter_ruby_1", - "code": "event.set('user_hash', Digest::MD5.hexdigest(event.get('user_name')))" - }, - "comments": [] - }, - { - "id": "filter_mutate_21", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_5", - "remove_field": [ - "user_name" - ] - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - } - ], - "else_ifs": [], - "else": null - } - } - ] - } - ], - "else": null - } - }, - { - "id": "filter_if_22", - "type": "filter", - "plugin": "if", - "config": { - "condition": "\"_grokparsefailure\" in [tags] or \"_jsonparsefailure\" in [tags]", - "plugins": [ - { - "id": "filter_mutate_23", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_6", - "add_field": { - "log_status": "FAILED_TO_PARSE" - } - }, - "comments": [] - } - ], - "else_ifs": [], - "else": { - "plugins": [ - { - "id": "filter_mutate_24", - "type": "filter", - "plugin": "mutate", - "config": { - "id": "filter_mutate_7", - "add_field": { - "log_status": "PROCESSED" - } - }, - "comments": [] - } - ] - } - } - }, - { - "id": "filter_if_25", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[log_recv_time] < now() - 86400000", - "plugins": [ - { - "id": "filter_drop_26", - "type": "filter", - "plugin": "drop", - "config": { - "id": "filter_drop_1" - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - } - ], - "output": [ - { - "id": "output_if_27", - "type": "output", - "plugin": "if", - "config": { - "condition": "[log_status] == \"PROCESSED\" and [severity] =~ /(ERROR|FATAL)/", - "plugins": [ - { - "id": "output_elasticsearch_28", - "type": "output", - "plugin": "elasticsearch", - "config": { - "id": "output_elasticsearch_1", - "hosts": [ - "https://es-hot.example.com:9200" - ], - "index": "high-priority-%{+YYYY.MM.dd}", - "user": "logstash_writer", - "password": "secure_password", - "ssl": "true", - "cacert": "/etc/logstash/certs/ca.crt", - "action": "index" - }, - "comments": [] - } - ], - "else_ifs": [ - { - "condition": "[log_status] == \"PROCESSED\"", - "plugins": [ - { - "id": "output_elasticsearch_29", - "type": "output", - "plugin": "elasticsearch", - "config": { - "id": "output_elasticsearch_2", - "hosts": [ - "https://es-warm.example.com:9200" - ], - "index": "general-logs-%{+YYYY.MM.dd}", - "user": "logstash_writer", - "password": "secure_password", - "ssl": "true", - "cacert": "/etc/logstash/certs/ca.crt", - "workers": "4", - "ilm_enabled": "false" - }, - "comments": [] - } - ] - } - ], - "else": null - } - }, - { - "id": "output_if_30", - "type": "output", - "plugin": "if", - "config": { - "condition": "[log_status] == \"FAILED_TO_PARSE\"", - "plugins": [ - { - "id": "output_file_31", - "type": "output", - "plugin": "file", - "config": { - "id": "output_file_1", - "path": "/var/log/logstash/dlq_failures.json", - "codec": { - "json": { - "pretty": "true" - } - }, - "add_tag": [ - "dlq_routed" - ] - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "output_if_32", - "type": "output", - "plugin": "if", - "config": { - "condition": "\"_aggregate_timeout\" in [tags]", - "plugins": [ - { - "id": "output_tcp_33", - "type": "output", - "plugin": "tcp", - "config": { - "id": "output_tcp_1", - "host": "alert-sys.example.com", - "port": "6514", - "codec": { - "gelf": { - "protocol": "TCP", - "short_message": "Aggregated Flow Timeout: %{aggregated_flow_id}" - } - }, - "socket_timeout": "5" - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - }, - { - "id": "output_if_34", - "type": "output", - "plugin": "if", - "config": { - "condition": "rand(100) < 1", - "plugins": [ - { - "id": "output_stdout_35", - "type": "output", - "plugin": "stdout", - "config": { - "id": "output_stdout_1", - "codec": { - "rubydebug": { - "metadata": "true" - } - } - }, - "comments": [] - } - ], - "else_ifs": [], - "else": null - } - } - ] -} diff --git a/src/logstashui/Common/tests/conversion_data/components/text-ls-repo-nginx-error.json b/src/logstashui/Common/tests/conversion_data/components/text-ls-repo-nginx-error.json deleted file mode 100644 index b247032..0000000 --- a/src/logstashui/Common/tests/conversion_data/components/text-ls-repo-nginx-error.json +++ /dev/null @@ -1,183 +0,0 @@ -{ - "input": [ - { - "id": "input_stdin_0", - "type": "input", - "plugin": "stdin", - "config": { - "codec": { - "line": {} - } - }, - "comments": [] - } - ], - "filter": [ - { - "id": "filter_mutate_1", - "type": "filter", - "plugin": "mutate", - "config": { - "add_field": { - "event.dataset": "nginx.access", - "service.name": "nginx" - } - }, - "comments": [] - }, - { - "id": "filter_grok_2", - "type": "filter", - "plugin": "grok", - "config": { - "match": { - "message": [ - "%{IPORHOST:source.address} %{DATA:nginx.ident} %{DATA:user.name} \\[%{HTTPDATE:nginx.access.time}\\] \"%{WORD:http.request.method} %{DATA:url.original} HTTP/%{NUMBER:http.version}\" %{NUMBER:http.response.status_code:int} (?:%{NUMBER:http.response.body.bytes:int}|-) \"%{DATA:http.request.referrer}\" \"%{DATA:user_agent.original}\" %{NUMBER:nginx.access.request_time:float}", - "%{IPORHOST:source.address} %{DATA:nginx.ident} %{DATA:user.name} \\[%{HTTPDATE:nginx.access.time}\\] \"%{WORD:http.request.method} %{DATA:url.original} HTTP/%{NUMBER:http.version}\" %{NUMBER:http.response.status_code:int} (?:%{NUMBER:http.response.body.bytes:int}|-) \"%{DATA:http.request.referrer}\" \"%{DATA:user_agent.original}\"", - "%{IPORHOST:source.address} %{DATA:nginx.ident} %{DATA:user.name} \\[%{HTTPDATE:nginx.access.time}\\] \"%{WORD:http.request.method} %{DATA:url.original} HTTP/%{NUMBER:http.version}\" %{NUMBER:http.response.status_code:int} (?:%{NUMBER:http.response.body.bytes:int}|-) \"%{DATA:http.request.referrer}\" \"%{DATA:user_agent.original}\" (?:rt=%{NUMBER:nginx.access.request_time:float}\\s+uct=%{NUMBER:nginx.access.upstream_connect_time:float}\\s+uht=%{NUMBER:nginx.access.upstream_header_time:float}\\s+urt=%{NUMBER:nginx.access.upstream_response_time:float})" - ] - }, - "tag_on_failure": [ - "_grok_nginx_access_fail" - ] - }, - "comments": [] - }, - { - "id": "filter_date_3", - "type": "filter", - "plugin": "date", - "config": { - "match": [ - "nginx.access.time", - "dd/MMM/yyyy:HH:mm:ss Z" - ], - "target": "@timestamp" - }, - "comments": [] - }, - { - "id": "filter_urldecode_4", - "type": "filter", - "plugin": "urldecode", - "config": { - "field": "url.original" - }, - "comments": [] - }, - { - "id": "filter_dissect_5", - "type": "filter", - "plugin": "dissect", - "config": { - "mapping": { - "url.original": "%{url.path}?%{url.query}" - } - }, - "comments": [] - }, - { - "id": "filter_useragent_6", - "type": "filter", - "plugin": "useragent", - "config": { - "source": "user_agent.original", - "target": "user_agent" - }, - "comments": [] - }, - { - "id": "filter_mutate_7", - "type": "filter", - "plugin": "mutate", - "config": { - "copy": { - "source.address": "source.ip" - } - }, - "comments": [] - }, - { - "id": "filter_geoip_8", - "type": "filter", - "plugin": "geoip", - "config": { - "source": "source.ip", - "target": "source.geo", - "tag_on_failure": [ - "_geoip_fail" - ] - }, - "comments": [] - }, - { - "id": "filter_mutate_9", - "type": "filter", - "plugin": "mutate", - "config": { - "gsub": [ - "http.request.referrer", - "^-$", - "", - "user.name", - "^-$", - "" - ] - }, - "comments": [] - }, - { - "id": "filter_if_10", - "type": "filter", - "plugin": "if", - "config": { - "condition": "[http][response][status_code] >= 500", - "plugins": [ - { - "id": "filter_mutate_11", - "type": "filter", - "plugin": "mutate", - "config": { - "add_tag": [ - "nginx_server_error" - ] - }, - "comments": [] - } - ], - "else_ifs": [ - { - "condition": "[http][response][status_code] >= 400", - "plugins": [ - { - "id": "filter_mutate_12", - "type": "filter", - "plugin": "mutate", - "config": { - "add_tag": [ - "nginx_client_error" - ] - }, - "comments": [] - } - ] - } - ], - "else": null - } - } - ], - "output": [ - { - "id": "output_stdout_13", - "type": "output", - "plugin": "stdout", - "config": { - "codec": { - "rubydebug": {} - } - }, - "comments": [] - } - ] -} diff --git a/src/logstashui/Common/tests/conversion_data/pipelines/ls-repo-apache2.conf b/src/logstashui/Common/tests/conversion_data/pipelines/ls-repo-apache2.conf deleted file mode 100644 index bf0eb1d..0000000 --- a/src/logstashui/Common/tests/conversion_data/pipelines/ls-repo-apache2.conf +++ /dev/null @@ -1,69 +0,0 @@ -input { - beats { - port => 5044 - host => "0.0.0.0" - } -} -filter { - if [fileset][module] == "apache2" { - if [fileset][name] == "access" { - grok { - match => { - "message" => [ - '%{IPORHOST:[apache2][access][remote_ip]} - %{DATA:[apache2][access][user_name]} \[%{HTTPDATE:[apache2][access][time]}\] "%{WORD:[apache2][access][method]} %{DATA:[apache2][access][url]} HTTP/%{NUMBER:[apache2][access][http_version]}" %{NUMBER:[apache2][access][response_code]} %{NUMBER:[apache2][access][body_sent][bytes]}( "%{DATA:[apache2][access][referrer]}")?( "%{DATA:[apache2][access][agent]}")?', - '%{IPORHOST:[apache2][access][remote_ip]} - %{DATA:[apache2][access][user_name]} \[%{HTTPDATE:[apache2][access][time]}\] "-" %{NUMBER:[apache2][access][response_code]} -' - ] - } - remove_field => "message" - } - mutate { - add_field => { - "read_timestamp" => "%{@timestamp}" - } - } - date { - match => ["[apache2][access][time]", "dd/MMM/YYYY:H:m:s Z"] - remove_field => "[apache2][access][time]" - } - useragent { - source => "[apache2][access][agent]" - target => "[apache2][access][user_agent]" - remove_field => "[apache2][access][agent]" - } - geoip { - source => "[apache2][access][remote_ip]" - target => "[apache2][access][geoip]" - } - } - else if [fileset][name] == "error" { - grok { - match => { - "message" => [ - "\[%{APACHE_TIME:[apache2][error][timestamp]}\] \[%{LOGLEVEL:[apache2][error][level]}\]( \[client %{IPORHOST:[apache2][error][client]}\])? %{GREEDYDATA:[apache2][error][message]}", - "\[%{APACHE_TIME:[apache2][error][timestamp]}\] \[%{DATA:[apache2][error][module]}:%{LOGLEVEL:[apache2][error][level]}\] \[pid %{NUMBER:[apache2][error][pid]}(:tid %{NUMBER:[apache2][error][tid]})?\]( \[client %{IPORHOST:[apache2][error][client]}\])? %{GREEDYDATA:[apache2][error][message1]}" - ] - } - pattern_definitions => { - "APACHE_TIME" => "%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR}" - } - remove_field => "message" - } - mutate { - rename => { - "[apache2][error][message1]" => "[apache2][error][message]" - } - } - date { - match => ["[apache2][error][timestamp]", "EEE MMM dd H:m:s YYYY", "EEE MMM dd H:m:s.SSSSSS YYYY"] - remove_field => "[apache2][error][timestamp]" - } - } - } -} -output { - elasticsearch { - hosts => "localhost" - manage_template => "false" - index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" - } -} diff --git a/src/logstashui/Common/tests/conversion_data/pipelines/test-asa-new.conf b/src/logstashui/Common/tests/conversion_data/pipelines/test-asa-new.conf deleted file mode 100644 index dbd9def..0000000 --- a/src/logstashui/Common/tests/conversion_data/pipelines/test-asa-new.conf +++ /dev/null @@ -1,897 +0,0 @@ -input { - udp { - id => "input_udp_1" - port => "5119" - } - cloudwatch { - } -} -filter { - mutate { - id => "filter_mutate_1" - rename => { - "message" => "log.original" - "host" => "observer.ip" - } - copy => { - "host" => "sysloghost" - } - } - grok { - id => "filter_grok_1" - match => { - "log.original" => [ - "%{CISCO_TAGGED_SYSLOG} %{GREEDYDATA:message}", - "^<%{POSINT:syslog_pri}>%{DATA}: %%{DATA:ciscotag}: %{GREEDYDATA:message}", - "^<%{POSINT:syslog_pri}>%%{DATA:ciscotag}: %{GREEDYDATA:message}" - ] - } - } - grok { - id => "filter_grok_2" - match => { - "ciscotag" => [ - "%{WORD}-%{INT:event.severity}-%{INT:event.code}", - "%{WORD}-%{WORD}-%{INT:event.severity}-%{INT:event.code}" - ] - } - } - mutate { - id => "filter_mutate_2" - add_field => { - "event.action" => "firewall-rule" - } - } - if [event.code] == "105012" { - grok { - id => "filter_grok_3" - match => { - "message" => [ - "Teardown dynamic %{WORD:network.transport} translation from %{DATA:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} to %{DATA:cisco.destination_interface}:%{IP:destination.address}/%{INT:destination.port} duration %{DATA:cisco.duration_hms}$" - ] - } - } - } - else if [event.code] == "106001" { - dissect { - id => "filter_dissect_1" - mapping => { - "message" => "%{network.direction} %{network.transport} connection %{event.outcome} from %{source.address}/%{source.port} to %{destination.address}/%{destination.port} flags %{} on interface %{source_interface}" - } - } - mutate { - id => "filter_mutate_3" - add_field => { - "event.category" => "network_traffic" - } - } - } - else if [event.code] == "106002" { - dissect { - id => "filter_dissect_2" - mapping => { - "message" => "%{network.transport} Connection %{event.outcome} by %{network.direction} list %{cisco.list_id} src %{source.address} dest %{destination.address}" - } - } - mutate { - id => "filter_mutate_4" - add_field => { - "event.category" => "network_traffic" - } - } - } - else if [event.code] == "106006" { - dissect { - id => "filter_dissect_3" - mapping => { - "message" => "%{event.outcome} %{network.direction} %{network.transport} from %{source.address}/%{source.port} to %{destination.address}/%{destination.port} on interface %{cisco.source_interface}" - } - } - mutate { - id => "filter_mutate_5" - add_field => { - "event.category" => "network_traffic" - } - } - } - else if [event.code] == "106007" { - dissect { - id => "filter_dissect_4" - mapping => { - "message" => "%{event.outcome} %{network.direction} %{network.transport} from %{source.address}/%{source.port} to %{destination.address}/%{destination.port} due to %{network.protocol} %{}" - } - } - mutate { - id => "filter_mutate_6" - add_field => { - "event.category" => "network_traffic" - } - } - } - else if [event.code] == "106010" { - dissect { - id => "filter_dissect_5" - mapping => { - "message" => "%{event.outcome} %{network.direction} %{network.transport} src %{cisco.source_interface}:%{source.address}/%{source.port} %{} dst %{cisco.destination_interface}:%{destination.address}/%{destination.port} %{}" - } - } - mutate { - id => "filter_mutate_7" - add_field => { - "event.category" => "network_traffic" - } - } - } - else if [event.code] == "106013" { - dissect { - id => "filter_dissect_6" - mapping => { - "message" => "Dropping echo request from %{source.address} to PAT address %{destination.address}" - } - } - mutate { - id => "filter_mutate_8" - add_field => { - "network.transport" => "icmp" - "network.direction" => "inbound" - } - } - } - else if [event.code] == "106014" { - dissect { - id => "filter_dissect_7" - mapping => { - "message" => "%{event.outcome} %{network.direction} %{network.transport} src %{cisco.source_interface}:%{source.address} %{}dst %{cisco.destination_interface}:%{destination.address} %{}" - } - } - mutate { - id => "filter_mutate_9" - add_field => { - "event.category" => "network_traffic" - } - } - } - else if [event.code] == "106015" { - dissect { - id => "filter_dissect_8" - mapping => { - "message" => "%{event.outcome} %{network.transport} (no connection) from %{source.address}/%{source.port} to %{destination.address}/%{destination.port} flags %{} on interface %{cisco.source_interface}" - } - } - mutate { - id => "filter_mutate_10" - add_field => { - "event.category" => "nat_translation" - } - } - } - else if [event.code] == "106016" { - dissect { - id => "filter_dissect_9" - mapping => { - "message" => "%{event.outcome} IP spoof from (%{source.address}) to %{destination.address} on interface %{cisco.source_interface}" - } - } - } - else if [event.code] == "106017" { - dissect { - id => "filter_dissect_10" - mapping => { - "message" => "%{event.outcome} IP due to Land Attack from %{source.address} to %{destination.address}" - } - } - } - else if [event.code] == "106018" { - dissect { - id => "filter_dissect_11" - mapping => { - "message" => "%{network.transport} packet type %{cisco.icmp_type} %{event.outcome} by %{network.direction} list %{cisco.list_id} src %{source.address} dest %{destination.address}" - } - } - } - else if [event.code] == "106020" { - dissect { - id => "filter_dissect_12" - mapping => { - "message" => "%{event.outcome} IP teardrop fragment (size = %{}, offset = %{}) from %{source.address} to %{destination.address}" - } - } - } - else if [event.code] == "106021" { - dissect { - id => "filter_dissect_13" - mapping => { - "message" => "%{event.outcome} %{network.transport} reverse path check from %{source.address} to %{destination.address} on interface %{cisco.source_interface}" - } - } - } - else if [event.code] == "106022" { - dissect { - id => "filter_dissect_14" - mapping => { - "message" => "%{event.outcome} %{network.transport} connection spoof from %{source.address} to %{destination.address} on interface %{cisco.source_interface}" - } - } - } - else if [event.code] == "106023" { - grok { - id => "filter_grok_4" - match => { - "message" => [ - '%{WORD:event.outcome} %{WORD:network.transport} src %{WORD:cisco.source_interface}:%{IPORHOST:source.address}?(/%{INT:source.port}) dst %{WORD:cisco.destination_interface}:%{IPORHOST:destination.address}?(/%{INT:destination.port}) by access-group "%{DATA:cisco.list_id}"', - '%{WORD:event.outcome} %{WORD:network.transport} src %{WORD:cisco.source_interface}:%{IPORHOST:source.address} dst %{WORD:destination.direction}:%{IPORHOST:destination.address} \(%{DATA}\) by access-group "%{DATA:cisco.list_id}"', - '%{WORD:event.outcome} %{WORD:network.transport} src %{WORD:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} dst %{WORD:cisco.destination.interface}:%{IPORHOST:destination.address}/%{INT:destination.port} by access-group "%{DATA:cisco.list_id}"' - ] - } - } - mutate { - id => "filter_mutate_11" - add_field => { - "event.category" => "network_traffic" - } - } - } - else if [event.code] == "106027" { - dissect { - id => "filter_dissect_15" - mapping => { - "message" => '%{} %{event.outcome} src %{source.address} dst %{destination.address} by access-group "%{cisco.list_id}"%{}' - } - } - } - else if [event.code] == "106100" { - dissect { - id => "filter_dissect_16" - mapping => { - "message" => "access-list %{cisco.list_id} %{event.outcome} %{network.transport} %{cisco.source_interface}/%{source.address}(%{source.port}) -> %{cisco.destination_interface}/%{destination.address}(%{destination.port}) %{}" - } - } - } - else if [event.code] == "106102" { - dissect { - id => "filter_dissect_17" - mapping => { - "message" => "access-list %{cisco.list_id} %{event.outcome} %{network.transport} for user %{cisco.username} %{cisco.source_interface}/%{source.address} %{source.port} %{cisco.destination_interface}/%{destination.address} %{destination.port} %{}" - } - } - } - else if [event.code] == "106103" { - dissect { - id => "filter_dissect_18" - mapping => { - "message" => "access-list %{cisco.list_id} %{event.outcome} %{network.transport} for user %{cisco.username} %{cisco.source_interface}/%{source.address} %{source.port} %{cisco.destination_interface}/%{destination.address} %{destination.port} %{}" - } - } - } - else if [event.code] == "113004" { - grok { - id => "filter_grok_5" - match => { - "message" => [ - "AAA user accounting %{WORD:cisco.auth_outcome} : server =%{SPACE}%{IP:source.address} : user =%{SPACE}%{DATA:source.user.name}$" - ] - } - } - mutate { - id => "filter_mutate_12" - add_field => { - "event.category" => "authentication" - } - } - if [cisco.auth_outcome] == "Successful" { - mutate { - id => "filter_mutate_13" - add_field => { - "event.action" => "authentication_success" - } - } - } - else { - mutate { - id => "filter_mutate_14" - add_field => { - "event.action" => "authentication_failure" - } - } - } - } - else if [event.code] == "302015" or [event.code] == "302013" { - grok { - id => "filter_grok_6" - match => { - "message" => [ - "Built %{WORD:network.direction} %{WORD:network.transport} connection %{INT} for %{DATA:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} \(%{IP}|\) to %{DATA:cisco.destination_interface}:%{IPORHOST:destination.address}/%{INT:destination.port} \(%{DATA}\)", - "Built %{WORD:network.direction} %{WORD:network.transport} connection %{INT} for %{DATA:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} \(%{DATA}\)?(\(%{DATA:cisco.source_username}\)) to %{DATA:cisco.destination_interface}:%{IPORHOST:destination.address}/%{INT:destination.port} \(%{DATA}\) ?(\(%{DATA:cisco.username}\))", - "Built %{WORD:network.direction} %{WORD:network.transport} connection %{INT:cisco.connection_id} for %{WORD:cisco.source_interface}:%{IPORHOST:source.address}\/%{INT:source.port} \(%{DATA}\) to %{WORD:cisco.destination_interface}:%{IPORHOST:destination.address}/%{INT:destination.port}" - ] - } - } - mutate { - id => "filter_mutate_15" - add_field => { - "event.category" => "nat_translation" - } - } - } - else if [event.code] == "110003" { - grok { - id => "filter_grok_7" - match => { - "message" => [ - "%{DATA:cisco.event_error} for %{WORD:network.transport} from %{DATA:cisco.source_interface}:%{IP:source.address}\/%{INT:source.port} to %{DATA:cisco.destination_interface}:%{IP:destination.address}\/%{INT:destination.port}" - ] - } - } - mutate { - id => "filter_mutate_16" - add_field => { - "event.category" => "error" - } - } - } - else if [event.code] == "113019" { - grok { - id => "filter_grok_8" - match => { - "message" => [ - "Group = %{DATA:cisco.group}, Username = %{DATA:user.name}, IP = %{IP:cisco.client_vpn_ip}, %{DATA:cisco.client_vpn_action}\. Session Type: %{DATA:cisco.session_type}, Duration: %{DATA:cisco.duration}, Bytes xmt: %{INT:cisco.vpn_transmit_byte_summary}, Bytes rcv: %{INT:cisco.vpn_receive_byte_summary}, Reason: %{DATA:cisco.client_vpn_outcome}$" - ] - } - } - } - else if [event.code] == "304001" { - dissect { - id => "filter_dissect_19" - mapping => { - "message" => "%{source.address} %{}ccessed URL %{destination.address}:%{url.original}" - } - } - mutate { - id => "filter_mutate_17" - add_field => { - "event.outcome" => "allow" - } - } - } - else if [event.code] == "304002" { - dissect { - id => "filter_dissect_20" - mapping => { - "message" => "Access %{event.outcome} URL %{url.original} SRC %{source.address} %{}EST %{destination.address} on interface %{cisco.source_interface}" - } - } - } - else if [event.code] == "305011" { - grok { - id => "filter_grok_9" - match => { - "message" => [ - "Built dynamic %{WORD:network.transport} translation from %{DATA:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} to %{DATA:cisco.destination_interface}:%{IP:destination.address}/%{INT:destination.port}" - ] - } - } - mutate { - id => "filter_mutate_18" - add_field => { - "event.category" => "nat_translation" - } - } - } - else if [event.code] == "305012" { - grok { - id => "filter_grok_10" - match => { - "message" => [ - "Teardown dynamic %{WORD:network.transport} translation from %{DATA:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} to %{DATA:cisco.destination_interface}:%{IP:destination.address}/%{INT:destination.port}" - ] - } - } - mutate { - id => "filter_mutate_19" - add_field => { - "event.category" => "nat_translation" - } - } - } - else if [event.code] == "313001" { - dissect { - id => "filter_dissect_21" - mapping => { - "message" => "%{event.outcome} %{network.transport} type=%{cisco.icmp_type}, code=%{cisco.icmp_code} from %{source.address} on interface %{cisco.source_interface}" - } - } - } - else if [event.code] == "313004" { - dissect { - id => "filter_dissect_22" - mapping => { - "message" => "%{event.outcome} %{network.transport} type=%{cisco.icmp_type}, from%{}addr %{source.address} on interface %{cisco.source_interface} to %{destination.address}: no matching session" - } - } - } - else if [event.code] == "313005" { - dissect { - id => "filter_dissect_23" - mapping => { - "message" => "No matching connection for %{network.transport} error message: %{} on %{cisco.source_interface} interface.%{}riginal IP payload: %{}" - } - } - } - else if [event.code] == "313008" { - dissect { - id => "filter_dissect_24" - mapping => { - "message" => "%{event.outcome} %{network.transport} type=%{cisco.icmp_type} , code=%{cisco.icmp_code} from %{source.address} on interface %{cisco.source_interface}" - } - } - } - else if [event.code] == "313009" { - dissect { - id => "filter_dissect_25" - mapping => { - "message" => "%{event.outcome} invalid %{network.transport} code %{cisco.icmp_code} , for %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}" - } - } - } - else if [event.code] == "322001" { - dissect { - id => "filter_dissect_26" - mapping => { - "message" => "%{event.outcome} MAC address %{source.mac}, possible spoof attempt on interface %{cisco.source_interface}" - } - } - } - else if [event.code] == "338001" { - dissect { - id => "filter_dissect_27" - mapping => { - "message" => "Dynamic filter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{source.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" - } - } - } - else if [event.code] == "338002" { - dissect { - id => "filter_dissect_28" - mapping => { - "message" => "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{destination.domain}" - } - } - mutate { - id => "filter_mutate_20" - add_field => { - "server.domain" => "[destination.domain]" - } - } - } - else if [event.code] == "338003" { - dissect { - id => "filter_dissect_29" - mapping => { - "message" => "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" - } - } - } - else if [event.code] == "338004" { - dissect { - id => "filter_dissect_30" - mapping => { - "message" => "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" - } - } - } - else if [event.code] == "338005" { - dissect { - id => "filter_dissect_31" - mapping => { - "message" => "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{source.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" - } - } - mutate { - id => "filter_mutate_21" - add_field => { - "server.domain" => "[source.domain]" - } - } - } - else if [event.code] == "338006" { - dissect { - id => "filter_dissect_32" - mapping => { - "message" => "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{destination.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" - } - } - mutate { - id => "filter_mutate_22" - add_field => { - "server.domain" => "server.domain" - } - } - } - else if [event.code] == "338007" { - dissect { - id => "filter_dissect_33" - mapping => { - "message" => "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" - } - } - } - else if [event.code] == "338008" { - dissect { - id => "filter_dissect_34" - mapping => { - "message" => "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" - } - } - } - else if [event.code] == "338101" { - dissect { - id => "filter_dissect_35" - mapping => { - "message" => "Dynamic %{}ilter %{event.outcome} white%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{source.domain}" - } - } - mutate { - id => "filter_mutate_23" - add_field => { - "server.domain" => "server.domain" - } - } - } - else if [event.code] == "338102" { - dissect { - id => "filter_dissect_36" - mapping => { - "message" => "Dynamic %{}ilter %{event.outcome} white%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{destination.domain}" - } - } - mutate { - id => "filter_mutate_24" - add_field => { - "server.domain" => "server.domain" - } - } - } - else if [event.code] == "338103" { - dissect { - id => "filter_dissect_37" - mapping => { - "message" => "Dynamic %{}ilter %{event.outcome} white%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{}" - } - } - } - else if [event.code] == "338104" { - dissect { - id => "filter_dissect_38" - mapping => { - "message" => "Dynamic %{}ilter %{event.outcome} white%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{}" - } - } - } - else if [event.code] == "338201" { - dissect { - id => "filter_dissect_39" - mapping => { - "message" => "Dynamic %{}ilter %{event.outcome} grey%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{source.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" - } - } - mutate { - id => "filter_mutate_25" - add_field => { - "server.domain" => "server.domain" - } - } - } - else if [event.code] == "338202" { - dissect { - id => "filter_dissect_40" - mapping => { - "message" => "Dynamic %{}ilter %{event.outcome} grey%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{destination.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" - } - } - mutate { - id => "filter_mutate_26" - add_field => { - "server.domain" => "server.domain" - } - } - } - else if [event.code] == "338203" { - dissect { - id => "filter_dissect_41" - mapping => { - "message" => "Dynamic %{}ilter %{event.outcome} grey%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{source.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" - } - } - mutate { - id => "filter_mutate_27" - add_field => { - "server.domain" => "server.domain" - } - } - } - else if [event.code] == "338204" { - dissect { - id => "filter_dissect_42" - mapping => { - "message" => "Dynamic %{}ilter %{event.outcome} grey%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{destination.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" - } - } - mutate { - id => "filter_mutate_28" - add_field => { - "server.domain" => "server.domain" - } - } - } - else if [event.code] == "338301" { - dissect { - id => "filter_dissect_43" - mapping => { - "message" => "Intercepted DNS reply for domain %{source.domain} from %{cisco.source_interface}:%{source.address}/%{source.port} to %{cisco.destination_interface}:%{destination.address}/%{destination.port}, matched %{cisco.list_id}" - } - } - mutate { - id => "filter_mutate_29" - add_field => { - "client.address" => "client.address" - } - } - mutate { - id => "filter_mutate_30" - add_field => { - "client.port" => "client.port" - } - } - mutate { - id => "filter_mutate_31" - add_field => { - "server.address" => "server.address" - } - } - mutate { - id => "filter_mutate_32" - add_field => { - "server.port" => "server.port" - } - } - } - else if [event.code] in ["302014", "302016", "302018", "302021", "302036", "302304", "302306", "302020"] { - grok { - id => "filter_grok_11" - pattern_definitions => { - "NOTCOLON" => "[^:]*" - "ECSSOURCEIPORHOST" => "(?:%{IP:source.address}|%{HOSTNAME:source.domain})" - "ECSDESTIPORHOST" => "(?:%{IP:destination.address}|%{HOSTNAME:destination.domain})" - "MAPPEDSRC" => "(?:%{DATA:cisco.mapped_source_ip}|%{HOSTNAME})" - } - match => { - "message" => [ - "Teardown %{NOTSPACE:network.transport} (?:state-bypass )?connection %{NOTSPACE:cisco.connection_id} (?:for|from) %{NOTCOLON}:%{DATA:source.address}/%{NUMBER:source.port:int}?(\(%{DATA:cisco.source_username}\)|) ?to %{NOTCOLON:cisco.destination_interface}:%{DATA:destination.address}/%{NUMBER:destination.port:int}?(\(%{DATA:cisco.source_username}\)|) ?(?:duration %{TIME:cisco.duration_hms} bytes %{NUMBER:network.bytes:int})%{GREEDYDATA}", - "Teardown %{NOTSPACE:network.transport} (?:state-bypass )?connection %{NOTSPACE:cisco.connection_id} (?:for|from) %{NOTCOLON}:%{DATA:source.address}/%{NUMBER:source.port:int} (?:%{NOTSPACE:cisco.source_username} )?to %{NOTCOLON:cisco.destination_interface}:%{DATA:destination.address}/%{NUMBER:destination.port:int} (?:%{NOTSPACE:cisco.destination_username} )?(?:duration %{TIME:cisco.duration_hms} bytes %{NUMBER:network.bytes:int})%{GREEDYDATA}", - "Teardown %{NOTSPACE:network.transport} connection for faddr (?:%{NOTCOLON:cisco.source_interface}:)?%{ECSDESTIPORHOST}/%{NUMBER} (?:%{NOTSPACE:cisco.destination_username} )?gaddr (?:%{NOTCOLON}:)?%{MAPPEDSRC}/%{NUMBER} laddr (?:%{NOTCOLON:cisco.source_interface}:)?%{ECSSOURCEIPORHOST}/%{NUMBER}(?: %{NOTSPACE:cisco.source_username})?%{GREEDYDATA}", - "Built %{WORD:network.direction} %{NOTSPACE:network.transport} connection for faddr (?:%{NOTCOLON:cisco.source_interface}:)?%{ECSDESTIPORHOST}/%{NUMBER} (?:%{NOTSPACE:cisco.destination_username} )?gaddr (?:%{NOTCOLON}:)?%{MAPPEDSRC}/%{NUMBER} laddr (?:%{NOTCOLON:cisco.source_interface}:)?%{ECSSOURCEIPORHOST}/%{NUMBER}(?: %{NOTSPACE:cisco.source_username})?%{GREEDYDATA}" - ] - } - } - mutate { - id => "filter_mutate_33" - add_field => { - "event.category" => "nat_translation" - } - } - } - else if [event.code] == "419002" { - grok { - id => "filter_grok_12" - match => { - "message" => [ - "%{DATA:cisco.event_error} from %{WORD:cisco.source_interface}:%{IPORHOST:source.address}\/%{INT:source.port} to %{WORD:cisco.destination_interface}:%{IPORHOST:destination.address}\/%{INT:destination.port}" - ] - } - } - mutate { - id => "filter_mutate_34" - add_field => { - "event.category" => "error" - } - } - } - else if [event.code] in ["733100", "752015", "752012"] { - grok { - id => "filter_grok_13" - match => { - "message" => [ - "%{GREEDYDATA:cisco.event_error}" - ] - } - } - mutate { - id => "filter_mutate_35" - add_field => { - "event.category" => "error" - } - } - } - else if [event.code] == "716002" { - grok { - id => "filter_grok_14" - match => { - "message" => [ - "Group \<%{DATA:cisco.group} User \<%{DATA:user.name}\> IP \<%{IP:cisco.client_vpn_ip}\> WebVPN session %{WORD:cisco.client_vpn_session_outcome}\: %{DATA:cisco.web_vpn_action}\." - ] - } - } - } - else if [event.code] in ['722022', '722033', '722055', '722051', '113039', '722023', '722037'] { - grok { - id => "filter_grok_15" - match => { - "message" => [ - "Group \<%{DATA:cisco.group} User \<%{DATA:user.name}\> IP \<%{IP:cisco.client_vpn_ip}\> %{GREEDYDATA:cisco.message}" - ] - } - } - } - else { - grok { - id => "filter_grok_16" - match => { - "message" => [ - "forced_failure" - ] - } - } - } - if [event.category] == "nat_translation" { - drop { - id => "filter_drop_1" - } - } - if [source.address] { - grok { - id => "filter_grok_17" - match => { - "source.address" => [ - "(?:%{IP:source.ip}|%{GREEDYDATA:source.domain})" - ] - } - } - } - if [destination.address] { - grok { - id => "filter_grok_18" - match => { - "destination.address" => [ - "(?:%{IP:destination.ip}|%{GREEDYDATA:destination.domain})" - ] - } - } - } - if [client.address] { - grok { - id => "filter_grok_19" - match => { - "client.address" => [ - "(?:%{IP:client.ip}|%{GREEDYDATA:client.domain})" - ] - } - } - } - if [server.address] { - grok { - id => "filter_grok_20" - match => { - "server.address" => [ - "(?:%{IP:server.ip}|%{GREEDYDATA:server.domain})" - ] - } - } - } - mutate { - id => "filter_mutate_36" - lowercase => ["network.transport", "network.protocol", "network.direction", "event.outcome"] - } - if [event.outcome] == "est-allowed" { - mutate { - id => "filter_mutate_37" - update => { - "event.outcome" => "allow" - } - } - } - else if [event.outcome] == "permitted" { - mutate { - id => "filter_mutate_38" - update => { - "event.outcome" => "allow" - } - } - } - else if [event.outcome] == "denied" { - mutate { - id => "filter_mutate_39" - update => { - "event.outcome" => "deny" - } - } - } - else if [event.outcome] == "dropped" { - mutate { - id => "filter_mutate_40" - update => { - "event.outcome" => "deny" - } - } - } - if [network.transport] == "icmpv6" { - mutate { - id => "filter_mutate_41" - update => { - "network.transport" => "ipv6-icmp" - } - } - } - translate { - id => "filter_translate_1" - field => "network.transport" - destination => "network.iana_number" - dictionary => { - "icmp" => "1" - "igmp" => "2" - "ipv4" => "4" - "tcp" => "6" - "egp" => "8" - "igp" => "9" - "pup" => "12" - "udp" => "17" - "rdp" => "27" - "irtp" => "28" - "dccp" => "33" - "idpr" => "35" - "ipv6" => "41" - "ipv6-route" => "43" - "ipv6-frag" => "44" - "rsvp" => "46" - "gre" => "47" - "esp" => "50" - "ipv6-icmp" => "58" - "ipv6-nonxt" => "59" - "ipv6-opts" => "60" - } - } - mutate { - id => "filter_mutate_42" - remove_field => ["ciscotag", "timestamp"] - } - mutate { - id => "filter_mutate_43" - add_field => { - "event.module" => "cisco" - "event.dataset" => "asa" - } - } - translate { - id => "filter_translate_2" - field => "[event.severity]" - destination => "[log.level]" - dictionary => { - "0" => "emergency" - "1" => "alert" - "2" => "critical" - "3" => "error" - "4" => "warning" - "5" => "notification" - "6" => "informational" - "7" => "debug" - } - } -} -output { - elasticsearch { - id => "output_elasticsearch_1" - api_key => "${es_api_key}" - hosts => "https://homedc-90e54c.es.us-east-2.aws.elastic-cloud.com:443" - index => "asa-1.2" - pipeline => "asa" - } -} diff --git a/src/logstashui/Common/tests/conversion_data/pipelines/test-asa.conf b/src/logstashui/Common/tests/conversion_data/pipelines/test-asa.conf deleted file mode 100644 index dbd9def..0000000 --- a/src/logstashui/Common/tests/conversion_data/pipelines/test-asa.conf +++ /dev/null @@ -1,897 +0,0 @@ -input { - udp { - id => "input_udp_1" - port => "5119" - } - cloudwatch { - } -} -filter { - mutate { - id => "filter_mutate_1" - rename => { - "message" => "log.original" - "host" => "observer.ip" - } - copy => { - "host" => "sysloghost" - } - } - grok { - id => "filter_grok_1" - match => { - "log.original" => [ - "%{CISCO_TAGGED_SYSLOG} %{GREEDYDATA:message}", - "^<%{POSINT:syslog_pri}>%{DATA}: %%{DATA:ciscotag}: %{GREEDYDATA:message}", - "^<%{POSINT:syslog_pri}>%%{DATA:ciscotag}: %{GREEDYDATA:message}" - ] - } - } - grok { - id => "filter_grok_2" - match => { - "ciscotag" => [ - "%{WORD}-%{INT:event.severity}-%{INT:event.code}", - "%{WORD}-%{WORD}-%{INT:event.severity}-%{INT:event.code}" - ] - } - } - mutate { - id => "filter_mutate_2" - add_field => { - "event.action" => "firewall-rule" - } - } - if [event.code] == "105012" { - grok { - id => "filter_grok_3" - match => { - "message" => [ - "Teardown dynamic %{WORD:network.transport} translation from %{DATA:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} to %{DATA:cisco.destination_interface}:%{IP:destination.address}/%{INT:destination.port} duration %{DATA:cisco.duration_hms}$" - ] - } - } - } - else if [event.code] == "106001" { - dissect { - id => "filter_dissect_1" - mapping => { - "message" => "%{network.direction} %{network.transport} connection %{event.outcome} from %{source.address}/%{source.port} to %{destination.address}/%{destination.port} flags %{} on interface %{source_interface}" - } - } - mutate { - id => "filter_mutate_3" - add_field => { - "event.category" => "network_traffic" - } - } - } - else if [event.code] == "106002" { - dissect { - id => "filter_dissect_2" - mapping => { - "message" => "%{network.transport} Connection %{event.outcome} by %{network.direction} list %{cisco.list_id} src %{source.address} dest %{destination.address}" - } - } - mutate { - id => "filter_mutate_4" - add_field => { - "event.category" => "network_traffic" - } - } - } - else if [event.code] == "106006" { - dissect { - id => "filter_dissect_3" - mapping => { - "message" => "%{event.outcome} %{network.direction} %{network.transport} from %{source.address}/%{source.port} to %{destination.address}/%{destination.port} on interface %{cisco.source_interface}" - } - } - mutate { - id => "filter_mutate_5" - add_field => { - "event.category" => "network_traffic" - } - } - } - else if [event.code] == "106007" { - dissect { - id => "filter_dissect_4" - mapping => { - "message" => "%{event.outcome} %{network.direction} %{network.transport} from %{source.address}/%{source.port} to %{destination.address}/%{destination.port} due to %{network.protocol} %{}" - } - } - mutate { - id => "filter_mutate_6" - add_field => { - "event.category" => "network_traffic" - } - } - } - else if [event.code] == "106010" { - dissect { - id => "filter_dissect_5" - mapping => { - "message" => "%{event.outcome} %{network.direction} %{network.transport} src %{cisco.source_interface}:%{source.address}/%{source.port} %{} dst %{cisco.destination_interface}:%{destination.address}/%{destination.port} %{}" - } - } - mutate { - id => "filter_mutate_7" - add_field => { - "event.category" => "network_traffic" - } - } - } - else if [event.code] == "106013" { - dissect { - id => "filter_dissect_6" - mapping => { - "message" => "Dropping echo request from %{source.address} to PAT address %{destination.address}" - } - } - mutate { - id => "filter_mutate_8" - add_field => { - "network.transport" => "icmp" - "network.direction" => "inbound" - } - } - } - else if [event.code] == "106014" { - dissect { - id => "filter_dissect_7" - mapping => { - "message" => "%{event.outcome} %{network.direction} %{network.transport} src %{cisco.source_interface}:%{source.address} %{}dst %{cisco.destination_interface}:%{destination.address} %{}" - } - } - mutate { - id => "filter_mutate_9" - add_field => { - "event.category" => "network_traffic" - } - } - } - else if [event.code] == "106015" { - dissect { - id => "filter_dissect_8" - mapping => { - "message" => "%{event.outcome} %{network.transport} (no connection) from %{source.address}/%{source.port} to %{destination.address}/%{destination.port} flags %{} on interface %{cisco.source_interface}" - } - } - mutate { - id => "filter_mutate_10" - add_field => { - "event.category" => "nat_translation" - } - } - } - else if [event.code] == "106016" { - dissect { - id => "filter_dissect_9" - mapping => { - "message" => "%{event.outcome} IP spoof from (%{source.address}) to %{destination.address} on interface %{cisco.source_interface}" - } - } - } - else if [event.code] == "106017" { - dissect { - id => "filter_dissect_10" - mapping => { - "message" => "%{event.outcome} IP due to Land Attack from %{source.address} to %{destination.address}" - } - } - } - else if [event.code] == "106018" { - dissect { - id => "filter_dissect_11" - mapping => { - "message" => "%{network.transport} packet type %{cisco.icmp_type} %{event.outcome} by %{network.direction} list %{cisco.list_id} src %{source.address} dest %{destination.address}" - } - } - } - else if [event.code] == "106020" { - dissect { - id => "filter_dissect_12" - mapping => { - "message" => "%{event.outcome} IP teardrop fragment (size = %{}, offset = %{}) from %{source.address} to %{destination.address}" - } - } - } - else if [event.code] == "106021" { - dissect { - id => "filter_dissect_13" - mapping => { - "message" => "%{event.outcome} %{network.transport} reverse path check from %{source.address} to %{destination.address} on interface %{cisco.source_interface}" - } - } - } - else if [event.code] == "106022" { - dissect { - id => "filter_dissect_14" - mapping => { - "message" => "%{event.outcome} %{network.transport} connection spoof from %{source.address} to %{destination.address} on interface %{cisco.source_interface}" - } - } - } - else if [event.code] == "106023" { - grok { - id => "filter_grok_4" - match => { - "message" => [ - '%{WORD:event.outcome} %{WORD:network.transport} src %{WORD:cisco.source_interface}:%{IPORHOST:source.address}?(/%{INT:source.port}) dst %{WORD:cisco.destination_interface}:%{IPORHOST:destination.address}?(/%{INT:destination.port}) by access-group "%{DATA:cisco.list_id}"', - '%{WORD:event.outcome} %{WORD:network.transport} src %{WORD:cisco.source_interface}:%{IPORHOST:source.address} dst %{WORD:destination.direction}:%{IPORHOST:destination.address} \(%{DATA}\) by access-group "%{DATA:cisco.list_id}"', - '%{WORD:event.outcome} %{WORD:network.transport} src %{WORD:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} dst %{WORD:cisco.destination.interface}:%{IPORHOST:destination.address}/%{INT:destination.port} by access-group "%{DATA:cisco.list_id}"' - ] - } - } - mutate { - id => "filter_mutate_11" - add_field => { - "event.category" => "network_traffic" - } - } - } - else if [event.code] == "106027" { - dissect { - id => "filter_dissect_15" - mapping => { - "message" => '%{} %{event.outcome} src %{source.address} dst %{destination.address} by access-group "%{cisco.list_id}"%{}' - } - } - } - else if [event.code] == "106100" { - dissect { - id => "filter_dissect_16" - mapping => { - "message" => "access-list %{cisco.list_id} %{event.outcome} %{network.transport} %{cisco.source_interface}/%{source.address}(%{source.port}) -> %{cisco.destination_interface}/%{destination.address}(%{destination.port}) %{}" - } - } - } - else if [event.code] == "106102" { - dissect { - id => "filter_dissect_17" - mapping => { - "message" => "access-list %{cisco.list_id} %{event.outcome} %{network.transport} for user %{cisco.username} %{cisco.source_interface}/%{source.address} %{source.port} %{cisco.destination_interface}/%{destination.address} %{destination.port} %{}" - } - } - } - else if [event.code] == "106103" { - dissect { - id => "filter_dissect_18" - mapping => { - "message" => "access-list %{cisco.list_id} %{event.outcome} %{network.transport} for user %{cisco.username} %{cisco.source_interface}/%{source.address} %{source.port} %{cisco.destination_interface}/%{destination.address} %{destination.port} %{}" - } - } - } - else if [event.code] == "113004" { - grok { - id => "filter_grok_5" - match => { - "message" => [ - "AAA user accounting %{WORD:cisco.auth_outcome} : server =%{SPACE}%{IP:source.address} : user =%{SPACE}%{DATA:source.user.name}$" - ] - } - } - mutate { - id => "filter_mutate_12" - add_field => { - "event.category" => "authentication" - } - } - if [cisco.auth_outcome] == "Successful" { - mutate { - id => "filter_mutate_13" - add_field => { - "event.action" => "authentication_success" - } - } - } - else { - mutate { - id => "filter_mutate_14" - add_field => { - "event.action" => "authentication_failure" - } - } - } - } - else if [event.code] == "302015" or [event.code] == "302013" { - grok { - id => "filter_grok_6" - match => { - "message" => [ - "Built %{WORD:network.direction} %{WORD:network.transport} connection %{INT} for %{DATA:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} \(%{IP}|\) to %{DATA:cisco.destination_interface}:%{IPORHOST:destination.address}/%{INT:destination.port} \(%{DATA}\)", - "Built %{WORD:network.direction} %{WORD:network.transport} connection %{INT} for %{DATA:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} \(%{DATA}\)?(\(%{DATA:cisco.source_username}\)) to %{DATA:cisco.destination_interface}:%{IPORHOST:destination.address}/%{INT:destination.port} \(%{DATA}\) ?(\(%{DATA:cisco.username}\))", - "Built %{WORD:network.direction} %{WORD:network.transport} connection %{INT:cisco.connection_id} for %{WORD:cisco.source_interface}:%{IPORHOST:source.address}\/%{INT:source.port} \(%{DATA}\) to %{WORD:cisco.destination_interface}:%{IPORHOST:destination.address}/%{INT:destination.port}" - ] - } - } - mutate { - id => "filter_mutate_15" - add_field => { - "event.category" => "nat_translation" - } - } - } - else if [event.code] == "110003" { - grok { - id => "filter_grok_7" - match => { - "message" => [ - "%{DATA:cisco.event_error} for %{WORD:network.transport} from %{DATA:cisco.source_interface}:%{IP:source.address}\/%{INT:source.port} to %{DATA:cisco.destination_interface}:%{IP:destination.address}\/%{INT:destination.port}" - ] - } - } - mutate { - id => "filter_mutate_16" - add_field => { - "event.category" => "error" - } - } - } - else if [event.code] == "113019" { - grok { - id => "filter_grok_8" - match => { - "message" => [ - "Group = %{DATA:cisco.group}, Username = %{DATA:user.name}, IP = %{IP:cisco.client_vpn_ip}, %{DATA:cisco.client_vpn_action}\. Session Type: %{DATA:cisco.session_type}, Duration: %{DATA:cisco.duration}, Bytes xmt: %{INT:cisco.vpn_transmit_byte_summary}, Bytes rcv: %{INT:cisco.vpn_receive_byte_summary}, Reason: %{DATA:cisco.client_vpn_outcome}$" - ] - } - } - } - else if [event.code] == "304001" { - dissect { - id => "filter_dissect_19" - mapping => { - "message" => "%{source.address} %{}ccessed URL %{destination.address}:%{url.original}" - } - } - mutate { - id => "filter_mutate_17" - add_field => { - "event.outcome" => "allow" - } - } - } - else if [event.code] == "304002" { - dissect { - id => "filter_dissect_20" - mapping => { - "message" => "Access %{event.outcome} URL %{url.original} SRC %{source.address} %{}EST %{destination.address} on interface %{cisco.source_interface}" - } - } - } - else if [event.code] == "305011" { - grok { - id => "filter_grok_9" - match => { - "message" => [ - "Built dynamic %{WORD:network.transport} translation from %{DATA:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} to %{DATA:cisco.destination_interface}:%{IP:destination.address}/%{INT:destination.port}" - ] - } - } - mutate { - id => "filter_mutate_18" - add_field => { - "event.category" => "nat_translation" - } - } - } - else if [event.code] == "305012" { - grok { - id => "filter_grok_10" - match => { - "message" => [ - "Teardown dynamic %{WORD:network.transport} translation from %{DATA:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} to %{DATA:cisco.destination_interface}:%{IP:destination.address}/%{INT:destination.port}" - ] - } - } - mutate { - id => "filter_mutate_19" - add_field => { - "event.category" => "nat_translation" - } - } - } - else if [event.code] == "313001" { - dissect { - id => "filter_dissect_21" - mapping => { - "message" => "%{event.outcome} %{network.transport} type=%{cisco.icmp_type}, code=%{cisco.icmp_code} from %{source.address} on interface %{cisco.source_interface}" - } - } - } - else if [event.code] == "313004" { - dissect { - id => "filter_dissect_22" - mapping => { - "message" => "%{event.outcome} %{network.transport} type=%{cisco.icmp_type}, from%{}addr %{source.address} on interface %{cisco.source_interface} to %{destination.address}: no matching session" - } - } - } - else if [event.code] == "313005" { - dissect { - id => "filter_dissect_23" - mapping => { - "message" => "No matching connection for %{network.transport} error message: %{} on %{cisco.source_interface} interface.%{}riginal IP payload: %{}" - } - } - } - else if [event.code] == "313008" { - dissect { - id => "filter_dissect_24" - mapping => { - "message" => "%{event.outcome} %{network.transport} type=%{cisco.icmp_type} , code=%{cisco.icmp_code} from %{source.address} on interface %{cisco.source_interface}" - } - } - } - else if [event.code] == "313009" { - dissect { - id => "filter_dissect_25" - mapping => { - "message" => "%{event.outcome} invalid %{network.transport} code %{cisco.icmp_code} , for %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}" - } - } - } - else if [event.code] == "322001" { - dissect { - id => "filter_dissect_26" - mapping => { - "message" => "%{event.outcome} MAC address %{source.mac}, possible spoof attempt on interface %{cisco.source_interface}" - } - } - } - else if [event.code] == "338001" { - dissect { - id => "filter_dissect_27" - mapping => { - "message" => "Dynamic filter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{source.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" - } - } - } - else if [event.code] == "338002" { - dissect { - id => "filter_dissect_28" - mapping => { - "message" => "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{destination.domain}" - } - } - mutate { - id => "filter_mutate_20" - add_field => { - "server.domain" => "[destination.domain]" - } - } - } - else if [event.code] == "338003" { - dissect { - id => "filter_dissect_29" - mapping => { - "message" => "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" - } - } - } - else if [event.code] == "338004" { - dissect { - id => "filter_dissect_30" - mapping => { - "message" => "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" - } - } - } - else if [event.code] == "338005" { - dissect { - id => "filter_dissect_31" - mapping => { - "message" => "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{source.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" - } - } - mutate { - id => "filter_mutate_21" - add_field => { - "server.domain" => "[source.domain]" - } - } - } - else if [event.code] == "338006" { - dissect { - id => "filter_dissect_32" - mapping => { - "message" => "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{destination.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" - } - } - mutate { - id => "filter_mutate_22" - add_field => { - "server.domain" => "server.domain" - } - } - } - else if [event.code] == "338007" { - dissect { - id => "filter_dissect_33" - mapping => { - "message" => "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" - } - } - } - else if [event.code] == "338008" { - dissect { - id => "filter_dissect_34" - mapping => { - "message" => "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" - } - } - } - else if [event.code] == "338101" { - dissect { - id => "filter_dissect_35" - mapping => { - "message" => "Dynamic %{}ilter %{event.outcome} white%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{source.domain}" - } - } - mutate { - id => "filter_mutate_23" - add_field => { - "server.domain" => "server.domain" - } - } - } - else if [event.code] == "338102" { - dissect { - id => "filter_dissect_36" - mapping => { - "message" => "Dynamic %{}ilter %{event.outcome} white%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{destination.domain}" - } - } - mutate { - id => "filter_mutate_24" - add_field => { - "server.domain" => "server.domain" - } - } - } - else if [event.code] == "338103" { - dissect { - id => "filter_dissect_37" - mapping => { - "message" => "Dynamic %{}ilter %{event.outcome} white%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{}" - } - } - } - else if [event.code] == "338104" { - dissect { - id => "filter_dissect_38" - mapping => { - "message" => "Dynamic %{}ilter %{event.outcome} white%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{}" - } - } - } - else if [event.code] == "338201" { - dissect { - id => "filter_dissect_39" - mapping => { - "message" => "Dynamic %{}ilter %{event.outcome} grey%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{source.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" - } - } - mutate { - id => "filter_mutate_25" - add_field => { - "server.domain" => "server.domain" - } - } - } - else if [event.code] == "338202" { - dissect { - id => "filter_dissect_40" - mapping => { - "message" => "Dynamic %{}ilter %{event.outcome} grey%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{destination.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" - } - } - mutate { - id => "filter_mutate_26" - add_field => { - "server.domain" => "server.domain" - } - } - } - else if [event.code] == "338203" { - dissect { - id => "filter_dissect_41" - mapping => { - "message" => "Dynamic %{}ilter %{event.outcome} grey%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{source.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" - } - } - mutate { - id => "filter_mutate_27" - add_field => { - "server.domain" => "server.domain" - } - } - } - else if [event.code] == "338204" { - dissect { - id => "filter_dissect_42" - mapping => { - "message" => "Dynamic %{}ilter %{event.outcome} grey%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{destination.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" - } - } - mutate { - id => "filter_mutate_28" - add_field => { - "server.domain" => "server.domain" - } - } - } - else if [event.code] == "338301" { - dissect { - id => "filter_dissect_43" - mapping => { - "message" => "Intercepted DNS reply for domain %{source.domain} from %{cisco.source_interface}:%{source.address}/%{source.port} to %{cisco.destination_interface}:%{destination.address}/%{destination.port}, matched %{cisco.list_id}" - } - } - mutate { - id => "filter_mutate_29" - add_field => { - "client.address" => "client.address" - } - } - mutate { - id => "filter_mutate_30" - add_field => { - "client.port" => "client.port" - } - } - mutate { - id => "filter_mutate_31" - add_field => { - "server.address" => "server.address" - } - } - mutate { - id => "filter_mutate_32" - add_field => { - "server.port" => "server.port" - } - } - } - else if [event.code] in ["302014", "302016", "302018", "302021", "302036", "302304", "302306", "302020"] { - grok { - id => "filter_grok_11" - pattern_definitions => { - "NOTCOLON" => "[^:]*" - "ECSSOURCEIPORHOST" => "(?:%{IP:source.address}|%{HOSTNAME:source.domain})" - "ECSDESTIPORHOST" => "(?:%{IP:destination.address}|%{HOSTNAME:destination.domain})" - "MAPPEDSRC" => "(?:%{DATA:cisco.mapped_source_ip}|%{HOSTNAME})" - } - match => { - "message" => [ - "Teardown %{NOTSPACE:network.transport} (?:state-bypass )?connection %{NOTSPACE:cisco.connection_id} (?:for|from) %{NOTCOLON}:%{DATA:source.address}/%{NUMBER:source.port:int}?(\(%{DATA:cisco.source_username}\)|) ?to %{NOTCOLON:cisco.destination_interface}:%{DATA:destination.address}/%{NUMBER:destination.port:int}?(\(%{DATA:cisco.source_username}\)|) ?(?:duration %{TIME:cisco.duration_hms} bytes %{NUMBER:network.bytes:int})%{GREEDYDATA}", - "Teardown %{NOTSPACE:network.transport} (?:state-bypass )?connection %{NOTSPACE:cisco.connection_id} (?:for|from) %{NOTCOLON}:%{DATA:source.address}/%{NUMBER:source.port:int} (?:%{NOTSPACE:cisco.source_username} )?to %{NOTCOLON:cisco.destination_interface}:%{DATA:destination.address}/%{NUMBER:destination.port:int} (?:%{NOTSPACE:cisco.destination_username} )?(?:duration %{TIME:cisco.duration_hms} bytes %{NUMBER:network.bytes:int})%{GREEDYDATA}", - "Teardown %{NOTSPACE:network.transport} connection for faddr (?:%{NOTCOLON:cisco.source_interface}:)?%{ECSDESTIPORHOST}/%{NUMBER} (?:%{NOTSPACE:cisco.destination_username} )?gaddr (?:%{NOTCOLON}:)?%{MAPPEDSRC}/%{NUMBER} laddr (?:%{NOTCOLON:cisco.source_interface}:)?%{ECSSOURCEIPORHOST}/%{NUMBER}(?: %{NOTSPACE:cisco.source_username})?%{GREEDYDATA}", - "Built %{WORD:network.direction} %{NOTSPACE:network.transport} connection for faddr (?:%{NOTCOLON:cisco.source_interface}:)?%{ECSDESTIPORHOST}/%{NUMBER} (?:%{NOTSPACE:cisco.destination_username} )?gaddr (?:%{NOTCOLON}:)?%{MAPPEDSRC}/%{NUMBER} laddr (?:%{NOTCOLON:cisco.source_interface}:)?%{ECSSOURCEIPORHOST}/%{NUMBER}(?: %{NOTSPACE:cisco.source_username})?%{GREEDYDATA}" - ] - } - } - mutate { - id => "filter_mutate_33" - add_field => { - "event.category" => "nat_translation" - } - } - } - else if [event.code] == "419002" { - grok { - id => "filter_grok_12" - match => { - "message" => [ - "%{DATA:cisco.event_error} from %{WORD:cisco.source_interface}:%{IPORHOST:source.address}\/%{INT:source.port} to %{WORD:cisco.destination_interface}:%{IPORHOST:destination.address}\/%{INT:destination.port}" - ] - } - } - mutate { - id => "filter_mutate_34" - add_field => { - "event.category" => "error" - } - } - } - else if [event.code] in ["733100", "752015", "752012"] { - grok { - id => "filter_grok_13" - match => { - "message" => [ - "%{GREEDYDATA:cisco.event_error}" - ] - } - } - mutate { - id => "filter_mutate_35" - add_field => { - "event.category" => "error" - } - } - } - else if [event.code] == "716002" { - grok { - id => "filter_grok_14" - match => { - "message" => [ - "Group \<%{DATA:cisco.group} User \<%{DATA:user.name}\> IP \<%{IP:cisco.client_vpn_ip}\> WebVPN session %{WORD:cisco.client_vpn_session_outcome}\: %{DATA:cisco.web_vpn_action}\." - ] - } - } - } - else if [event.code] in ['722022', '722033', '722055', '722051', '113039', '722023', '722037'] { - grok { - id => "filter_grok_15" - match => { - "message" => [ - "Group \<%{DATA:cisco.group} User \<%{DATA:user.name}\> IP \<%{IP:cisco.client_vpn_ip}\> %{GREEDYDATA:cisco.message}" - ] - } - } - } - else { - grok { - id => "filter_grok_16" - match => { - "message" => [ - "forced_failure" - ] - } - } - } - if [event.category] == "nat_translation" { - drop { - id => "filter_drop_1" - } - } - if [source.address] { - grok { - id => "filter_grok_17" - match => { - "source.address" => [ - "(?:%{IP:source.ip}|%{GREEDYDATA:source.domain})" - ] - } - } - } - if [destination.address] { - grok { - id => "filter_grok_18" - match => { - "destination.address" => [ - "(?:%{IP:destination.ip}|%{GREEDYDATA:destination.domain})" - ] - } - } - } - if [client.address] { - grok { - id => "filter_grok_19" - match => { - "client.address" => [ - "(?:%{IP:client.ip}|%{GREEDYDATA:client.domain})" - ] - } - } - } - if [server.address] { - grok { - id => "filter_grok_20" - match => { - "server.address" => [ - "(?:%{IP:server.ip}|%{GREEDYDATA:server.domain})" - ] - } - } - } - mutate { - id => "filter_mutate_36" - lowercase => ["network.transport", "network.protocol", "network.direction", "event.outcome"] - } - if [event.outcome] == "est-allowed" { - mutate { - id => "filter_mutate_37" - update => { - "event.outcome" => "allow" - } - } - } - else if [event.outcome] == "permitted" { - mutate { - id => "filter_mutate_38" - update => { - "event.outcome" => "allow" - } - } - } - else if [event.outcome] == "denied" { - mutate { - id => "filter_mutate_39" - update => { - "event.outcome" => "deny" - } - } - } - else if [event.outcome] == "dropped" { - mutate { - id => "filter_mutate_40" - update => { - "event.outcome" => "deny" - } - } - } - if [network.transport] == "icmpv6" { - mutate { - id => "filter_mutate_41" - update => { - "network.transport" => "ipv6-icmp" - } - } - } - translate { - id => "filter_translate_1" - field => "network.transport" - destination => "network.iana_number" - dictionary => { - "icmp" => "1" - "igmp" => "2" - "ipv4" => "4" - "tcp" => "6" - "egp" => "8" - "igp" => "9" - "pup" => "12" - "udp" => "17" - "rdp" => "27" - "irtp" => "28" - "dccp" => "33" - "idpr" => "35" - "ipv6" => "41" - "ipv6-route" => "43" - "ipv6-frag" => "44" - "rsvp" => "46" - "gre" => "47" - "esp" => "50" - "ipv6-icmp" => "58" - "ipv6-nonxt" => "59" - "ipv6-opts" => "60" - } - } - mutate { - id => "filter_mutate_42" - remove_field => ["ciscotag", "timestamp"] - } - mutate { - id => "filter_mutate_43" - add_field => { - "event.module" => "cisco" - "event.dataset" => "asa" - } - } - translate { - id => "filter_translate_2" - field => "[event.severity]" - destination => "[log.level]" - dictionary => { - "0" => "emergency" - "1" => "alert" - "2" => "critical" - "3" => "error" - "4" => "warning" - "5" => "notification" - "6" => "informational" - "7" => "debug" - } - } -} -output { - elasticsearch { - id => "output_elasticsearch_1" - api_key => "${es_api_key}" - hosts => "https://homedc-90e54c.es.us-east-2.aws.elastic-cloud.com:443" - index => "asa-1.2" - pipeline => "asa" - } -} diff --git a/src/logstashui/Common/tests/conversion_data/pipelines/test-boolean-numeric.conf b/src/logstashui/Common/tests/conversion_data/pipelines/test-boolean-numeric.conf deleted file mode 100644 index 663b5bd..0000000 --- a/src/logstashui/Common/tests/conversion_data/pipelines/test-boolean-numeric.conf +++ /dev/null @@ -1,58 +0,0 @@ -input { - tcp { - port => 5000 - ssl_enable => "false" - buffer_size => 65536 - } - udp { - port => 514 - queue_size => 2000 - workers => 4 - } -} -filter { - grok { - match => { - "message" => "%{NUMBER:duration:float} %{NUMBER:status:int}" - } - keep_empty_captures => "false" - tag_on_failure => ["_grokfailure"] - timeout_millis => 30000 - break_on_match => "true" - } - mutate { - convert => { - "duration" => "float" - "status" => "integer" - "bytes" => "integer" - } - } - if [duration] > 1.5 { - mutate { - add_field => { - "slow_request" => "true" - } - } - } - if [status] >= 500 { - mutate { - add_field => { - "is_error" => "true" - "error_code" => 500 - "threshold_pct" => 0.99 - } - } - } - throttle { - before_count => 3 - after_count => 1 - period => 60 - key => "%{host}" - add_tag => ["throttled"] - } -} -output { - stdout { - codec => rubydebug - } -} diff --git a/src/logstashui/Common/tests/conversion_data/pipelines/test-comments-brace-in-comment.conf b/src/logstashui/Common/tests/conversion_data/pipelines/test-comments-brace-in-comment.conf deleted file mode 100644 index e8be51d..0000000 --- a/src/logstashui/Common/tests/conversion_data/pipelines/test-comments-brace-in-comment.conf +++ /dev/null @@ -1,39 +0,0 @@ -input { -} -filter { - mutate { - # add_field => {"this_key" => "this_value"} - # rename => {"old_field" => "new_field"} - # remove_field => ["field1", "field2"] - # replace => {"message" => "override: %{message}"} - add_field => { - "real_field" => "real_value" - } - remove_field => ["unwanted"] - } - grok { - # match => { "message" => "%{COMBINEDAPACHELOG}" } - # pattern_definitions => { "MY_PATTERN" => "\\w+" } - match => { - "message" => "%{GREEDYDATA:raw_message}" - } - } - date { - # match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z", "ISO8601"] - # target => "@timestamp" - match => ["timestamp", "ISO8601"] - target => "@timestamp" - } - translate { - # dictionary => { "200" => "OK", "404" => "Not Found", "500" => "Error" } - field => "status_code" - destination => "status_label" - dictionary => { - "200" => "OK" - "404" => "Not Found" - } - fallback => "Unknown" - } -} -output { -} diff --git a/src/logstashui/Common/tests/conversion_data/pipelines/test-comments-mixed.conf b/src/logstashui/Common/tests/conversion_data/pipelines/test-comments-mixed.conf deleted file mode 100644 index f369943..0000000 --- a/src/logstashui/Common/tests/conversion_data/pipelines/test-comments-mixed.conf +++ /dev/null @@ -1,57 +0,0 @@ -input { - # inline on section opener - udp { - # standalone inside plugin - # add_field => {"commented_out" => "value"} standalone with braces - # inline on plugin opener - # inline on scalar value - # inline on array - port => 5140 - buffer_size => 65536 - tags => ["syslog"] - } - # inline on plugin closer -> section comment -} -filter { - # inline on filter opener - # standalone at section level - mutate { - # standalone before first pair - # standalone between pairs - # standalone at end of plugin - # inline on mutate opener - # inline on hash opener - # inline on hash pair - # inline on hash closer - # inline on array pair - add_field => { - "key1" => "value1" - "key2" => "value2" - } - remove_field => ["old"] - } - # inline on plugin closer -> section comment - # standalone between plugins - if [type] == "web" { - # standalone inside conditional - grok { - # standalone inside nested plugin - # match => {"message" => "%{GREEDYDATA}"} standalone with braces in conditional - # inline on nested plugin opener - # inline inside nested hash - # inline on nested hash closer - match => { - "message" => "%{COMBINEDAPACHELOG}" - } - } - # inline on nested plugin closer - # standalone at end of conditional block - } - drop { - } -} -output { - stdout { - codec => rubydebug - } -} diff --git a/src/logstashui/Common/tests/conversion_data/pipelines/test-comments-plugin-inline.conf b/src/logstashui/Common/tests/conversion_data/pipelines/test-comments-plugin-inline.conf deleted file mode 100644 index 621e651..0000000 --- a/src/logstashui/Common/tests/conversion_data/pipelines/test-comments-plugin-inline.conf +++ /dev/null @@ -1,46 +0,0 @@ -input { - udp { - # inline comment on plugin opener - # inline comment on a scalar value - # another scalar inline comment - # inline comment after an array - port => 5140 - buffer_size => 65536 - tags => ["udp", "syslog"] - } - # inline comment on plugin closer — becomes section-level comment -} -filter { - mutate { - # opener comment - # inline comment on hash opener - # inline comment on hash pair - # another hash pair comment - # inline comment on hash closer - # inline on array - # another hash opener with inline - # pair comment - # hash closer comment - add_field => { - "first" => "value1" - "second" => "value2" - } - remove_field => ["unwanted", "junk"] - rename => { - "old_name" => "new_name" - } - } - # plugin closer — section-level - drop { - # opener comment on empty plugin - } - # closer comment on empty plugin -} -output { - stdout { - # output plugin with inline - # inline on codec line - codec => rubydebug - } - # closer -} diff --git a/src/logstashui/Common/tests/conversion_data/pipelines/test-comments-section-opener.conf b/src/logstashui/Common/tests/conversion_data/pipelines/test-comments-section-opener.conf deleted file mode 100644 index ce0490b..0000000 --- a/src/logstashui/Common/tests/conversion_data/pipelines/test-comments-section-opener.conf +++ /dev/null @@ -1,29 +0,0 @@ -input { - # inline comment on input opener - beats { - # inline on beats opener - port => 5044 - } -} -filter { - # inline comment on filter opener - mutate { - add_field => { - "processed" => "true" - } - } - # standalone comment between plugins at section level - drop { - } -} -output { - # inline comment on output opener - elasticsearch { - hosts => ["localhost:9200"] - index => "logs-%{+YYYY.MM.dd}" - } - # standalone at section level before second output plugin - stdout { - codec => rubydebug - } -} diff --git a/src/logstashui/Common/tests/conversion_data/pipelines/test-comments-standalone-in-plugin.conf b/src/logstashui/Common/tests/conversion_data/pipelines/test-comments-standalone-in-plugin.conf deleted file mode 100644 index 1beb186..0000000 --- a/src/logstashui/Common/tests/conversion_data/pipelines/test-comments-standalone-in-plugin.conf +++ /dev/null @@ -1,29 +0,0 @@ -input { -} -filter { - mutate { - # This is a standalone comment at the top of a plugin block - # Standalone comment in the middle of a plugin block - # Another standalone at the bottom - add_field => { - "field1" => "value1" - } - remove_field => ["junk"] - } - grok { - # Standalone comment before the only config key - # Standalone at the end of plugin - match => { - "message" => "%{COMBINEDAPACHELOG}" - } - } - # Section-level comment between plugins - mutate { - # Leading standalone - # Second leading standalone - # Trailing standalone - uppercase => ["log_level"] - } -} -output { -} diff --git a/src/logstashui/Common/tests/conversion_data/pipelines/test-complex2.conf b/src/logstashui/Common/tests/conversion_data/pipelines/test-complex2.conf deleted file mode 100644 index 555150b..0000000 --- a/src/logstashui/Common/tests/conversion_data/pipelines/test-complex2.conf +++ /dev/null @@ -1,329 +0,0 @@ -input { - # - # "LogstashUI kitchen sink" pipeline - # Goal: be extremely feature-rich while staying within known-valid plugin options. - # - # Beats / Elastic Agent style shippers - beats { - id => "in_beats_5044" - port => "5044" - add_field => { - "ingest_transport" => "beats" - } - tags => ["from_beats"] - } - # JSON-over-TCP (common for app logs) - tcp { - id => "in_tcp_json_5514" - port => 5514 - mode => "server" - codec => json - add_field => { - "ingest_transport" => "tcp" - } - tags => ["from_tcp"] - } - # Syslog-ish UDP - udp { - id => "in_udp_5515" - port => 5515 - codec => plain - add_field => { - "ingest_transport" => "udp" - } - tags => ["from_udp"] - } - # HTTP event intake (webhooks, apps posting JSON, etc.) - http { - id => "in_http_8080" - port => 8080 - codec => json - add_field => { - "ingest_transport" => "http" - } - tags => ["from_http"] - } - # Local dev/testing input - stdin { - id => "in_stdin" - codec => line - add_field => { - "ingest_transport" => "stdin" - } - tags => ["from_stdin"] - } - # Synthetic test data (makes it easy to validate end-to-end quickly) - generator { - id => "in_generator" - lines => ["Feb 21 09:12:01 host1 sshd[123]: Failed password for invalid user admin from 10.1.2.3 port 51234 ssh2", "{\"@timestamp\":\"2026-02-21T14:12:02Z\",\"message\":\"GET /health 200\",\"source_ip\":\"8.8.8.8\",\"user_agent\":\"Mozilla/5.0\"}", "level=info service=api latency_ms=42 source_ip=192.168.1.50 msg=\"request completed\""] - count => 1 - add_field => { - "ingest_transport" => "generator" - } - tags => ["from_generator"] - } -} -filter { - # - # Normalize a few shared fields - # - mutate { - id => "f_mutate_bootstrap" - add_field => { - "[@metadata][pipeline]" => "logstashui_kitchen_sink" - "event.module" => "logstashui" - } - } - # Keep a canonical message field - if ![message] and [event][original] { - mutate { - id => "f_mutate_event_original_to_message" - copy => { - "[event][original]" => "message" - } - } - } - # - # Try to parse JSON *if* message looks like JSON (common when tcp/udp/plain feed JSON strings) - # - if [message] =~ "^[[:space:]]*\\{" { - json { - id => "f_json_from_message" - source => "message" - target => "json" - tag_on_failure => ["_jsonparsefailure_message"] - } - # If json parsed, promote a few expected keys (only if present) - if [json][@timestamp] { - mutate { - id => "f_promote_json_ts" - copy => { - "[json][@timestamp]" => "@timestamp" - } - } - } - if [json][source_ip] { - mutate { - id => "f_promote_json_source_ip" - copy => { - "[json][source_ip]" => "source_ip" - } - } - } - if [json][user_agent] { - mutate { - id => "f_promote_json_ua" - copy => { - "[json][user_agent]" => "user_agent" - } - } - } - } - # - # Syslog-ish parsing (UDP and some TCP) - # - if "from_udp" in [tags] or "from_tcp" in [tags] { - # Try dissect first (fast) and fall back to grok - dissect { - id => "f_dissect_syslogish" - mapping => { - "message" => "%{syslog_timestamp} %{syslog_host} %{syslog_program}[%{syslog_pid}]: %{syslog_message}" - } - tag_on_failure => ["_dissectfailure_syslogish"] - } - if "_dissectfailure_syslogish" in [tags] { - grok { - id => "f_grok_syslogish" - match => { - "message" => [ - "%{SYSLOGTIMESTAMP:syslog_timestamp} %{HOSTNAME:syslog_host} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" - ] - } - tag_on_failure => ["_grokparsefailure_syslogish"] - } - } - # If we extracted a syslog timestamp, use it - if [syslog_timestamp] { - date { - id => "f_date_syslog" - match => ["syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"] - tag_on_failure => ["_dateparsefailure_syslog"] - } - } - } - # - # key=value parsing for “flat” log lines - # - if [message] =~ "([A-Za-z0-9_.-]+)=([^\"]\\S+|\"[^\"]*\")" { - kv { - id => "f_kv_message" - source => "message" - trim_key => " " - trim_value => " " - value_split => "=" - field_split_pattern => "\s+" - tag_on_failure => ["_kvfailure_message"] - } - } - # - # Basic typing / normalization - # - mutate { - id => "f_mutate_normalize" - rename => { - "msg" => "message_short" - } - convert => { - "latency_ms" => "integer" - } - lowercase => ["level"] - } - # - # Enrichments: useragent, geoip, cidr, dns - # - if [user_agent] { - useragent { - id => "f_useragent" - source => "user_agent" - target => "user_agent_parsed" - } - } - # Canonicalize IP into source_ip if it exists elsewhere - if ![source_ip] and [source][ip] { - mutate { - id => "f_copy_source_ip" - copy => { - "[source][ip]" => "source_ip" - } - } - } - if [source_ip] { - # Tag private vs public - cidr { - id => "f_cidr_private" - address => ["%{source_ip}"] - network => ["10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"] - add_tag => ["src_private"] - } - # GeoIP typically only makes sense for public IPs, so do it only if not private-tagged - if "src_private" not in [tags] { - geoip { - id => "f_geoip" - source => "source_ip" - target => "source_geo" - } - } - # Reverse DNS lookup; replace source_ip with hostname when possible (or leave as-is) - dns { - id => "f_dns_reverse" - reverse => ["source_ip"] - action => "replace" - } - } - # - # Translate severity/level into a normalized numeric - # - translate { - id => "f_translate_level_to_severity" - source => "level" - target => "severity" - dictionary => { - "trace" => "0" - "debug" => "1" - "info" => "2" - "warn" => "3" - "error" => "4" - "fatal" => "5" - } - fallback => "2" - } - mutate { - id => "f_convert_severity_int" - convert => { - "severity" => "integer" - } - } - # - # Stable fingerprint for dedup / correlation - # - fingerprint { - id => "f_fingerprint_message" - source => ["message"] - method => "MURMUR3" - target => "[@metadata][fingerprint]" - } - # - # Example branching: treat auth-ish messages specially - # - if [syslog_program] == "sshd" or [message] =~ "(?i)failed password|authentication failure|invalid user" { - mutate { - id => "f_tag_auth" - add_tag => ["category_auth"] - add_field => { - "event.category" => "authentication" - } - } - } - else if [message] =~ "(?i)GET\\s+/health|/ready|/live" { - mutate { - id => "f_tag_health" - add_tag => ["category_healthcheck"] - add_field => { - "event.category" => "availability" - } - } - } - else { - mutate { - id => "f_tag_generic" - add_tag => ["category_generic"] - } - } - # - # Prune down noisy fields (keeps top-level essentials) - # - prune { - id => "f_prune" - whitelist_names => ["^@timestamp$", "^message$", "^message_short$", "^host$", "^source_ip$", "^source_geo$", "^severity$", "^level$", "^tags$", "^event\\..*$", "^user_agent.*$", "^syslog_.*$", "^ingest_transport$"] - } -} -output { - # Always see something in console during dev - stdout { - id => "out_stdout_rubydebug" - codec => rubydebug { - metadata => "true" - } - } - # Write to disk (great for debugging replay) - file { - id => "out_file_jsonl" - path => "/tmp/logstashui-%{+YYYY.MM.dd}.jsonl" - codec => json_lines - } - # Elasticsearch (local default) - elasticsearch { - id => "out_es_local" - hosts => ["http://localhost:9200"] - index => "logstashui-%{+YYYY.MM.dd}" - ilm_enabled => "false" - } - # Webhook back to your UI/API (example) - http { - id => "out_http_callback" - url => "http://localhost:9000/logstash/callback" - http_method => "post" - format => "json" - } - # Kafka (example) - kafka { - id => "out_kafka" - bootstrap_servers => "localhost:9092" - topic_id => "logstashui-events" - } - # Pipeline-to-pipeline (requires another pipeline with pipeline input address => "downstream") - pipeline { - id => "out_pipeline_downstream" - send_to => ["downstream"] - } -} diff --git a/src/logstashui/Common/tests/conversion_data/pipelines/test-complex3.conf b/src/logstashui/Common/tests/conversion_data/pipelines/test-complex3.conf deleted file mode 100644 index 83e9b91..0000000 --- a/src/logstashui/Common/tests/conversion_data/pipelines/test-complex3.conf +++ /dev/null @@ -1,652 +0,0 @@ -input { - beats { - port => "5044" - ssl => "true" - ssl_certificate => "/etc/logstash/certs/server.crt" - ssl_key => "/etc/logstash/certs/server.key" - ssl_verify_mode => "force_peer" - ssl_certificate_authorities => ["/etc/logstash/certs/ca.crt"] - codec => json - type => "beats" - tags => ["beats_ssl"] - } - http { - port => "8080" - codec => json - ssl => "true" - ssl_certificate => "/etc/logstash/certs/http.crt" - ssl_key => "/etc/logstash/certs/http.key" - threads => "4" - max_pending_requests => "100" - response_headers => { - "Content-Type" => "application/json" - } - type => "webhook" - tags => ["http_api"] - } - kafka { - bootstrap_servers => "kafka1:9092,kafka2:9092,kafka3:9092" - topics => ["app-logs", "security-events", "metrics"] - group_id => "logstash-consumer" - consumer_threads => "3" - codec => avro { - schema_uri => "http://schema-registry:8081/schemas/ids/1" - } - decorate_events => "true" - security_protocol => "SASL_SSL" - sasl_mechanism => "SCRAM-SHA-512" - sasl_jaas_config => "org.apache.kafka.common.security.scram.ScramLoginModule required username='logstash' password='${KAFKA_PASS}';" - type => "kafka" - tags => ["kafka_stream"] - } - jdbc { - jdbc_driver_library => "/usr/share/logstash/vendor/jar/jdbc/postgresql.jar" - jdbc_driver_class => "org.postgresql.Driver" - jdbc_connection_string => "jdbc:postgresql://db:5432/prod" - jdbc_user => "${DB_USER}" - jdbc_password => "${DB_PASS}" - schedule => "*/5 * * * *" - statement => "SELECT * FROM events WHERE created_at > :sql_last_value" - use_column_value => "true" - tracking_column => "created_at" - tracking_column_type => "timestamp" - type => "database" - tags => ["jdbc_poll"] - } - file { - path => ["/var/log/nginx/*.log", "/var/log/app/**/*.log"] - start_position => "beginning" - sincedb_path => "/var/lib/logstash/sincedb" - codec => multiline { - pattern => "^%{TIMESTAMP_ISO8601}" - negate => "true" - what => "previous" - max_lines => 500 - } - type => "file" - tags => ["file_input"] - } - tcp { - port => "5000" - codec => json_lines - ssl_enable => "true" - ssl_cert => "/etc/logstash/certs/tcp.crt" - ssl_key => "/etc/logstash/certs/tcp.key" - type => "tcp_json" - tags => ["tcp_secure"] - } - udp { - port => "514" - codec => cef - type => "syslog" - tags => ["syslog_udp"] - } - rabbitmq { - host => "rabbitmq" - port => "5672" - user => "${RABBIT_USER}" - password => "${RABBIT_PASS}" - queue => "logs" - exchange => "logs-exchange" - exchange_type => "topic" - key => "logs.#" - durable => "true" - codec => json - type => "rabbitmq" - tags => ["amqp"] - } - redis { - host => "redis" - port => "6379" - password => "${REDIS_PASS}" - data_type => "list" - key => "logstash:queue" - codec => json - type => "redis" - tags => ["redis_queue"] - } - s3 { - bucket => "logs-archive" - region => "us-east-1" - access_key_id => "${AWS_KEY}" - secret_access_key => "${AWS_SECRET}" - interval => "300" - codec => json_lines - type => "s3" - tags => ["s3_archive"] - } - kinesis { - kinesis_stream_name => "app-stream" - region => "us-west-2" - codec => json - type => "kinesis" - tags => ["aws_kinesis"] - } -} -filter { - if [type] == "beats" { - if [agent][type] == "filebeat" { - if [log][file][path] =~ /nginx/ { - grok { - match => { - "message" => '%{IPORHOST:client_ip} - %{DATA:user} \[%{HTTPDATE:timestamp}\] "%{WORD:method} %{DATA:path} HTTP/%{NUMBER:version}" %{NUMBER:status:int} %{NUMBER:bytes:int} "%{DATA:referrer}" "%{DATA:agent}"' - } - } - date { - match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"] - target => "@timestamp" - } - useragent { - source => "agent" - target => "ua" - } - geoip { - source => "client_ip" - target => "geo" - database => "/usr/share/GeoIP/GeoLite2-City.mmdb" - } - if [status] >= 500 { - mutate { - add_tag => ["error", "server_error"] - add_field => { - "severity" => "critical" - } - } - } - else if [status] >= 400 { - mutate { - add_tag => ["error", "client_error"] - add_field => { - "severity" => "warning" - } - } - } - ruby { - code => ' - bytes = event.get("bytes").to_i - if bytes > 10485760 - event.set("size_class", "large") - elsif bytes > 1048576 - event.set("size_class", "medium") - else - event.set("size_class", "small") - end - ' - } - fingerprint { - source => ["client_ip", "path", "timestamp"] - target => "[@metadata][fingerprint]" - method => "SHA256" - } - } - else if [log][file][path] =~ /application/ { - json { - source => "message" - target => "app" - } - if [app][level] { - translate { - field => "[app][level]" - destination => "severity_num" - dictionary => { - "DEBUG" => "1" - "INFO" => "2" - "WARN" => "3" - "ERROR" => "4" - "FATAL" => "5" - } - fallback => "2" - } - mutate { - convert => { - "severity_num" => "integer" - } - } - } - if [app][exception] { - mutate { - add_tag => ["exception"] - } - ruby { - code => ' - exc = event.get("[app][exception]") - if exc.is_a?(Hash) - event.set("exception_class", exc["class"]) - event.set("exception_msg", exc["message"]) - end - ' - } - } - } - } - else if [agent][type] == "metricbeat" { - if [system][cpu] { - ruby { - code => ' - cpu = event.get("[system][cpu]") - if cpu && cpu["cores"] - total = 0.0 - cpu["cores"].each { |c| total += c["user"]["pct"].to_f if c["user"] } - avg = total / cpu["cores"].length - event.set("[system][cpu][avg_pct]", avg.round(2)) - event.tag("cpu_warning") if avg > 75 - event.tag("cpu_critical") if avg > 90 - end - ' - } - } - if [system][memory][actual][used][pct] { - ruby { - code => ' - pct = event.get("[system][memory][actual][used][pct]").to_f * 100 - event.set("mem_used_pct", pct.round(2)) - event.tag("memory_warning") if pct > 85 - event.tag("memory_critical") if pct > 95 - ' - } - } - } - } - else if [type] == "kafka" { - if [kafka][topic] == "security-events" { - json { - source => "message" - target => "security" - } - if [security][ip] { - cidr { - address => ["%{[security][ip]}"] - network => ["10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"] - add_tag => ["internal_ip"] - } - if "internal_ip" not in [tags] { - geoip { - source => "[security][ip]" - target => "threat_geo" - database => "/usr/share/GeoIP/GeoLite2-City.mmdb" - } - geoip { - source => "[security][ip]" - target => "threat_asn" - database => "/usr/share/GeoIP/GeoLite2-ASN.mmdb" - default_database_type => "ASN" - } - } - } - if [security][event_type] { - translate { - field => "[security][event_type]" - destination => "threat_score" - dictionary => { - "brute_force" => "75" - "sql_injection" => "90" - "xss" => "85" - "unauthorized" => "80" - "privilege_escalation" => "95" - "malware" => "100" - } - fallback => "50" - } - mutate { - convert => { - "threat_score" => "integer" - } - } - if [threat_score] >= 90 { - mutate { - add_tag => ["critical_threat"] - } - } - } - ruby { - code => ' - score = event.get("threat_score").to_i - is_ext = event.get("tags").include?("internal_ip") ? 0 : 20 - composite = score + is_ext - event.set("composite_risk", [composite, 100].min) - - if composite >= 100 - event.set("risk", "critical") - elsif composite >= 80 - event.set("risk", "high") - elsif composite >= 60 - event.set("risk", "medium") - else - event.set("risk", "low") - end - ' - } - } - else if [kafka][topic] == "metrics" { - dissect { - mapping => { - "metric" => "%{env}.%{dc}.%{host}.%{service}.%{type}.%{name}" - } - } - if [type] == "response_time" { - ruby { - code => ' - val = event.get("value").to_f - if val > 5000 - event.set("perf_status", "critical") - elsif val > 2000 - event.set("perf_status", "slow") - else - event.set("perf_status", "normal") - end - ' - } - } - aggregate { - task_id => "%{service}_%{name}" - code => ' - map["count"] ||= 0 - map["sum"] ||= 0.0 - map["min"] ||= Float::INFINITY - map["max"] ||= -Float::INFINITY - - val = event.get("value").to_f - map["count"] += 1 - map["sum"] += val - map["min"] = [map["min"], val].min - map["max"] = [map["max"], val].max - - avg = map["sum"] / map["count"] - event.set("rolling_avg", avg.round(2)) - event.set("rolling_min", map["min"]) - event.set("rolling_max", map["max"]) - ' - timeout => "300" - } - } - } - else if [type] == "database" { - if [event_data] { - json { - source => "event_data" - target => "evt" - } - } - date { - match => ["created_at", "ISO8601", "yyyy-MM-dd HH:mm:ss"] - target => "@timestamp" - } - elasticsearch { - hosts => ["http://elasticsearch:9200"] - index => "user-profiles" - query_template => "user_lookup.json" - fields => { - "department" => "user_dept" - "role" => "user_role" - } - } - if [event_type] =~ /^(login|logout|password_change)$/ { - mutate { - add_tag => ["auth_event"] - } - } - else if [event_type] =~ /^(create|update|delete)$/ { - mutate { - add_tag => ["data_operation"] - } - } - } - else if [type] == "webhook" { - if [headers][user-agent] { - useragent { - source => "[headers][user-agent]" - target => "webhook_ua" - } - } - if [headers][x-signature] { - ruby { - init => 'require "openssl"' - code => ' - sig = event.get("[headers][x-signature]") - payload = event.get("message").to_s - secret = ENV["WEBHOOK_SECRET"] - expected = "sha256=" + OpenSSL::HMAC.hexdigest("SHA256", secret, payload) - - if sig == expected - event.set("sig_valid", true) - else - event.set("sig_valid", false) - event.tag("invalid_signature") - end - ' - } - } - aggregate { - task_id => "%{[headers][x-forwarded-for]}" - code => ' - map["count"] ||= 0 - map["count"] += 1 - event.set("request_count", map["count"]) - ' - timeout => "60" - } - if [request_count] and [request_count] > 100 { - mutate { - add_tag => ["rate_limit_exceeded"] - } - } - } - if [message] =~ /=/ { - kv { - source => "message" - field_split => "&" - value_split => "=" - target => "parsed" - } - } - if [user_agent] and ![ua] { - useragent { - source => "user_agent" - target => "ua" - } - } - if [source_ip] { - dns { - reverse => ["source_ip"] - action => "append" - nameserver => ["8.8.8.8"] - hit_cache_size => "10000" - hit_cache_ttl => "3600" - } - } - prune { - whitelist_names => ["^@", "^_", "type", "tags", "message"] - } - mutate { - add_field => { - "env" => "${ENVIRONMENT:prod}" - "cluster" => "${CLUSTER:default}" - } - remove_field => ["@version"] - } - if [env] == "production" and ([tags] and "debug" in [tags]) { - drop { - } - } - throttle { - before_count => "3" - after_count => "1" - period => "60" - key => "%{fingerprint}" - add_tag => ["throttled"] - } - if "critical_threat" in [tags] { - clone { - clones => ["siem"] - add_field => { - "cloned" => "true" - } - } - } - metrics { - meter => ["events"] - add_tag => ["metric"] - flush_interval => "30" - rates => [1, 5, 15] - } - if ([severity] == "critical" or [severity] == "error") and ([status] >= 500 or [threat_score] >= 90) { - mutate { - add_field => { - "priority" => "P1" - "oncall" => "true" - } - add_tag => ["p1"] - } - } - else if ([severity] == "warning") and ([status] >= 400 or [threat_score] >= 70) { - mutate { - add_field => { - "priority" => "P2" - } - add_tag => ["p2"] - } - } - fingerprint { - source => "message" - target => "event_hash" - method => "MURMUR3" - } - uuid { - target => "event_id" - } - ruby { - code => ' - event.set("processed_at", Time.now.utc.iso8601) - event.set("pipeline_v", "2.0") - ' - } -} -output { - if "throttled" not in [tags] and "metric" not in [tags] { - elasticsearch { - hosts => ["https://es1:9200", "https://es2:9200"] - user => "${ES_USER}" - password => "${ES_PASS}" - ssl => "true" - cacert => "/etc/logstash/certs/ca.crt" - index => "%{type}-%{+YYYY.MM.dd}" - document_id => "%{event_id}" - pipeline => "enrich" - ilm_enabled => "true" - ilm_rollover_alias => "%{type}" - ilm_pattern => "{now/d}-000001" - ilm_policy => "logs-policy" - http_compression => "true" - } - } - if "siem" in [tags] { - elasticsearch { - hosts => ["https://siem-es:9200"] - user => "${SIEM_USER}" - password => "${SIEM_PASS}" - ssl => "true" - cacert => "/etc/logstash/certs/siem-ca.crt" - index => "security-%{+YYYY.MM.dd}" - } - } - if [env] == "production" { - s3 { - access_key_id => "${AWS_KEY}" - secret_access_key => "${AWS_SECRET}" - region => "us-east-1" - bucket => "logs-archive" - size_file => "104857600" - time_file => "15" - codec => json_lines - prefix => "logs/%{type}/year=%{+YYYY}/month=%{+MM}/day=%{+dd}" - encoding => "gzip" - server_side_encryption => "true" - } - } - kafka { - bootstrap_servers => "kafka1:9092,kafka2:9092" - topic_id => "processed-%{type}" - codec => json - compression_type => "snappy" - acks => "all" - security_protocol => "SASL_SSL" - sasl_mechanism => "SCRAM-SHA-512" - sasl_jaas_config => "org.apache.kafka.common.security.scram.ScramLoginModule required username='${KAFKA_USER}' password='${KAFKA_PASS}';" - } - if "p1" in [tags] or "critical" in [tags] { - redis { - host => ["redis1", "redis2"] - port => "26379" - password => "${REDIS_PASS}" - data_type => "list" - key => "alerts:critical" - } - http { - url => "https://alerts.example.com/api/events" - http_method => "post" - format => "json" - headers => { - "Authorization" => "Bearer ${ALERT_TOKEN}" - "Content-Type" => "application/json" - } - automatic_retries => "3" - } - } - if [env] != "production" { - file { - path => "/var/log/logstash/debug-%{type}.log" - codec => json_lines - } - } - if "metric" in [tags] { - graphite { - host => "graphite" - port => "2003" - metrics_format => "logstash.%{env}.%{type}.count" - fields_are_metrics => "true" - } - influxdb { - host => "influxdb" - port => "8086" - db => "metrics" - user => "${INFLUX_USER}" - password => "${INFLUX_PASS}" - measurement => "%{type}_metrics" - use_event_fields_for_data_points => "true" - } - } - if [type] == "rabbitmq" { - mongodb { - uri => "mongodb://${MONGO_USER}:${MONGO_PASS}@mongo:27017/logs" - database => "logs" - collection => "%{type}" - isodate => "true" - bulk => "true" - bulk_size => "100" - } - } - if "p1" in [tags] { - email { - to => "oncall@example.com" - from => "alerts@example.com" - subject => "P1 Alert: %{type}" - body => 'Event: %{event_id} - Time: %{@timestamp} - Severity: %{severity} - Message: %{message}' - address => "smtp.example.com" - port => "587" - use_tls => "true" - username => "${SMTP_USER}" - password => "${SMTP_PASS}" - } - } - tcp { - host => "logstash-secondary" - port => "5005" - codec => json_lines - } - stdout { - codec => rubydebug - } -} diff --git a/src/logstashui/Common/tests/conversion_data/pipelines/test-data-types.conf b/src/logstashui/Common/tests/conversion_data/pipelines/test-data-types.conf deleted file mode 100644 index 94180f6..0000000 --- a/src/logstashui/Common/tests/conversion_data/pipelines/test-data-types.conf +++ /dev/null @@ -1,33 +0,0 @@ -input { -} -filter { - grok { - match => { - "test" => "test" - } - pattern_definitions => { - "1" => "2" - "test" => "test" - "asd" => "asf" - } - patterns_dir => ["test"] - tag_on_failure => [] - } - grok { - match => { - "test" => [ - "test", - "test2" - ] - } - pattern_definitions => { - "test" => "test" - } - patterns_dir => "test" - } - # test - # multi - # row -} -output { -} diff --git a/src/logstashui/Common/tests/conversion_data/pipelines/test-datatypes.conf b/src/logstashui/Common/tests/conversion_data/pipelines/test-datatypes.conf deleted file mode 100644 index 94180f6..0000000 --- a/src/logstashui/Common/tests/conversion_data/pipelines/test-datatypes.conf +++ /dev/null @@ -1,33 +0,0 @@ -input { -} -filter { - grok { - match => { - "test" => "test" - } - pattern_definitions => { - "1" => "2" - "test" => "test" - "asd" => "asf" - } - patterns_dir => ["test"] - tag_on_failure => [] - } - grok { - match => { - "test" => [ - "test", - "test2" - ] - } - pattern_definitions => { - "test" => "test" - } - patterns_dir => "test" - } - # test - # multi - # row -} -output { -} diff --git a/src/logstashui/Common/tests/conversion_data/pipelines/test-devopsschool-1.conf b/src/logstashui/Common/tests/conversion_data/pipelines/test-devopsschool-1.conf deleted file mode 100644 index 6d068c5..0000000 --- a/src/logstashui/Common/tests/conversion_data/pipelines/test-devopsschool-1.conf +++ /dev/null @@ -1,21 +0,0 @@ -input { - beats { - port => "5044" - } -} -filter { - grok { - match => { - "message" => "%{COMBINEDAPACHELOG}" - } - } -} -output { - elasticsearch { - hosts => ["http://elasticsearch:9200"] - index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" - } - stdout { - codec => rubydebug - } -} diff --git a/src/logstashui/Common/tests/conversion_data/pipelines/test-devopsschool-2.conf b/src/logstashui/Common/tests/conversion_data/pipelines/test-devopsschool-2.conf deleted file mode 100644 index bd9f369..0000000 --- a/src/logstashui/Common/tests/conversion_data/pipelines/test-devopsschool-2.conf +++ /dev/null @@ -1,17 +0,0 @@ -input { - beats { - port => "5044" - } -} -filter { - grok { - match => { - "message" => "%{SYSLOGLINE}" - } - } -} -output { - stdout { - codec => rubydebug - } -} diff --git a/src/logstashui/Common/tests/conversion_data/pipelines/test-devopsschool-4.conf b/src/logstashui/Common/tests/conversion_data/pipelines/test-devopsschool-4.conf deleted file mode 100644 index 4e32410..0000000 --- a/src/logstashui/Common/tests/conversion_data/pipelines/test-devopsschool-4.conf +++ /dev/null @@ -1,25 +0,0 @@ -input { - file { - path => "/var/log/apache2/access.log" - start_position => "beginning" - sincedb_path => "/dev/null" - } -} -filter { - grok { - match => { - "message" => "%{COMBINEDAPACHELOG}" - } - } - date { - match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"] - } - geoip { - source => "clientip" - } -} -output { - elasticsearch { - hosts => ["localhost:9200"] - } -} diff --git a/src/logstashui/Common/tests/conversion_data/pipelines/test-devopsschool-5.conf b/src/logstashui/Common/tests/conversion_data/pipelines/test-devopsschool-5.conf deleted file mode 100644 index 87350a1..0000000 --- a/src/logstashui/Common/tests/conversion_data/pipelines/test-devopsschool-5.conf +++ /dev/null @@ -1,20 +0,0 @@ -input { - beats { - port => "5044" - } -} -filter { - grok { - match => { - "message" => "%{COMBINEDAPACHELOG}" - } - } - geoip { - source => "clientip" - } -} -output { - elasticsearch { - hosts => ["localhost:9200"] - } -} diff --git a/src/logstashui/Common/tests/conversion_data/pipelines/test-elasticdocs-apache.conf b/src/logstashui/Common/tests/conversion_data/pipelines/test-elasticdocs-apache.conf deleted file mode 100644 index 9f87b11..0000000 --- a/src/logstashui/Common/tests/conversion_data/pipelines/test-elasticdocs-apache.conf +++ /dev/null @@ -1,31 +0,0 @@ -input { - file { - path => "/tmp/access_log" - start_position => "beginning" - } -} -filter { - if [path] =~ "access" { - mutate { - replace => { - "type" => "apache_access" - } - } - grok { - match => { - "message" => "%{COMBINEDAPACHELOG}" - } - } - } - date { - match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"] - } -} -output { - elasticsearch { - hosts => ["localhost:9200"] - } - stdout { - codec => rubydebug - } -} diff --git a/src/logstashui/Common/tests/conversion_data/pipelines/test-elasticdocs-configuring_filters.conf b/src/logstashui/Common/tests/conversion_data/pipelines/test-elasticdocs-configuring_filters.conf deleted file mode 100644 index 24d2f57..0000000 --- a/src/logstashui/Common/tests/conversion_data/pipelines/test-elasticdocs-configuring_filters.conf +++ /dev/null @@ -1,22 +0,0 @@ -input { - stdin { - } -} -filter { - grok { - match => { - "message" => "%{COMBINEDAPACHELOG}" - } - } - date { - match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"] - } -} -output { - elasticsearch { - hosts => ["localhost:9200"] - } - stdout { - codec => rubydebug - } -} diff --git a/src/logstashui/Common/tests/conversion_data/pipelines/test-elasticdocs-syslog.conf b/src/logstashui/Common/tests/conversion_data/pipelines/test-elasticdocs-syslog.conf deleted file mode 100644 index 761797d..0000000 --- a/src/logstashui/Common/tests/conversion_data/pipelines/test-elasticdocs-syslog.conf +++ /dev/null @@ -1,31 +0,0 @@ -input { - tcp { - port => "5000" - type => "syslog" - } - udp { - port => "5000" - type => "syslog" - } -} -filter { - if [type] == "syslog" { - grok { - match => { - "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" - } - add_field => ["received_at", "%{@timestamp}", "received_from", "%{host}"] - } - date { - match => ["syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"] - } - } -} -output { - elasticsearch { - hosts => ["localhost:9200"] - } - stdout { - codec => rubydebug - } -} diff --git a/src/logstashui/Common/tests/conversion_data/pipelines/test-es-input.conf b/src/logstashui/Common/tests/conversion_data/pipelines/test-es-input.conf deleted file mode 100644 index a0771af..0000000 --- a/src/logstashui/Common/tests/conversion_data/pipelines/test-es-input.conf +++ /dev/null @@ -1,26 +0,0 @@ -input { - elasticsearch { - api_key => "test" - cloud_id => "test" - index => "kibana_sample_data_ecommerce" - query => '{"query":{"match_all":{}}}' - slices => "6" - ssl_enabled => "true" - connect_timeout_seconds => "120" - request_timeout_seconds => "600" - socket_timeout_seconds => "600" - } -} -filter { - mutate { - add_field => { - "test" => "test" - } - } -} -output { - csv { - fields => ["test"] - path => "/home/ubuntu/test.csv" - } -} diff --git a/src/logstashui/Common/tests/conversion_data/pipelines/test-ls-repo-mysql.conf b/src/logstashui/Common/tests/conversion_data/pipelines/test-ls-repo-mysql.conf deleted file mode 100644 index 8b3a968..0000000 --- a/src/logstashui/Common/tests/conversion_data/pipelines/test-ls-repo-mysql.conf +++ /dev/null @@ -1,69 +0,0 @@ -input { - beats { - port => 5044 - host => "0.0.0.0" - } -} -filter { - if [fileset][module] == "mysql" { - if [fileset][name] == "error" { - grok { - match => { - "message" => [ - "%{LOCALDATETIME:[mysql][error][timestamp]} (\[%{DATA:[mysql][error][level]}\] )?%{GREEDYDATA:[mysql][error][message]}", - "%{TIMESTAMP_ISO8601:[mysql][error][timestamp]} %{NUMBER:[mysql][error][thread_id]} \[%{DATA:[mysql][error][level]}\] %{GREEDYDATA:[mysql][error][message1]}", - "%{GREEDYDATA:[mysql][error][message2]}" - ] - } - pattern_definitions => { - "LOCALDATETIME" => "[0-9]+ %{TIME}" - } - remove_field => "message" - } - mutate { - rename => { - "[mysql][error][message1]" => "[mysql][error][message]" - } - } - mutate { - rename => { - "[mysql][error][message2]" => "[mysql][error][message]" - } - } - date { - match => ["[mysql][error][timestamp]", "ISO8601", "YYMMdd H:m:s"] - remove_field => "[mysql][error][time]" - } - } - else if [fileset][name] == "slowlog" { - grok { - match => { - "message" => [ - '^# User@Host: %{USER:[mysql][slowlog][user]}(\[[^\]]+\])? @ %{HOSTNAME:[mysql][slowlog][host]} \[(IP:[mysql][slowlog][ip])?\](\s*Id:\s* %{NUMBER:[mysql][slowlog][id]})? - # Query_time: %{NUMBER:[mysql][slowlog][query_time][sec]}\s* Lock_time: %{NUMBER:[mysql][slowlog][lock_time][sec]}\s* Rows_sent: %{NUMBER:[mysql][slowlog][rows_sent]}\s* Rows_examined: %{NUMBER:[mysql][slowlog][rows_examined]} - (SET timestamp=%{NUMBER:[mysql][slowlog][timestamp]}; - )?%{GREEDYMULTILINE:[mysql][slowlog][query]}' - ] - } - pattern_definitions => { - "GREEDYMULTILINE" => '(.| - )*' - } - remove_field => "message" - } - date { - match => ["[mysql][slowlog][timestamp]", "UNIX"] - } - mutate { - gsub => ["[mysql][slowlog][query]", "\n# Time: [0-9]+ [0-9][0-9]:[0-9][0-9]:[0-9][0-9](\\.[0-9]+)?$", ""] - } - } - } -} -output { - elasticsearch { - hosts => "localhost" - manage_template => "false" - index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" - } -} diff --git a/src/logstashui/Common/tests/conversion_data/pipelines/test-ls-repo-nginx.conf b/src/logstashui/Common/tests/conversion_data/pipelines/test-ls-repo-nginx.conf deleted file mode 100644 index 8956944..0000000 --- a/src/logstashui/Common/tests/conversion_data/pipelines/test-ls-repo-nginx.conf +++ /dev/null @@ -1,64 +0,0 @@ -input { - beats { - port => 5044 - host => "0.0.0.0" - } -} -filter { - if [fileset][module] == "nginx" { - if [fileset][name] == "access" { - grok { - match => { - "message" => [ - '%{IPORHOST:[nginx][access][remote_ip]} - %{DATA:[nginx][access][user_name]} \[%{HTTPDATE:[nginx][access][time]}\] "%{WORD:[nginx][access][method]} %{DATA:[nginx][access][url]} HTTP/%{NUMBER:[nginx][access][http_version]}" %{NUMBER:[nginx][access][response_code]} %{NUMBER:[nginx][access][body_sent][bytes]} "%{DATA:[nginx][access][referrer]}" "%{DATA:[nginx][access][agent]}"' - ] - } - remove_field => "message" - } - mutate { - add_field => { - "read_timestamp" => "%{@timestamp}" - } - } - date { - match => ["[nginx][access][time]", "dd/MMM/YYYY:H:m:s Z"] - remove_field => "[nginx][access][time]" - } - useragent { - source => "[nginx][access][agent]" - target => "[nginx][access][user_agent]" - remove_field => "[nginx][access][agent]" - } - geoip { - source => "[nginx][access][remote_ip]" - target => "[nginx][access][geoip]" - } - } - else if [fileset][name] == "error" { - grok { - match => { - "message" => [ - "%{DATA:[nginx][error][time]} \[%{DATA:[nginx][error][level]}\] %{NUMBER:[nginx][error][pid]}#%{NUMBER:[nginx][error][tid]}: (\*%{NUMBER:[nginx][error][connection_id]} )?%{GREEDYDATA:[nginx][error][message]}" - ] - } - remove_field => "message" - } - mutate { - rename => { - "@timestamp" => "read_timestamp" - } - } - date { - match => ["[nginx][error][time]", "YYYY/MM/dd H:m:s"] - remove_field => "[nginx][error][time]" - } - } - } -} -output { - elasticsearch { - hosts => "localhost" - manage_template => "false" - index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" - } -} diff --git a/src/logstashui/Common/tests/conversion_data/pipelines/test-ls-repo-system.conf b/src/logstashui/Common/tests/conversion_data/pipelines/test-ls-repo-system.conf deleted file mode 100644 index ca865b2..0000000 --- a/src/logstashui/Common/tests/conversion_data/pipelines/test-ls-repo-system.conf +++ /dev/null @@ -1,61 +0,0 @@ -input { - beats { - port => 5044 - host => "0.0.0.0" - } -} -filter { - if [fileset][module] == "system" { - if [fileset][name] == "auth" { - grok { - match => { - "message" => [ - "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: %{DATA:[system][auth][ssh][event]} %{DATA:[system][auth][ssh][method]} for (invalid user )?%{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]} port %{NUMBER:[system][auth][ssh][port]} ssh2(: %{GREEDYDATA:[system][auth][ssh][signature]})?", - "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: %{DATA:[system][auth][ssh][event]} user %{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]}", - "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: Did not receive identification string from %{IPORHOST:[system][auth][ssh][dropped_ip]}", - "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sudo(?:\[%{POSINT:[system][auth][pid]}\])?: \s*%{DATA:[system][auth][user]} :( %{DATA:[system][auth][sudo][error]} ;)? TTY=%{DATA:[system][auth][sudo][tty]} ; PWD=%{DATA:[system][auth][sudo][pwd]} ; USER=%{DATA:[system][auth][sudo][user]} ; COMMAND=%{GREEDYDATA:[system][auth][sudo][command]}", - "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} groupadd(?:\[%{POSINT:[system][auth][pid]}\])?: new group: name=%{DATA:system.auth.groupadd.name}, GID=%{NUMBER:system.auth.groupadd.gid}", - "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} useradd(?:\[%{POSINT:[system][auth][pid]}\])?: new user: name=%{DATA:[system][auth][useradd][name]}, UID=%{NUMBER:[system][auth][useradd][uid]}, GID=%{NUMBER:[system][auth][useradd][gid]}, home=%{DATA:[system][auth][useradd][home]}, shell=%{DATA:[system][auth][useradd][shell]}$", - "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} %{DATA:[system][auth][program]}(?:\[%{POSINT:[system][auth][pid]}\])?: %{GREEDYMULTILINE:[system][auth][message]}" - ] - } - pattern_definitions => { - "GREEDYMULTILINE" => '(.| - )*' - } - remove_field => "message" - } - date { - match => ["[system][auth][timestamp]", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"] - } - geoip { - source => "[system][auth][ssh][ip]" - target => "[system][auth][ssh][geoip]" - } - } - else if [fileset][name] == "syslog" { - grok { - match => { - "message" => [ - "%{SYSLOGTIMESTAMP:[system][syslog][timestamp]} %{SYSLOGHOST:[system][syslog][hostname]} %{DATA:[system][syslog][program]}(?:\[%{POSINT:[system][syslog][pid]}\])?: %{GREEDYMULTILINE:[system][syslog][message]}" - ] - } - pattern_definitions => { - "GREEDYMULTILINE" => '(.| - )*' - } - remove_field => "message" - } - date { - match => ["[system][syslog][timestamp]", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"] - } - } - } -} -output { - elasticsearch { - hosts => "localhost" - manage_template => "false" - index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" - } -} diff --git a/src/logstashui/Common/tests/conversion_data/pipelines/test-multiline-ruby-with-hash.conf b/src/logstashui/Common/tests/conversion_data/pipelines/test-multiline-ruby-with-hash.conf deleted file mode 100644 index ee59e51..0000000 --- a/src/logstashui/Common/tests/conversion_data/pipelines/test-multiline-ruby-with-hash.conf +++ /dev/null @@ -1,49 +0,0 @@ -input { - stdin { - codec => line - } -} -filter { - ruby { - code => ' - require "digest" - msg = event.get("message").to_s - event.set("hash", Digest::MD5.hexdigest(msg)) - # this # is NOT a comment — it is inside a single-quoted string - if event.get("level") == "ERROR" - event.set("alert", true) - event.set("severity", "high") - elsif event.get("level") == "WARN" - event.set("severity", "medium") - else - event.set("severity", "low") - end - # another hash # mark inside the string — still not a comment - ' - } - ruby { - init => ' - require "openssl" - require "base64" - # init comment inside single-quoted string - @secret = ENV["SIGNING_SECRET"] || "default" - ' - code => ' - payload = event.get("message").to_s - sig = Base64.strict_encode64( - OpenSSL::HMAC.digest("SHA256", @secret, payload) - ) - event.set("signature", sig) - ' - } - mutate { - add_field => { - "pipeline" => "ruby-test" - } - } -} -output { - stdout { - codec => rubydebug - } -} diff --git a/src/logstashui/Common/tests/conversion_data/pipelines/test-nested-conditionals-comments.conf b/src/logstashui/Common/tests/conversion_data/pipelines/test-nested-conditionals-comments.conf deleted file mode 100644 index 207efc1..0000000 --- a/src/logstashui/Common/tests/conversion_data/pipelines/test-nested-conditionals-comments.conf +++ /dev/null @@ -1,71 +0,0 @@ -input { - stdin { - } -} -filter { - # Top-level comment before any conditional - if [type] == "web" { - # Comment inside first if branch - if [status] >= 500 { - # Comment inside nested if - mutate { - add_tag => ["server_error"] - add_field => { - "severity" => "high" - } - } - # Comment after plugin inside nested if - if [status] == 503 { - mutate { - add_tag => ["service_unavailable"] - } - } - # Trailing comment inside nested if - } - else if [status] >= 400 { - # Comment inside else-if - mutate { - add_tag => ["client_error"] - add_field => { - "severity" => "medium" - } - } - # Trailing comment inside else-if - } - else { - # Comment inside else - mutate { - add_tag => ["success"] - add_field => { - "severity" => "low" - } - } - } - # Comment at end of outer if block - } - else if [type] == "db" { - # Comment at start of else-if block - mutate { - add_field => { - "source" => "database" - } - } - } - else { - # Comment in final else - drop { - } - } - # Comment between conditional and next plugin at section level - mutate { - add_field => { - "processed_by" => "logstash" - } - } - # Trailing section-level comment -} -output { - stdout { - codec => rubydebug - } -} diff --git a/src/logstashui/Common/tests/conversion_data/pipelines/test-regex-conditions.conf b/src/logstashui/Common/tests/conversion_data/pipelines/test-regex-conditions.conf deleted file mode 100644 index b9e6306..0000000 --- a/src/logstashui/Common/tests/conversion_data/pipelines/test-regex-conditions.conf +++ /dev/null @@ -1,62 +0,0 @@ -input { - syslog { - port => 514 - } -} -filter { - if [message] =~ /^ERROR/ { - mutate { - add_tag => ["error"] - } - } - if [message] =~ /\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}/ { - date { - match => ["message", "ISO8601"] - target => "@timestamp" - } - } - if [host] =~ /^(web|app|db)-\d+\.example\.com$/ { - mutate { - add_field => { - "internal" => "true" - } - } - } - if [message] !~ /^$/ { - grok { - match => { - "message" => "%{GREEDYDATA:content}" - } - } - } - if [path] =~ /\/api\/v[12]\// and [method] == "POST" { - mutate { - add_tag => ["api_write"] - } - } - if [status] =~ /^5\d\d$/ { - mutate { - add_tag => ["server_error"] - } - } - else if [status] =~ /^4\d\d$/ { - mutate { - add_tag => ["client_error"] - } - } - if [user_agent] =~ /(?i)bot|crawler|spider/ { - drop { - } - } -} -output { - if "error" in [tags] { - file { - path => "/var/log/errors.log" - codec => json - } - } - stdout { - codec => rubydebug - } -} diff --git a/src/logstashui/Common/tests/conversion_data/pipelines/test-sample-nginx.conf b/src/logstashui/Common/tests/conversion_data/pipelines/test-sample-nginx.conf deleted file mode 100644 index 95ff7f9..0000000 --- a/src/logstashui/Common/tests/conversion_data/pipelines/test-sample-nginx.conf +++ /dev/null @@ -1,67 +0,0 @@ -input { - stdin { - codec => line - } -} -filter { - mutate { - add_field => { - "event.dataset" => "nginx.access" - "service.name" => "nginx" - } - } - grok { - match => { - "message" => [ - '%{IPORHOST:source.address} %{DATA:nginx.ident} %{DATA:user.name} \[%{HTTPDATE:nginx.access.time}\] "%{WORD:http.request.method} %{DATA:url.original} HTTP/%{NUMBER:http.version}" %{NUMBER:http.response.status_code:int} (?:%{NUMBER:http.response.body.bytes:int}|-) "%{DATA:http.request.referrer}" "%{DATA:user_agent.original}" %{NUMBER:nginx.access.request_time:float}', - '%{IPORHOST:source.address} %{DATA:nginx.ident} %{DATA:user.name} \[%{HTTPDATE:nginx.access.time}\] "%{WORD:http.request.method} %{DATA:url.original} HTTP/%{NUMBER:http.version}" %{NUMBER:http.response.status_code:int} (?:%{NUMBER:http.response.body.bytes:int}|-) "%{DATA:http.request.referrer}" "%{DATA:user_agent.original}"', - '%{IPORHOST:source.address} %{DATA:nginx.ident} %{DATA:user.name} \[%{HTTPDATE:nginx.access.time}\] "%{WORD:http.request.method} %{DATA:url.original} HTTP/%{NUMBER:http.version}" %{NUMBER:http.response.status_code:int} (?:%{NUMBER:http.response.body.bytes:int}|-) "%{DATA:http.request.referrer}" "%{DATA:user_agent.original}" (?:rt=%{NUMBER:nginx.access.request_time:float}\s+uct=%{NUMBER:nginx.access.upstream_connect_time:float}\s+uht=%{NUMBER:nginx.access.upstream_header_time:float}\s+urt=%{NUMBER:nginx.access.upstream_response_time:float})' - ] - } - tag_on_failure => ["_grok_nginx_access_fail"] - } - date { - match => ["nginx.access.time", "dd/MMM/yyyy:HH:mm:ss Z"] - target => "@timestamp" - } - urldecode { - field => "url.original" - } - dissect { - mapping => { - "url.original" => "%{url.path}?%{url.query}" - } - } - useragent { - source => "user_agent.original" - target => "user_agent" - } - mutate { - copy => { - "source.address" => "source.ip" - } - } - geoip { - source => "source.ip" - target => "source.geo" - tag_on_failure => ["_geoip_fail"] - } - mutate { - gsub => ["http.request.referrer", "^-$", "", "user.name", "^-$", ""] - } - if [http][response][status_code] and [http][response][status_code] >= 500 { - mutate { - add_tag => ["nginx_server_error"] - } - } - else if [http][response][status_code] and [http][response][status_code] >= 400 { - mutate { - add_tag => ["nginx_client_error"] - } - } -} -output { - stdout { - codec => rubydebug - } -} diff --git a/src/logstashui/Common/tests/conversion_data/pipelines/test-snmp-v0.2.conf b/src/logstashui/Common/tests/conversion_data/pipelines/test-snmp-v0.2.conf deleted file mode 100644 index 007f0b1..0000000 --- a/src/logstashui/Common/tests/conversion_data/pipelines/test-snmp-v0.2.conf +++ /dev/null @@ -1,213 +0,0 @@ -input { - snmp { - hosts => [ - { host => "udp:1.2.3.4/161" version => "3" timeout => 1000 retries => 2 } - ] - interval => "30" - security_name => "test" - security_level => "authPriv" - ecs_compatibility => "disabled" - oid_mapping_format => "dotted_string" - auth_protocol => "sha" - auth_pass => "test" - priv_protocol => "aes" - priv_pass => "test" - get => ["1.3.6.1.4.1.9.2.1.57.0", "1.3.6.1.4.1.9.9.48.1.1.1.5.1", "1.3.6.1.4.1.9.9.48.1.1.1.6.1", "1.3.6.1.2.1.1.1.0", "1.3.6.1.2.1.1.5.0", "1.3.6.1.2.1.1.2.0", "1.3.6.1.2.1.1.3.0"] - tables => [ - { name => "cdpCacheTable" columns => ["1.3.6.1.4.1.9.9.23.1.2.1.1.1", "1.3.6.1.4.1.9.9.23.1.2.1.1.6", "1.3.6.1.4.1.9.9.23.1.2.1.1.7", "1.3.6.1.4.1.9.9.23.1.2.1.1.8", "1.3.6.1.4.1.9.9.23.1.2.1.1.9", "1.3.6.1.4.1.9.9.23.1.2.1.1.5", "1.3.6.1.4.1.9.9.23.1.2.1.1.4"] }, - { name => "sensors" columns => ["1.3.6.1.4.1.9.9.13.1.3.1.2", "1.3.6.1.4.1.9.9.13.1.3.1.3", "1.3.6.1.4.1.9.9.13.1.3.1.4", "1.3.6.1.4.1.9.9.13.1.3.1.5", "1.3.6.1.4.1.9.9.13.1.3.1.6"] }, - { name => "fans" columns => ["1.3.6.1.4.1.9.9.13.1.4.1.2", "1.3.6.1.4.1.9.9.13.1.4.1.3"] }, - { name => "interfaces" columns => ["1.3.6.1.2.1.2.2.1.1", "1.3.6.1.2.1.2.2.1.2", "1.3.6.1.2.1.2.2.1.3", "1.3.6.1.2.1.2.2.1.7", "1.3.6.1.2.1.2.2.1.8", "1.3.6.1.2.1.31.1.1.1.1", "1.3.6.1.2.1.31.1.1.1.18", "1.3.6.1.2.1.31.1.1.1.15", "1.3.6.1.2.1.2.2.1.5", "1.3.6.1.2.1.2.2.1.6", "1.3.6.1.2.1.2.2.1.4", "1.3.6.1.2.1.31.1.1.1.6", "1.3.6.1.2.1.31.1.1.1.10", "1.3.6.1.2.1.31.1.1.1.9", "1.3.6.1.2.1.31.1.1.1.13", "1.3.6.1.2.1.31.1.1.1.7", "1.3.6.1.2.1.31.1.1.1.11", "1.3.6.1.2.1.31.1.1.1.8", "1.3.6.1.2.1.31.1.1.1.12", "1.3.6.1.2.1.2.2.1.9", "1.3.6.1.2.1.17.7.1.4.5.1.1", "1.3.6.1.2.1.2.2.1.14", "1.3.6.1.2.1.2.2.1.20", "1.3.6.1.2.1.2.2.1.13", "1.3.6.1.2.1.2.2.1.19"] } - ] - } -} -filter { - mutate { - rename => { - "host" => "[host][hostname]" - } - } - mutate { - rename => { - "1.3.6.1.4.1.9.2.1.57.0" => "[system][cpu][total][norm][pct]" - "1.3.6.1.4.1.9.9.48.1.1.1.5.1" => "[system][memory][actual][used][bytes]" - "1.3.6.1.4.1.9.9.48.1.1.1.6.1" => "[system][memory][actual][free][bytes]" - "1.3.6.1.2.1.1.1.0" => "[host][description]" - "1.3.6.1.2.1.1.5.0" => "[host][name]" - "1.3.6.1.2.1.1.2.0" => "[host][id]" - "1.3.6.1.2.1.1.3.0" => "[host][uptime]" - } - } - mutate { - add_field => { - "[network][name]" => "home-segment-1 (192.168.4.0/24)" - "[metricset][module]" => "system" - } - } - ruby { - code => ' v = event.get("[system][cpu][total][norm][pct]") - if v - event.set("[system][cpu][total][norm][pct]", v.to_f / 100.0) - end' - } - ruby { - code => ' - used = event.get("[system][memory][actual][used][bytes]") - free = event.get("[system][memory][actual][free][bytes]") - - if used && free - used_f = used.to_f - free_f = free.to_f - total_f = used_f + free_f - - if total_f > 0 - event.set("[system][memory][total]", total_f) - event.set("[system][memory][actual][used][pct]", (used_f / total_f)) - event.set("[system][memory][actual][free][pct]", (free_f / total_f)) - end - end - ' - } - ruby { - code => 'rows = event.get(\'[cdpCacheTable]\') - if rows.is_a?(Array) - host_name = event.get(\'[host][name]\') - host_hostname = event.get(\'[host][hostname]\') - network_name = event.get(\'[network][name]\') - timestamp = event.get(\'@timestamp\') - rows.each do |row| - next unless row.is_a?(Hash) - row[\'cdpCacheIfIndex\'] = row.delete(\'1.3.6.1.4.1.9.9.23.1.2.1.1.1\') - row[\'cdpCacheDeviceId\'] = row.delete(\'1.3.6.1.4.1.9.9.23.1.2.1.1.6\') - row[\'cdpCacheDevicePort\'] = row.delete(\'1.3.6.1.4.1.9.9.23.1.2.1.1.7\') - row[\'cdpCachePlatform\'] = row.delete(\'1.3.6.1.4.1.9.9.23.1.2.1.1.8\') - row[\'cdpCacheCapabilities\'] = row.delete(\'1.3.6.1.4.1.9.9.23.1.2.1.1.9\') - row[\'cdpCacheVersion\'] = row.delete(\'1.3.6.1.4.1.9.9.23.1.2.1.1.5\') - row[\'cdpCacheAddress\'] = row.delete(\'1.3.6.1.4.1.9.9.23.1.2.1.1.4\') - new_event = LogStash::Event.new({ - \'@timestamp\' => timestamp, - \'host\' => { \'name\' => host_name, \'hostname\' => host_hostname }, - \'network\' => { \'name\' => network_name }, - \'table\' => row, - \'metricset\' => { \'module\' => \'snmp\' }, - \'event\' => { \'kind\' => \'cdpcachetable\' } - }) - new_event_block.call(new_event) - end - event.remove(\'[cdpCacheTable]\') - event.set(\'[event][kind]\', \'metrics\') - end' - } - ruby { - code => 'rows = event.get(\'[sensors]\') - if rows.is_a?(Array) - host_name = event.get(\'[host][name]\') - host_hostname = event.get(\'[host][hostname]\') - network_name = event.get(\'[network][name]\') - timestamp = event.get(\'@timestamp\') - rows.each do |row| - next unless row.is_a?(Hash) - row[\'description\'] = row.delete(\'1.3.6.1.4.1.9.9.13.1.3.1.2\') - row[\'temp_celsius\'] = row.delete(\'1.3.6.1.4.1.9.9.13.1.3.1.3\') - row[\'temp_threshold\'] = row.delete(\'1.3.6.1.4.1.9.9.13.1.3.1.4\') - row[\'temp_last_shutdown\'] = row.delete(\'1.3.6.1.4.1.9.9.13.1.3.1.5\') - row[\'state\'] = row.delete(\'1.3.6.1.4.1.9.9.13.1.3.1.6\') - new_event = LogStash::Event.new({ - \'@timestamp\' => timestamp, - \'host\' => { \'name\' => host_name, \'hostname\' => host_hostname }, - \'network\' => { \'name\' => network_name }, - \'table\' => row, - \'metricset\' => { \'module\' => \'snmp\' }, - \'event\' => { \'kind\' => \'sensors\' } - }) - new_event_block.call(new_event) - end - event.remove(\'[sensors]\') - event.set(\'[event][kind]\', \'metrics\') - end' - } - ruby { - code => 'rows = event.get(\'[fans]\') - if rows.is_a?(Array) - host_name = event.get(\'[host][name]\') - host_hostname = event.get(\'[host][hostname]\') - network_name = event.get(\'[network][name]\') - timestamp = event.get(\'@timestamp\') - rows.each do |row| - next unless row.is_a?(Hash) - row[\'description\'] = row.delete(\'1.3.6.1.4.1.9.9.13.1.4.1.2\') - row[\'state\'] = row.delete(\'1.3.6.1.4.1.9.9.13.1.4.1.3\') - new_event = LogStash::Event.new({ - \'@timestamp\' => timestamp, - \'host\' => { \'name\' => host_name, \'hostname\' => host_hostname }, - \'network\' => { \'name\' => network_name }, - \'table\' => row, - \'metricset\' => { \'module\' => \'snmp\' }, - \'event\' => { \'kind\' => \'fans\' } - }) - new_event_block.call(new_event) - end - event.remove(\'[fans]\') - event.set(\'[event][kind]\', \'metrics\') - end' - } - ruby { - code => 'rows = event.get(\'[interfaces]\') - if rows.is_a?(Array) - host_name = event.get(\'[host][name]\') - host_hostname = event.get(\'[host][hostname]\') - network_name = event.get(\'[network][name]\') - timestamp = event.get(\'@timestamp\') - rows.each do |row| - next unless row.is_a?(Hash) - row[\'ifIndex\'] = row.delete(\'1.3.6.1.2.1.2.2.1.1\') - row[\'ifDescr\'] = row.delete(\'1.3.6.1.2.1.2.2.1.2\') - row[\'ifType\'] = row.delete(\'1.3.6.1.2.1.2.2.1.3\') - row[\'ifAdminStatus\'] = row.delete(\'1.3.6.1.2.1.2.2.1.7\') - row[\'ifOperStatus\'] = row.delete(\'1.3.6.1.2.1.2.2.1.8\') - row[\'ifName\'] = row.delete(\'1.3.6.1.2.1.31.1.1.1.1\') - row[\'ifAlias\'] = row.delete(\'1.3.6.1.2.1.31.1.1.1.18\') - row[\'ifHighSpeed\'] = row.delete(\'1.3.6.1.2.1.31.1.1.1.15\') - row[\'ifSpeed\'] = row.delete(\'1.3.6.1.2.1.2.2.1.5\') - row[\'ifPhysAddress\'] = row.delete(\'1.3.6.1.2.1.2.2.1.6\') - row[\'ifMtu\'] = row.delete(\'1.3.6.1.2.1.2.2.1.4\') - row[\'ifHCInOctets\'] = row.delete(\'1.3.6.1.2.1.31.1.1.1.6\') - row[\'ifHCOutOctets\'] = row.delete(\'1.3.6.1.2.1.31.1.1.1.10\') - row[\'ifHCInBroadcastPkts\'] = row.delete(\'1.3.6.1.2.1.31.1.1.1.9\') - row[\'ifHCOutBroadcastPkts\'] = row.delete(\'1.3.6.1.2.1.31.1.1.1.13\') - row[\'ifHCInUcastPkts\'] = row.delete(\'1.3.6.1.2.1.31.1.1.1.7\') - row[\'ifHCOutUcastPkts\'] = row.delete(\'1.3.6.1.2.1.31.1.1.1.11\') - row[\'ifHCInMulticastPkts\'] = row.delete(\'1.3.6.1.2.1.31.1.1.1.8\') - row[\'ifHCOutMulticastPkts\'] = row.delete(\'1.3.6.1.2.1.31.1.1.1.12\') - row[\'ifLastChange\'] = row.delete(\'1.3.6.1.2.1.2.2.1.9\') - row[\'dot1qPvid\'] = row.delete(\'1.3.6.1.2.1.17.7.1.4.5.1.1\') - row[\'ifInErrors\'] = row.delete(\'1.3.6.1.2.1.2.2.1.14\') - row[\'ifOutErrors\'] = row.delete(\'1.3.6.1.2.1.2.2.1.20\') - row[\'ifInDiscards\'] = row.delete(\'1.3.6.1.2.1.2.2.1.13\') - row[\'ifOutDiscards\'] = row.delete(\'1.3.6.1.2.1.2.2.1.19\') - new_event = LogStash::Event.new({ - \'@timestamp\' => timestamp, - \'host\' => { \'name\' => host_name, \'hostname\' => host_hostname }, - \'network\' => { \'name\' => network_name }, - \'table\' => row, - \'metricset\' => { \'module\' => \'snmp\' }, - \'event\' => { \'kind\' => \'interfaces\' } - }) - new_event_block.call(new_event) - end - event.remove(\'[interfaces]\') - event.set(\'[event][kind]\', \'metrics\') - end' - } -} -output { - elasticsearch { - data_stream => "true" - data_stream_type => "metrics" - data_stream_namespace => "default" - data_stream_dataset => "snmp.polling" - cloud_id => "test" - user => "test" - password => "test" - } -} diff --git a/src/logstashui/Common/tests/conversion_data/pipelines/test-string-escaping.conf b/src/logstashui/Common/tests/conversion_data/pipelines/test-string-escaping.conf deleted file mode 100644 index d3de6f5..0000000 --- a/src/logstashui/Common/tests/conversion_data/pipelines/test-string-escaping.conf +++ /dev/null @@ -1,33 +0,0 @@ -input { -} -filter { - mutate { - add_field => { - "double_quoted_with_hash" => "value # not a comment" - "with_brackets" => "data [in] brackets {and} braces" - "with_arrow" => "key => value pattern" - "env_ref" => "${MY_VAR}" - "sprintf_ref" => "prefix-%{field_name}" - } - } - grok { - match => { - "message" => '%{IP:client} \[%{HTTPDATE:ts}\] "%{WORD:method} %{URIPATHPARAM:path}' - } - pattern_definitions => { - "CUSTOM_IP" => "\b(?:\d{1,3}\.){3}\d{1,3}\b" - } - } - mutate { - rename => { - "@timestamp" => "event_time" - "host" => "source_host" - } - } -} -output { - file { - path => "/var/log/output/%{type}/%{+YYYY}/%{+MM}/%{+dd}.log" - codec => json - } -} diff --git a/src/logstashui/Common/tests/conversion_data/pipelines/test-twitter.conf b/src/logstashui/Common/tests/conversion_data/pipelines/test-twitter.conf deleted file mode 100644 index 13a8860..0000000 --- a/src/logstashui/Common/tests/conversion_data/pipelines/test-twitter.conf +++ /dev/null @@ -1,41 +0,0 @@ -input { - # This is the sample pipeline whose screenshots are used in - # the Pipeline Viewer documentation (../pipeline-viewer.asciidoc) - # - # Whenever the Pipeline Viewer UI changes, run this pipeline and - # open in the new UI to take updated screenshots. - # - # Note: you will have to setup the environment variables used - # below. Refer to the Twitter Logstash Input plugin documentation - # for their expected values - twitter { - id => "tweet harvester" - consumer_key => "${TWITTER_API_CONSUMER_KEY}" - consumer_secret => "${TWITTER_API_CONSUMER_SECRET}" - keywords => ["rain", "monsoon", "shower", "drizzle"] - oauth_token => "${TWITTER_API_OAUTH_TOKEN}" - oauth_token_secret => "${TWITTER_API_OAUTH_TOKEN_SECRET}" - } -} -filter { - grok { - match => { - "message" => "%{WORD:is_rt}" - } - } - if [is_rt] == "RT" { - drop { - id => "drop_all_RTs" - } - } -} -output { - stdout { - codec => dots - } - elasticsearch { - user => "elastic" - password => "changeme" - index => "tweets" - } -} diff --git a/src/logstashui/Common/tests/conversion_data/pipelines/test_complex1.conf b/src/logstashui/Common/tests/conversion_data/pipelines/test_complex1.conf deleted file mode 100644 index 18570e7..0000000 --- a/src/logstashui/Common/tests/conversion_data/pipelines/test_complex1.conf +++ /dev/null @@ -1,329 +0,0 @@ -input { - # - # "LogstashUI kitchen sink" pipeline - # Goal: be extremely feature-rich while staying within known-valid plugin options. - # - # Beats / Elastic Agent style shippers - beats { - id => "in_beats_5044" - port => "5044" - add_field => { - "ingest_transport" => "beats" - } - tags => ["from_beats"] - } - # JSON-over-TCP (common for app logs) - tcp { - id => "in_tcp_json_5514" - port => "5514" - mode => "server" - codec => json - add_field => { - "ingest_transport" => "tcp" - } - tags => ["from_tcp"] - } - # Syslog-ish UDP - udp { - id => "in_udp_5515" - port => "5515" - codec => plain - add_field => { - "ingest_transport" => "udp" - } - tags => ["from_udp"] - } - # HTTP event intake (webhooks, apps posting JSON, etc.) - http { - id => "in_http_8080" - port => "8080" - codec => json - add_field => { - "ingest_transport" => "http" - } - tags => ["from_http"] - } - # Local dev/testing input - stdin { - id => "in_stdin" - codec => line - add_field => { - "ingest_transport" => "stdin" - } - tags => ["from_stdin"] - } - # Synthetic test data (makes it easy to validate end-to-end quickly) - generator { - id => "in_generator" - lines => ["Feb 21 09:12:01 host1 sshd[123]: Failed password for invalid user admin from 10.1.2.3 port 51234 ssh2", "{\"@timestamp\":\"2026-02-21T14:12:02Z\",\"message\":\"GET /health 200\",\"source_ip\":\"8.8.8.8\",\"user_agent\":\"Mozilla/5.0\"}", "level=info service=api latency_ms=42 source_ip=192.168.1.50 msg=\"request completed\""] - count => "1" - add_field => { - "ingest_transport" => "generator" - } - tags => ["from_generator"] - } -} -filter { - # - # Normalize a few shared fields - # - mutate { - id => "f_mutate_bootstrap" - add_field => { - "[@metadata][pipeline]" => "logstashui_kitchen_sink" - "event.module" => "logstashui" - } - } - # Keep a canonical message field - if ![message] and [event][original] { - mutate { - id => "f_mutate_event_original_to_message" - copy => { - "[event][original]" => "message" - } - } - } - # - # Try to parse JSON *if* message looks like JSON (common when tcp/udp/plain feed JSON strings) - # - if [message] =~ "^[[:space:]]*\\{" { - json { - id => "f_json_from_message" - source => "message" - target => "json" - tag_on_failure => ["_jsonparsefailure_message"] - } - # If json parsed, promote a few expected keys (only if present) - if [json][@timestamp] { - mutate { - id => "f_promote_json_ts" - copy => { - "[json][@timestamp]" => "@timestamp" - } - } - } - if [json][source_ip] { - mutate { - id => "f_promote_json_source_ip" - copy => { - "[json][source_ip]" => "source_ip" - } - } - } - if [json][user_agent] { - mutate { - id => "f_promote_json_ua" - copy => { - "[json][user_agent]" => "user_agent" - } - } - } - } - # - # Syslog-ish parsing (UDP and some TCP) - # - if "from_udp" in [tags] or "from_tcp" in [tags] { - # Try dissect first (fast) and fall back to grok - dissect { - id => "f_dissect_syslogish" - mapping => { - "message" => "%{syslog_timestamp} %{syslog_host} %{syslog_program}[%{syslog_pid}]: %{syslog_message}" - } - tag_on_failure => ["_dissectfailure_syslogish"] - } - if "_dissectfailure_syslogish" in [tags] { - grok { - id => "f_grok_syslogish" - match => { - "message" => [ - "%{SYSLOGTIMESTAMP:syslog_timestamp} %{HOSTNAME:syslog_host} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" - ] - } - tag_on_failure => ["_grokparsefailure_syslogish"] - } - } - # If we extracted a syslog timestamp, use it - if [syslog_timestamp] { - date { - id => "f_date_syslog" - match => ["syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"] - tag_on_failure => ["_dateparsefailure_syslog"] - } - } - } - # - # key=value parsing for “flat” log lines - # - if [message] =~ "([A-Za-z0-9_.-]+)=([^\"]\\S+|\"[^\"]*\")" { - kv { - id => "f_kv_message" - source => "message" - trim_key => " " - trim_value => " " - value_split => "=" - field_split_pattern => "\s+" - tag_on_failure => ["_kvfailure_message"] - } - } - # - # Basic typing / normalization - # - mutate { - id => "f_mutate_normalize" - rename => { - "msg" => "message_short" - } - convert => { - "latency_ms" => "integer" - } - lowercase => ["level"] - } - # - # Enrichments: useragent, geoip, cidr, dns - # - if [user_agent] { - useragent { - id => "f_useragent" - source => "user_agent" - target => "user_agent_parsed" - } - } - # Canonicalize IP into source_ip if it exists elsewhere - if ![source_ip] and [source][ip] { - mutate { - id => "f_copy_source_ip" - copy => { - "[source][ip]" => "source_ip" - } - } - } - if [source_ip] { - # Tag private vs public - cidr { - id => "f_cidr_private" - address => ["%{source_ip}"] - network => ["10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"] - add_tag => ["src_private"] - } - # GeoIP typically only makes sense for public IPs, so do it only if not private-tagged - if "src_private" not in [tags] { - geoip { - id => "f_geoip" - source => "source_ip" - target => "source_geo" - } - } - # Reverse DNS lookup; replace source_ip with hostname when possible (or leave as-is) - dns { - id => "f_dns_reverse" - reverse => ["source_ip"] - action => "replace" - } - } - # - # Translate severity/level into a normalized numeric - # - translate { - id => "f_translate_level_to_severity" - source => "level" - target => "severity" - dictionary => { - "trace" => "0" - "debug" => "1" - "info" => "2" - "warn" => "3" - "error" => "4" - "fatal" => "5" - } - fallback => "2" - } - mutate { - id => "f_convert_severity_int" - convert => { - "severity" => "integer" - } - } - # - # Stable fingerprint for dedup / correlation - # - fingerprint { - id => "f_fingerprint_message" - source => ["message"] - method => "MURMUR3" - target => "[@metadata][fingerprint]" - } - # - # Example branching: treat auth-ish messages specially - # - if [syslog_program] == "sshd" or [message] =~ "(?i)failed password|authentication failure|invalid user" { - mutate { - id => "f_tag_auth" - add_tag => ["category_auth"] - add_field => { - "event.category" => "authentication" - } - } - } - else if [message] =~ "(?i)GET\\s+/health|/ready|/live" { - mutate { - id => "f_tag_health" - add_tag => ["category_healthcheck"] - add_field => { - "event.category" => "availability" - } - } - } - else { - mutate { - id => "f_tag_generic" - add_tag => ["category_generic"] - } - } - # - # Prune down noisy fields (keeps top-level essentials) - # - prune { - id => "f_prune" - whitelist_names => ["^@timestamp$", "^message$", "^message_short$", "^host$", "^source_ip$", "^source_geo$", "^severity$", "^level$", "^tags$", "^event\\..*$", "^user_agent.*$", "^syslog_.*$", "^ingest_transport$"] - } -} -output { - # Always see something in console during dev - stdout { - id => "out_stdout_rubydebug" - codec => rubydebug { - metadata => "true" - } - } - # Write to disk (great for debugging replay) - file { - id => "out_file_jsonl" - path => "/tmp/logstashui-%{+YYYY.MM.dd}.jsonl" - codec => json_lines - } - # Elasticsearch (local default) - elasticsearch { - id => "out_es_local" - hosts => ["http://localhost:9200"] - index => "logstashui-%{+YYYY.MM.dd}" - ilm_enabled => "false" - } - # Webhook back to your UI/API (example) - http { - id => "out_http_callback" - url => "http://localhost:9000/logstash/callback" - http_method => "post" - format => "json" - } - # Kafka (example) - kafka { - id => "out_kafka" - bootstrap_servers => "localhost:9092" - topic_id => "logstashui-events" - } - # Pipeline-to-pipeline (requires another pipeline with pipeline input address => "downstream") - pipeline { - id => "out_pipeline_downstream" - send_to => ["downstream"] - } -} diff --git a/src/logstashui/Common/tests/conversion_data/pipelines/test_elasticdocs-conditional.conf b/src/logstashui/Common/tests/conversion_data/pipelines/test_elasticdocs-conditional.conf deleted file mode 100644 index 7f58662..0000000 --- a/src/logstashui/Common/tests/conversion_data/pipelines/test_elasticdocs-conditional.conf +++ /dev/null @@ -1,44 +0,0 @@ -input { - file { - path => "/tmp/*_log" - } -} -filter { - if [path] =~ "access" { - mutate { - replace => { - "type" => "apache_access" - } - } - grok { - match => { - "message" => "%{COMBINEDAPACHELOG}" - } - } - date { - match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"] - } - } - else if [path] =~ "error" { - mutate { - replace => { - "type" => "apache_error" - } - } - } - else { - mutate { - replace => { - "type" => "random_logs" - } - } - } -} -output { - elasticsearch { - hosts => ["localhost:9200"] - } - stdout { - codec => rubydebug - } -} diff --git a/src/logstashui/Common/tests/conversion_data/pipelines/text-complex4.conf b/src/logstashui/Common/tests/conversion_data/pipelines/text-complex4.conf deleted file mode 100644 index 76b3692..0000000 --- a/src/logstashui/Common/tests/conversion_data/pipelines/text-complex4.conf +++ /dev/null @@ -1,123 +0,0 @@ -input { - generator { - id => "gen_edgeA" - count => 1 - lines => ["2026-02-22T01:23:45Z level=INFO service=api trace.id=abc123 method=GET path=\"/api/v2/items/42\" ip=8.8.8.8 ua=\"Mozilla/5.0 (Windows NT 10.0; Win64; x64)\" msg=\"hello\\world\""] - add_field => { - "[@metadata][source]" => "generator" - "event.original" => "%{message}" - } - } -} -filter { - # Fast split: timestamp + remainder - dissect { - id => "dissect_ts_rest" - mapping => { - "message" => "%{ts} %{rest}" - } - tag_on_failure => ["_dissectfailure_ts_rest"] - } - date { - id => "date_ts" - match => ["ts", "ISO8601"] - tag_on_failure => ["_dateparsefailure_ts"] - } - # Parse key=value in rest - kv { - id => "kv_rest" - source => "rest" - trim_key => " " - trim_value => " " - value_split => "=" - field_split_pattern => "\s+" - include_brackets => "false" - tag_on_failure => ["_kvfailure_rest"] - } - # Normalize: remove surrounding quotes on selected fields (common log format) - mutate { - id => "mutate_strip_quotes" - gsub => ["path", "^\"|\"$", "", "ua", "^\"|\"$", "", "msg", "^\"|\"$", ""] - } - # Promote a few fields into ECS-ish places - mutate { - id => "mutate_promote" - rename => { - "ip" => "[source][ip]" - "ua" => "[user_agent][original]" - "method" => "[http][request][method]" - "path" => "[url][path]" - "trace.id" => "[trace][id]" - } - lowercase => ["level"] - } - # Route key uses nested refs in sprintf (great UI test) - mutate { - id => "mutate_route_key" - add_field => { - "route_key" => "%{[@metadata][source]}::%{[service]}::%{[http][request][method]}::%{[url][path]}" - } - } - # Regex literals (escaped slashes) - if [url][path] =~ /^\/api\/v2\/items\/[0-9]+$/ { - mutate { - id => "tag_items" - add_tag => ["route_items"] - } - } - else if [url][path] =~ /^\/api\/v2\/[A-Za-z0-9._-]+$/ { - mutate { - id => "tag_api_generic" - add_tag => ["route_api_generic"] - } - } - else { - mutate { - id => "tag_other" - add_tag => ["route_other"] - } - } - # useragent parsing - if [user_agent][original] { - useragent { - id => "ua_parse" - source => "[user_agent][original]" - target => "[user_agent][parsed]" - } - } - # geoip on public source.ip - if [source][ip] { - geoip { - id => "geoip_source" - source => "[source][ip]" - target => "[source][geo]" - } - } - # fingerprint based on nested refs + message - fingerprint { - id => "fp_event" - source => ["route_key", "message"] - method => "MURMUR3" - target => "[@metadata][fp]" - } - # Replace literal backslash with slash in msg (escape-heavy but valid) - mutate { - id => "mutate_gsub_backslash" - gsub => ["msg", "\\\\", "/"] - } - prune { - id => "prune_edgeA" - whitelist_names => ["^@timestamp$", "^message$", "^tags$", "^level$", "^service$", "^route_key$", "^trace\\..*$", "^http\\..*$", "^url\\..*$", "^source\\..*$", "^user_agent\\..*$", "^msg$", "^@metadata\\..*$"] - } -} -output { - stdout { - codec => rubydebug { - metadata => "true" - } - } - file { - path => "/tmp/edgeA-%{+YYYY.MM.dd}.jsonl" - codec => json_lines - } -} diff --git a/src/logstashui/Common/tests/conversion_data/pipelines/text-complex5.conf b/src/logstashui/Common/tests/conversion_data/pipelines/text-complex5.conf deleted file mode 100644 index e407bbb..0000000 --- a/src/logstashui/Common/tests/conversion_data/pipelines/text-complex5.conf +++ /dev/null @@ -1,40 +0,0 @@ -input { - generator { - id => "gen_edgecase_3" - count => 1 - lines => ["path=\"C:\\Program Files\\App\\\" msg=\"quote:\" and backslash:\\\\\""] - } -} -filter { - # This ruby code contains both quote styles and backslashes. - ruby { - id => "ruby_edgecase_3" - code => ' - # Double quotes inside single-quoted LSCL string - event.set("[edge][note]", "He said: "hello"") - # Single quote inside Ruby string - event.set("[edge][apostrophe]", "it\'s fine") - # Trailing backslash in a field value (nasty for serializers) - event.set("[edge][trail]", "C: emp") - ' - } - # Parse key=val - kv { - id => "kv_edgecase_3" - source => "message" - value_split => "=" - field_split_pattern => "\s+" - } - # Replace literal backslash "\" with "/" - mutate { - id => "gsub_edgecase_3" - gsub => ["path", "\\\\", "/"] - } -} -output { - stdout { - codec => rubydebug { - metadata => "true" - } - } -} diff --git a/src/logstashui/Common/tests/conversion_data/pipelines/text-complex6.conf b/src/logstashui/Common/tests/conversion_data/pipelines/text-complex6.conf deleted file mode 100644 index bd67ec0..0000000 --- a/src/logstashui/Common/tests/conversion_data/pipelines/text-complex6.conf +++ /dev/null @@ -1,192 +0,0 @@ -input { - kafka { - id => "input_kafka_1" - topics => ["critical_business_events", "low_latency_metrics"] - bootstrap_servers => "kafka1:9092,kafka2:9092" - group_id => "logstash_critical_group" - codec => json_lines - type => "business_event" - tags => ["kafka_input", "critical"] - max_poll_records => "500" - } - redis { - id => "input_redis_1" - host => "redis-cache.example.com" - port => "6379" - data_type => "list" - key => "service_log_queue" - type => "service_log" - tags => ["redis_input", "service_data"] - codec => plain - } - tcp { - id => "input_tcp_1" - port => "9999" - type => "audit_log" - ssl_enable => "true" - ssl_cert => "/etc/logstash/certs/logstash.crt" - ssl_key => "/etc/logstash/certs/logstash.key" - ssl_verify => "true" - codec => json { - delimiter => ' - ' - } - tags => ["tcp_input", "sensitive"] - } -} -filter { - if "service_data" in [tags] { - json { - id => "filter_json_1" - source => "message" - target => "parsed_service_log" - remove_field => ["message"] - add_tag => ["json_attempt"] - } - if "_jsonparsefailure" in [tags] { - grok { - id => "filter_grok_1" - match => { - "message" => "(?%{TIMESTAMP_ISO8601}) %{DATA:service_id} \[%{LOGLEVEL:level}] %{NUMBER:req_id:int} - %{GREEDYDATA:log_msg}" - } - add_tag => ["grok_fallback_success"] - remove_tag => ["_jsonparsefailure"] - } - } - if [parsed_service_log][sensitive_data] == true or [tags] =~ /_grokparsefailure/ { - mutate { - id => "filter_mutate_1" - gsub => ["message", "(\\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Z|a-z]{2,}\\b)", "email_masked"] - } - } - mutate { - id => "filter_mutate_2" - rename => { - "[parsed_service_log][level]" => "log_level" - } - add_field => { - "correlation_id" => "%{[parsed_service_log][request_id]}" - } - } - } - if [correlation_id] and [log_level] { - aggregate { - id => "filter_aggregate_1" - task_id => "%{correlation_id}" - code => ' - if event.get(\'log_level\') == \'START\' - map[\'start_time\'] = event.get(\'@timestamp\').time.to_f - map[\'service\'] = event.get(\'service_id\') - event.cancel - elsif event.get(\'log_level\') == \'END\' and map[\'start_time\'] - end_time = event.get(\'@timestamp\').time.to_f - duration = (end_time - map[\'start_time\']) * 1000 # Duration in ms - event.set(\'request_duration_ms\', duration.round(3)) - event.set(\'service_name\', map[\'service\']) - event.set(\'type\', \'request_summary\') - end - ' - map_action => "create_or_update" - push_map_as_event_on_timeout => "true" - timeout => "60" - timeout_code => "event.set('error_reason', 'Unmatched_START_Event')" - timeout_task_id_field => "unmatched_correlation_id" - } - } - if "critical" in [tags] or [type] == "audit_log" { - translate { - id => "filter_translate_1" - field => "tenant_id" - destination => "tenant_name" - dictionary_path => "/etc/logstash/dicts/tenant_map.yml" - fallback => "Unknown_Tenant" - refresh_interval => "600" - } - mutate { - id => "filter_mutate_3" - convert => { - "transaction_amount" => "float" - } - remove_field => ["host", "port"] - } - date { - id => "filter_date_1" - match => ["[event_time]", "ISO8601", "UNIX_MS"] - target => "@timestamp" - remove_tag => ["_dateparsefailure"] - } - } - if ("_grokparsefailure" in [tags] or "_jsonparsefailure" in [tags]) and [type] != "audit_log" { - mutate { - id => "filter_mutate_4" - add_tag => ["dlq_candidate", "parsing_error"] - add_field => { - "dlq_reason" => "Parsing_Failed" - } - } - } - mutate { - id => "filter_mutate_5" - remove_tag => ["_jsonparsefailure", "_grokparsefailure", "_dateparsefailure"] - } -} -output { - if [tenant_name] =~ /^PRIORITY_/ { - elasticsearch { - id => "output_elasticsearch_1" - hosts => ["https://es-priority:9200"] - index => "tenant_priority-%{tenant_name}-%{+YYYY.MM}" - workers => "1" - manage_template => "false" - } - } - else if [tenant_name] { - elasticsearch { - id => "output_elasticsearch_2" - hosts => ["https://es-main:9200"] - index => "tenant_general-%{+YYYY.MM.dd}" - dlq_enabled => "true" - dlq_path => "/var/lib/logstash/dlq" - } - } - if "dlq_candidate" in [tags] { - file { - id => "output_file_1" - path => "/var/log/logstash/error_logs/dlq_parsing_failures.log" - codec => json_lines { - target => "original_event" - } - add_tag => ["s3_backup"] - } - } - if [type] == "audit_log" or [type] == "request_summary" or "s3_backup" in [tags] { - s3 { - id => "output_s3_1" - bucket => "logstash-archive-bucket" - region => "us-west-2" - time_file => "15" - size_file => "50" - codec => json_lines - temporary_directory => "/tmp/logstash_s3_tmp" - } - } - if "unmatched_correlation_id" in [tags] { - tcp { - id => "output_tcp_1" - host => "graylog-server.example.com" - port => "12201" - codec => gelf { - level => 1 - short_message => "Log Aggregation Timeout/Error: %{unmatched_correlation_id}" - } - } - } - if [log_level] =~ /(START|END|FATAL)/ { - stdout { - id => "output_stdout_1" - codec => rubydebug { - metadata => "true" - } - } - } -} diff --git a/src/logstashui/Common/tests/conversion_data/pipelines/text-complex7.conf b/src/logstashui/Common/tests/conversion_data/pipelines/text-complex7.conf deleted file mode 100644 index 18f018e..0000000 --- a/src/logstashui/Common/tests/conversion_data/pipelines/text-complex7.conf +++ /dev/null @@ -1,222 +0,0 @@ -input { - beats { - id => "input_beats_1" - port => "5044" - ssl => "true" - ssl_certificate => "/etc/logstash/certs/logstash.crt" - ssl_key => "/etc/logstash/certs/logstash.key" - codec => json - tags => ["beats_input", "app_log"] - } - udp { - id => "input_udp_1" - port => "5140" - buffer_size => "8192" - codec => plain { - charset => "UTF-8" - } - type => "network_flow" - tags => ["udp_input", "unstructured"] - } - jdbc { - id => "input_jdbc_1" - jdbc_driver_library => "/usr/share/logstash/logstash-core/lib/jars/postgresql-42.2.8.jar" - jdbc_driver_class => "org.postgresql.Driver" - jdbc_connection_string => "jdbc:postgresql://db.example.com:5432/config_db" - jdbc_user => "logstash_user" - jdbc_password => "${JDBC_PASSWORD}" - schedule => "0 * * * *" - statement => "SELECT id, user_name, config_item, change_timestamp FROM config_changes WHERE change_timestamp > :sql_last_value ORDER BY change_timestamp ASC" - use_column_value => "true" - tracking_column => "change_timestamp" - tracking_column_type => "timestamp" - last_run_metadata_path => "/var/lib/logstash/.jdbc_last_run_config_db" - type => "config_audit" - tags => ["jdbc_input", "audit"] - } -} -filter { - mutate { - id => "filter_mutate_1" - rename => { - "@timestamp" => "log_recv_time" - } - add_field => { - "severity" => "INFO" - } - } - if "app_log" in [tags] { - if "_jsonparsefailure" in [tags] { - grok { - id => "filter_grok_1" - match => { - "message" => "(?\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3}) \[%{DATA:thread}] %{LOGLEVEL:log_level} %{DATA:logger} - %{GREEDYDATA:log_message}" - } - add_tag => ["grok_fallback_success"] - remove_tag => ["_jsonparsefailure"] - } - } - if [log_level] { - mutate { - id => "filter_mutate_2" - uppercase => ["log_level"] - copy => { - "log_level" => "severity" - } - } - } - if [severity] =~ /(WARNING|ERROR|FATAL)/ { - geoip { - id => "filter_geoip_1" - source => "[fields][source_ip]" - target => "geo" - database => "/etc/logstash/geoip/GeoLite2-City.mmdb" - remove_field => ["continent_code", "location"] - } - translate { - id => "filter_translate_1" - field => "[service_code]" - destination => "service_name" - dictionary_path => "/etc/logstash/dictionaries/service_codes.csv" - fallback => "Unknown Service" - refresh_interval => "300" - } - } - mutate { - id => "filter_mutate_3" - remove_field => ["message", "agent"] - } - } - else if [type] == "network_flow" { - grok { - id => "filter_grok_2" - match => { - "message" => "%{NETFLOW_V9}" - } - on_failure => ["_netflowparsefailure"] - } - if "_netflowparsefailure" in [tags] { - mutate { - id => "filter_mutate_4" - add_tag => ["unparsed_flow"] - remove_tag => ["_grokparsefailure", "_netflowparsefailure"] - copy => { - "message" => "unparsed_data" - } - replace => { - "message" => "Truncated unparsed flow data." - } - } - } - else { - aggregate { - id => "filter_aggregate_1" - task_id => "%{source_ip}_%{destination_ip}_%{protocol}" - code => "map['total_packets'] ||= 0; map['total_packets'] += event.get('packets').to_i; map['total_bytes'] ||= 0; map['total_bytes'] += event.get('bytes').to_i" - map_action => "create_or_update" - push_map_as_event_on_timeout => "true" - timeout => "120" - timeout_task_id_field => "aggregated_flow_id" - timeout_tags => ["_aggregate_timeout"] - } - } - } - else if [type] == "config_audit" { - if [change_timestamp] { - date { - id => "filter_date_1" - match => ["change_timestamp", "YYYY-MM-dd HH:mm:ss.SSSSSS"] - target => "@timestamp" - remove_field => ["change_timestamp"] - } - if [user_name] != "system" { - ruby { - id => "filter_ruby_1" - code => "event.set('user_hash', Digest::MD5.hexdigest(event.get('user_name')))" - } - mutate { - id => "filter_mutate_5" - remove_field => ["user_name"] - } - } - } - } - if "_grokparsefailure" in [tags] or "_jsonparsefailure" in [tags] { - mutate { - id => "filter_mutate_6" - add_field => { - "log_status" => "FAILED_TO_PARSE" - } - } - } - else { - mutate { - id => "filter_mutate_7" - add_field => { - "log_status" => "PROCESSED" - } - } - } - if [log_recv_time] < now() - 86400000 { - drop { - id => "filter_drop_1" - } - } -} -output { - if [log_status] == "PROCESSED" and [severity] =~ /(ERROR|FATAL)/ { - elasticsearch { - id => "output_elasticsearch_1" - hosts => ["https://es-hot.example.com:9200"] - index => "high-priority-%{+YYYY.MM.dd}" - user => "logstash_writer" - password => "secure_password" - ssl => "true" - cacert => "/etc/logstash/certs/ca.crt" - action => "index" - } - } - else if [log_status] == "PROCESSED" { - elasticsearch { - id => "output_elasticsearch_2" - hosts => ["https://es-warm.example.com:9200"] - index => "general-logs-%{+YYYY.MM.dd}" - user => "logstash_writer" - password => "secure_password" - ssl => "true" - cacert => "/etc/logstash/certs/ca.crt" - workers => "4" - ilm_enabled => "false" - } - } - if [log_status] == "FAILED_TO_PARSE" { - file { - id => "output_file_1" - path => "/var/log/logstash/dlq_failures.json" - codec => json { - pretty => "true" - } - add_tag => ["dlq_routed"] - } - } - if "_aggregate_timeout" in [tags] { - tcp { - id => "output_tcp_1" - host => "alert-sys.example.com" - port => "6514" - codec => gelf { - protocol => "TCP" - short_message => "Aggregated Flow Timeout: %{aggregated_flow_id}" - } - socket_timeout => "5" - } - } - if rand(100) < 1 { - stdout { - id => "output_stdout_1" - codec => rubydebug { - metadata => "true" - } - } - } -} diff --git a/src/logstashui/Common/tests/conversion_data/pipelines/text-ls-repo-nginx-error.conf b/src/logstashui/Common/tests/conversion_data/pipelines/text-ls-repo-nginx-error.conf deleted file mode 100644 index 0d5f148..0000000 --- a/src/logstashui/Common/tests/conversion_data/pipelines/text-ls-repo-nginx-error.conf +++ /dev/null @@ -1,67 +0,0 @@ -input { - stdin { - codec => line - } -} -filter { - mutate { - add_field => { - "event.dataset" => "nginx.access" - "service.name" => "nginx" - } - } - grok { - match => { - "message" => [ - '%{IPORHOST:source.address} %{DATA:nginx.ident} %{DATA:user.name} \[%{HTTPDATE:nginx.access.time}\] "%{WORD:http.request.method} %{DATA:url.original} HTTP/%{NUMBER:http.version}" %{NUMBER:http.response.status_code:int} (?:%{NUMBER:http.response.body.bytes:int}|-) "%{DATA:http.request.referrer}" "%{DATA:user_agent.original}" %{NUMBER:nginx.access.request_time:float}', - '%{IPORHOST:source.address} %{DATA:nginx.ident} %{DATA:user.name} \[%{HTTPDATE:nginx.access.time}\] "%{WORD:http.request.method} %{DATA:url.original} HTTP/%{NUMBER:http.version}" %{NUMBER:http.response.status_code:int} (?:%{NUMBER:http.response.body.bytes:int}|-) "%{DATA:http.request.referrer}" "%{DATA:user_agent.original}"', - '%{IPORHOST:source.address} %{DATA:nginx.ident} %{DATA:user.name} \[%{HTTPDATE:nginx.access.time}\] "%{WORD:http.request.method} %{DATA:url.original} HTTP/%{NUMBER:http.version}" %{NUMBER:http.response.status_code:int} (?:%{NUMBER:http.response.body.bytes:int}|-) "%{DATA:http.request.referrer}" "%{DATA:user_agent.original}" (?:rt=%{NUMBER:nginx.access.request_time:float}\s+uct=%{NUMBER:nginx.access.upstream_connect_time:float}\s+uht=%{NUMBER:nginx.access.upstream_header_time:float}\s+urt=%{NUMBER:nginx.access.upstream_response_time:float})' - ] - } - tag_on_failure => ["_grok_nginx_access_fail"] - } - date { - match => ["nginx.access.time", "dd/MMM/yyyy:HH:mm:ss Z"] - target => "@timestamp" - } - urldecode { - field => "url.original" - } - dissect { - mapping => { - "url.original" => "%{url.path}?%{url.query}" - } - } - useragent { - source => "user_agent.original" - target => "user_agent" - } - mutate { - copy => { - "source.address" => "source.ip" - } - } - geoip { - source => "source.ip" - target => "source.geo" - tag_on_failure => ["_geoip_fail"] - } - mutate { - gsub => ["http.request.referrer", "^-$", "", "user.name", "^-$", ""] - } - if [http][response][status_code] >= 500 { - mutate { - add_tag => ["nginx_server_error"] - } - } - else if [http][response][status_code] >= 400 { - mutate { - add_tag => ["nginx_client_error"] - } - } -} -output { - stdout { - codec => rubydebug - } -} diff --git a/src/logstashui/Common/tests/test_components_to_pipeline.py b/src/logstashui/Common/tests/test_components_to_pipeline.py deleted file mode 100644 index 06422a0..0000000 --- a/src/logstashui/Common/tests/test_components_to_pipeline.py +++ /dev/null @@ -1,57 +0,0 @@ -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. - -from Common.logstash_config_parse import ComponentToPipeline, logstash_config_to_components -import pytest -import json -import os - -# Load test cases from external files -def load_test_cases(): - """Load test cases from conversion_data directory.""" - base_dir = os.path.dirname(os.path.abspath(__file__)) - pipelines_dir = os.path.join(base_dir, "conversion_data", "pipelines") - components_dir = os.path.join(base_dir, "conversion_data", "components") - - test_cases = [] - - # Get all .conf files - for filename in sorted(os.listdir(pipelines_dir)): - if filename.endswith('.conf'): - name = filename[:-5] # Remove .conf extension - - # Load pipeline config - pipeline_file = os.path.join(pipelines_dir, filename) - with open(pipeline_file, 'r', encoding='utf-8') as f: - pipeline = f.read() - - # Load components JSON - components_file = os.path.join(components_dir, f"{name}.json") - with open(components_file, 'r', encoding='utf-8') as f: - components = f.read() - - test_cases.append((name, pipeline, components)) - - return test_cases - -test_cases = load_test_cases() - - - -@pytest.mark.parametrize( - "name, pipeline, components", - test_cases, - ids=[case[0] for case in test_cases] -) -def test_components_to_config(name, pipeline, components): - """ - Test that ComponentToPipeline can generate a pipeline config from components. - Compares the original pipeline with the generated pipeline from stored components. - """ - # Load the stored components and convert to pipeline - parser = ComponentToPipeline(json.loads(components)) - generated_pipeline = parser.components_to_logstash_config() - - # Compare the original pipeline with the generated one - assert pipeline == generated_pipeline diff --git a/src/logstashui/Common/tests/test_context_processors.py b/src/logstashui/Common/tests/test_context_processors.py deleted file mode 100644 index 65b484b..0000000 --- a/src/logstashui/Common/tests/test_context_processors.py +++ /dev/null @@ -1,224 +0,0 @@ -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. - -import pytest -from django.test import RequestFactory -from unittest.mock import patch, Mock - -from Common.context_processors import ( - version_update_info, - navigation_highlight -) -from PipelineManager.models import Connection - - -@pytest.fixture -def request_factory(): - """Django RequestFactory for creating mock requests""" - return RequestFactory() - - -@pytest.fixture -def mock_request(request_factory): - """Create a basic mock request""" - return request_factory.get('/') - - -class TestVersionUpdateInfo: - """Test version_update_info context processor""" - - @patch('Common.context_processors.check_for_update') - def test_version_update_info_returns_context(self, mock_check_update, mock_request): - """Test that version_update_info returns correct context""" - mock_update_data = { - 'update_available': True, - 'latest_version': '2.0.0', - 'current_version': '1.0.0' - } - mock_check_update.return_value = mock_update_data - - context = version_update_info(mock_request) - - assert 'version_update' in context - assert context['version_update'] == mock_update_data - mock_check_update.assert_called_once() - - @patch('Common.context_processors.check_for_update') - def test_version_update_info_no_update(self, mock_check_update, mock_request): - """Test version_update_info when no update is available""" - mock_update_data = { - 'update_available': False, - 'latest_version': '1.0.0', - 'current_version': '1.0.0' - } - mock_check_update.return_value = mock_update_data - - context = version_update_info(mock_request) - - assert context['version_update']['update_available'] is False - - @patch('Common.context_processors.check_for_update') - def test_version_update_info_none_response(self, mock_check_update, mock_request): - """Test version_update_info when check_for_update returns None""" - mock_check_update.return_value = None - - context = version_update_info(mock_request) - - assert 'version_update' in context - assert context['version_update'] is None - - @patch('Common.context_processors.check_for_update') - def test_version_update_info_error_handling(self, mock_check_update, mock_request): - """Test version_update_info handles errors gracefully""" - mock_check_update.side_effect = Exception("Network error") - - # Should raise the exception (no error handling in the function) - with pytest.raises(Exception): - version_update_info(mock_request) - - -class TestNavigationHighlight: - """Test navigation_highlight context processor. - - highlight_snmp_devices was removed from the server-side context processor; - that logic now lives in client-side localStorage. The processor only tracks - whether any Connection exists and exposes: - - highlight_connection_manager (bool) - - has_connections (bool) - """ - - def test_no_connections_highlights_connection_manager(self, mock_request, db): - """Connection Manager is highlighted when no connections exist""" - Connection.objects.all().delete() - - context = navigation_highlight(mock_request) - - assert context['highlight_connection_manager'] is True - assert context['has_connections'] is False - - def test_connections_exist_does_not_highlight_connection_manager(self, mock_request, db): - """Connection Manager is NOT highlighted when at least one connection exists""" - Connection.objects.create( - name='Test Connection', - connection_type='CENTRALIZED', - host='https://localhost:9200', - username='elastic', - password='changeme', - port=None, - ) - - context = navigation_highlight(mock_request) - - assert context['highlight_connection_manager'] is False - assert context['has_connections'] is True - - def test_multiple_connections(self, mock_request, db): - """has_connections is True when multiple connections exist""" - Connection.objects.create( - name='Connection 1', - connection_type='CENTRALIZED', - host='https://localhost:9200', - username='elastic', - password='changeme', - port=None, - ) - Connection.objects.create( - name='Connection 2', - connection_type='CENTRALIZED', - cloud_id='test-id', - api_key='test-api-key', - ) - - context = navigation_highlight(mock_request) - - assert context['highlight_connection_manager'] is False - assert context['has_connections'] is True - - def test_context_keys_always_present(self, mock_request, db): - """highlight_connection_manager and has_connections are always in context""" - Connection.objects.all().delete() - - context = navigation_highlight(mock_request) - - assert 'highlight_connection_manager' in context - assert 'has_connections' in context - assert isinstance(context['highlight_connection_manager'], bool) - assert isinstance(context['has_connections'], bool) - - def test_navigation_highlight_with_different_request_types(self, request_factory, db): - """navigation_highlight works regardless of HTTP method""" - Connection.objects.all().delete() - - for method in ('get', 'post', 'put'): - request = getattr(request_factory, method)('/test/') - context = navigation_highlight(request) - assert context['highlight_connection_manager'] is True - - def test_navigation_highlight_database_queries(self, mock_request, db): - """navigation_highlight queries Connection.objects.exists() exactly once""" - Connection.objects.all().delete() - - with patch.object(Connection.objects, 'exists', return_value=False) as mock_conn_exists: - context = navigation_highlight(mock_request) - - mock_conn_exists.assert_called_once() - assert context['highlight_connection_manager'] is True - assert context['has_connections'] is False - - def test_navigation_highlight_logic_flow(self, mock_request, db): - """Complete logic: no connections → highlight; connection added → no highlight""" - Connection.objects.all().delete() - - # State 1: No connections - context = navigation_highlight(mock_request) - assert context == { - 'highlight_connection_manager': True, - 'has_connections': False, - } - - # State 2: Connection added - Connection.objects.create( - name='Test', - connection_type='CENTRALIZED', - host='https://localhost:9200', - username='elastic', - password='changeme', - port=None, - ) - context = navigation_highlight(mock_request) - assert context == { - 'highlight_connection_manager': False, - 'has_connections': True, - } - - -class TestContextProcessorsIntegration: - """Integration tests for context processors""" - - @patch('Common.context_processors.check_for_update') - def test_both_context_processors_together(self, mock_check_update, mock_request, db): - """Both context processors can be used together without key collisions""" - mock_check_update.return_value = {'update_available': True} - Connection.objects.all().delete() - - version_context = version_update_info(mock_request) - navigation_context = navigation_highlight(mock_request) - - combined_context = {**version_context, **navigation_context} - - assert 'version_update' in combined_context - assert 'highlight_connection_manager' in combined_context - assert 'has_connections' in combined_context - # version_update + highlight_connection_manager + has_connections - assert len(combined_context) == 3 - - def test_context_processors_dont_interfere(self, mock_request, db): - """Context processors return disjoint key sets""" - with patch('Common.context_processors.check_for_update') as mock_check: - mock_check.return_value = {'test': 'data'} - - version_context = version_update_info(mock_request) - navigation_context = navigation_highlight(mock_request) - - assert set(version_context.keys()).isdisjoint(set(navigation_context.keys())) diff --git a/src/logstashui/Common/tests/test_decorators.py b/src/logstashui/Common/tests/test_decorators.py deleted file mode 100644 index 00bd5d8..0000000 --- a/src/logstashui/Common/tests/test_decorators.py +++ /dev/null @@ -1,269 +0,0 @@ -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. - -import pytest -from django.contrib.auth.models import User -from django.http import HttpRequest, HttpResponse -from django.test import RequestFactory -from unittest.mock import Mock - -from Common.decorators import require_admin_role -from Management.models import UserProfile - - -@pytest.fixture -def request_factory(): - """Django RequestFactory for creating mock requests""" - return RequestFactory() - - -@pytest.fixture -def admin_user(db): - """Create a user with admin profile""" - user = User.objects.create_user( - username='admin_user', - password='testpass123', - email='admin@example.com' - ) - # Signal creates profile automatically, just ensure it's admin - profile, created = UserProfile.objects.get_or_create(user=user, defaults={'role': 'admin'}) - if not created: - profile.role = 'admin' - profile.save() - return user - - -@pytest.fixture -def readonly_user(db): - """Create a user with readonly profile""" - user = User.objects.create_user( - username='readonly_user', - password='testpass123', - email='readonly@example.com' - ) - # Signal creates profile automatically, update to readonly - profile = UserProfile.objects.get(user=user) - profile.role = 'readonly' - profile.save() - # Refresh user from database to get updated profile relationship - user.refresh_from_db() - return user - - -@pytest.fixture -def user_without_profile(db): - """Create a user without a profile (edge case for bug 2a)""" - from django.db.models.signals import post_save - from Management.models import create_user_profile - - # Temporarily disconnect the signal to prevent auto-creation - post_save.disconnect(create_user_profile, sender=User) - - try: - user = User.objects.create_user( - username='no_profile_user', - password='testpass123', - email='noprofile@example.com' - ) - # Explicitly ensure no profile exists - UserProfile.objects.filter(user=user).delete() - finally: - # Reconnect the signal - post_save.connect(create_user_profile, sender=User) - - return user - - -@pytest.fixture -def mock_view(): - """Create a mock view function""" - def view_func(request, *args, **kwargs): - return HttpResponse("Success", status=200) - view_func.__name__ = "mock_view_function" - return view_func - - -class TestRequireAdminRoleDecorator: - """Test require_admin_role decorator""" - - def test_unauthenticated_request_denied(self, request_factory, mock_view): - """Test that unauthenticated requests are denied""" - request = request_factory.get('/test/') - request.user = Mock() - request.user.is_authenticated = False - - decorated_view = require_admin_role(mock_view) - response = decorated_view(request) - - assert response.status_code == 403 - assert b'You must be logged in to perform this action' in response.content - assert 'HX-Trigger' in response - assert 'showToastEvent' in response['HX-Trigger'] - - def test_admin_user_allowed(self, request_factory, mock_view, admin_user): - """Test that admin users are allowed access""" - request = request_factory.get('/test/') - request.user = admin_user - - decorated_view = require_admin_role(mock_view) - response = decorated_view(request) - - assert response.status_code == 200 - assert b'Success' in response.content - - def test_readonly_user_denied(self, request_factory, mock_view, readonly_user): - """Test that readonly users are denied access""" - request = request_factory.get('/test/') - request.user = readonly_user - - decorated_view = require_admin_role(mock_view) - response = decorated_view(request) - - assert response.status_code == 403 - assert b'Access denied: Admin role required' in response.content - assert 'HX-Trigger' in response - assert 'showToastEvent' in response['HX-Trigger'] - - def test_user_without_profile_denied(self, request_factory, mock_view, user_without_profile): - """ - CRITICAL TEST for bug 2a: Test that users without profiles are denied access. - This is the missing-profile edge case that was a security vulnerability. - """ - request = request_factory.get('/test/') - request.user = user_without_profile - - # Verify user has no profile - assert not hasattr(user_without_profile, 'profile') or not UserProfile.objects.filter(user=user_without_profile).exists() - - decorated_view = require_admin_role(mock_view) - response = decorated_view(request) - - # User should be DENIED, not allowed - assert response.status_code == 403 - assert b'Access denied: Admin role required' in response.content - assert 'HX-Trigger' in response - assert 'showToastEvent' in response['HX-Trigger'] - - def test_superuser_without_profile_denied(self, request_factory, mock_view, db): - """ - Test that even superusers without profiles are denied. - This simulates a superuser created via createsuperuser before signal fires. - """ - from django.db.models.signals import post_save - from Management.models import create_user_profile - - # Temporarily disconnect the signal - post_save.disconnect(create_user_profile, sender=User) - - try: - superuser = User.objects.create_superuser( - username='superuser', - password='testpass123', - email='super@example.com' - ) - # Ensure no profile exists - UserProfile.objects.filter(user=superuser).delete() - finally: - # Reconnect the signal - post_save.connect(create_user_profile, sender=User) - - request = request_factory.get('/test/') - request.user = superuser - - decorated_view = require_admin_role(mock_view) - response = decorated_view(request) - - # Even superuser should be denied without profile - assert response.status_code == 403 - assert b'Access denied: Admin role required' in response.content - - def test_decorator_preserves_view_metadata(self, mock_view): - """Test that decorator preserves original view function metadata""" - decorated_view = require_admin_role(mock_view) - - # functools.wraps should preserve __name__ - assert decorated_view.__name__ == mock_view.__name__ - - def test_decorator_with_view_args_and_kwargs(self, request_factory, admin_user): - """Test that decorator properly passes args and kwargs to view""" - def view_with_args(request, arg1, arg2, kwarg1=None): - return HttpResponse(f"{arg1}-{arg2}-{kwarg1}", status=200) - - view_with_args.__name__ = "view_with_args" - - request = request_factory.get('/test/') - request.user = admin_user - - decorated_view = require_admin_role(view_with_args) - response = decorated_view(request, "val1", "val2", kwarg1="val3") - - assert response.status_code == 200 - assert b'val1-val2-val3' in response.content - - def test_logging_for_readonly_user(self, request_factory, mock_view, readonly_user, caplog): - """Test that readonly user access attempts are logged""" - request = request_factory.get('/test/') - request.user = readonly_user - - decorated_view = require_admin_role(mock_view) - response = decorated_view(request) - - # Check that warning was logged with role information - assert "readonly_user" in caplog.text - assert "'readonly'" in caplog.text - assert "mock_view_function" in caplog.text - - def test_logging_for_user_without_profile(self, request_factory, mock_view, user_without_profile, caplog): - """Test that users without profiles have 'no profile' logged""" - request = request_factory.get('/test/') - request.user = user_without_profile - - decorated_view = require_admin_role(mock_view) - response = decorated_view(request) - - # Check that warning was logged with 'no profile' information - assert "no_profile_user" in caplog.text - assert "no profile" in caplog.text - assert "mock_view_function" in caplog.text - - def test_htmx_trigger_header_format(self, request_factory, mock_view, readonly_user): - """Test that HX-Trigger header is properly formatted JSON""" - import json - - request = request_factory.get('/test/') - request.user = readonly_user - - decorated_view = require_admin_role(mock_view) - response = decorated_view(request) - - # Verify HX-Trigger is valid JSON - trigger_data = json.loads(response['HX-Trigger']) - assert 'showToastEvent' in trigger_data - assert trigger_data['showToastEvent']['type'] == 'error' - assert 'Admin role required' in trigger_data['showToastEvent']['message'] - - def test_multiple_decorators_stacking(self, request_factory, admin_user): - """Test that decorator can be stacked with other decorators""" - def another_decorator(view_func): - def wrapper(request, *args, **kwargs): - response = view_func(request, *args, **kwargs) - response['X-Custom-Header'] = 'test' - return response - wrapper.__name__ = view_func.__name__ - return wrapper - - def view_func(request): - return HttpResponse("Success", status=200) - view_func.__name__ = "stacked_view" - - # Stack decorators - decorated_view = require_admin_role(another_decorator(view_func)) - - request = request_factory.get('/test/') - request.user = admin_user - - response = decorated_view(request) - - assert response.status_code == 200 - assert 'X-Custom-Header' in response diff --git a/src/logstashui/Common/tests/test_elastic_utils.py b/src/logstashui/Common/tests/test_elastic_utils.py deleted file mode 100644 index c79af6d..0000000 --- a/src/logstashui/Common/tests/test_elastic_utils.py +++ /dev/null @@ -1,654 +0,0 @@ -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. - -import pytest -from unittest.mock import Mock, patch, MagicMock -from elasticsearch import Elasticsearch - -from Common.elastic_utils import ( - test_elastic_connectivity as es_test_connectivity, - get_elastic_connections_from_list, - get_elastic_connection, - _get_creds, - get_elasticsearch_indices, - get_elasticsearch_field_mappings, - _extract_field_names, - query_elasticsearch_documents, - normalize_kibana_url, -) -from PipelineManager.models import Connection - - -@pytest.fixture -def mock_connection(db): - """Create a mock connection for testing""" - connection = Connection.objects.create( - name='Test Connection', - connection_type='CENTRALIZED', - host='https://localhost:9200', - username='elastic', - password='changeme', - port=None, # host already contains the port; avoid double-appending - ) - return connection - - -@pytest.fixture -def mock_cloud_connection(db): - """Create a mock cloud connection for testing""" - connection = Connection.objects.create( - name='Cloud Connection', - connection_type='CENTRALIZED', - cloud_id='test-cloud-id:dGVzdA==', - api_key='test-api-key' - ) - return connection - - -class TestGetCreds: - """Test _get_creds function""" - - def test_get_creds_with_host_and_password(self, mock_connection): - """Test getting credentials with host and password auth""" - creds = _get_creds(mock_connection.id) - - assert 'hosts' in creds - assert creds['hosts'] == 'https://localhost:9200' - assert 'http_auth' in creds - assert creds['http_auth'][0] == 'elastic' - - def test_get_creds_with_cloud_id_and_api_key(self, mock_cloud_connection): - """Test getting credentials with cloud_id and api_key""" - creds = _get_creds(mock_cloud_connection.id) - - assert 'cloud_id' in creds - assert creds['cloud_id'] == 'test-cloud-id:dGVzdA==' - assert 'api_key' in creds - assert 'http_auth' not in creds - assert 'hosts' not in creds - - -class TestGetElasticConnection: - """Test get_elastic_connection function""" - - @patch('Common.elastic_utils.Elasticsearch') - def test_get_elastic_connection(self, mock_es_class, mock_connection): - """Test getting Elasticsearch connection""" - mock_es_instance = Mock() - mock_es_class.return_value = mock_es_instance - - result = get_elastic_connection(mock_connection.id) - - assert result == mock_es_instance - mock_es_class.assert_called_once() - call_kwargs = mock_es_class.call_args[1] - assert 'hosts' in call_kwargs or 'cloud_id' in call_kwargs - - -class TestGetElasticConnectionsFromList: - """Test get_elastic_connections_from_list function""" - - @patch('Common.elastic_utils.get_elastic_connection') - def test_get_connections_from_list(self, mock_get_connection, mock_connection): - """Test getting list of connections""" - mock_es = Mock() - mock_get_connection.return_value = mock_es - - connections = get_elastic_connections_from_list() - - assert len(connections) == 1 - assert connections[0]['name'] == 'Test Connection' - assert connections[0]['es'] == mock_es - assert connections[0]['id'] == mock_connection.id - assert connections[0]['connection_type'] == 'CENTRALIZED' - - @patch('Common.elastic_utils.get_elastic_connection') - def test_get_multiple_connections(self, mock_get_connection, db): - """Test getting multiple connections""" - Connection.objects.create( - name='Connection 1', - connection_type='CENTRALIZED', - host='https://localhost:9200', - username='elastic', - password='changeme' - ) - Connection.objects.create( - name='Connection 2', - connection_type='CENTRALIZED', - cloud_id='test-id', - api_key='test-api-key' - ) - - mock_get_connection.return_value = Mock() - - connections = get_elastic_connections_from_list() - - assert len(connections) == 2 - assert connections[0]['name'] == 'Connection 1' - assert connections[1]['name'] == 'Connection 2' - - -class TestGetElasticsearchIndices: - """Test get_elasticsearch_indices function""" - - @patch('Common.elastic_utils.get_elastic_connection') - def test_get_indices_default_pattern(self, mock_get_connection, mock_connection): - """Test getting indices with default pattern""" - mock_es = Mock() - mock_es.cat.indices.return_value = [ - {'index': 'index-1'}, - {'index': 'index-2'}, - {'index': 'index-3'} - ] - mock_get_connection.return_value = mock_es - - indices = get_elasticsearch_indices(mock_connection.id) - - assert len(indices) == 3 - assert 'index-1' in indices - assert 'index-2' in indices - assert 'index-3' in indices - mock_es.cat.indices.assert_called_once_with(index='*', format='json', h='index') - - @patch('Common.elastic_utils.get_elastic_connection') - def test_get_indices_custom_pattern(self, mock_get_connection, mock_connection): - """Test getting indices with custom pattern""" - mock_es = Mock() - mock_es.cat.indices.return_value = [ - {'index': 'logs-2024-01'}, - {'index': 'logs-2024-02'} - ] - mock_get_connection.return_value = mock_es - - indices = get_elasticsearch_indices(mock_connection.id, pattern='logs-*') - - assert len(indices) == 2 - mock_es.cat.indices.assert_called_once_with(index='logs-*', format='json', h='index') - - @patch('Common.elastic_utils.get_elastic_connection') - def test_get_indices_sorted(self, mock_get_connection, mock_connection): - """Test that indices are returned sorted""" - mock_es = Mock() - mock_es.cat.indices.return_value = [ - {'index': 'zebra'}, - {'index': 'alpha'}, - {'index': 'beta'} - ] - mock_get_connection.return_value = mock_es - - indices = get_elasticsearch_indices(mock_connection.id) - - assert indices == ['alpha', 'beta', 'zebra'] - - @patch('Common.elastic_utils.get_elastic_connection') - def test_get_indices_limited_to_50(self, mock_get_connection, mock_connection): - """Test that only top 50 indices are returned""" - mock_es = Mock() - mock_es.cat.indices.return_value = [ - {'index': f'index-{i:03d}'} for i in range(100) - ] - mock_get_connection.return_value = mock_es - - indices = get_elasticsearch_indices(mock_connection.id) - - assert len(indices) == 50 - - @patch('Common.elastic_utils.get_elastic_connection') - def test_get_indices_error_handling(self, mock_get_connection, mock_connection): - """Test error handling when fetching indices fails""" - mock_es = Mock() - mock_es.cat.indices.side_effect = Exception("Connection error") - mock_get_connection.return_value = mock_es - - indices = get_elasticsearch_indices(mock_connection.id) - - assert indices == [] - - -class TestGetElasticsearchFieldMappings: - """Test get_elasticsearch_field_mappings function""" - - @patch('Common.elastic_utils.get_elastic_connection') - def test_get_field_mappings(self, mock_get_connection, mock_connection): - """Test getting field mappings from index""" - mock_es = Mock() - mock_es.indices.get_mapping.return_value = { - 'test-index': { - 'mappings': { - 'properties': { - 'field1': {'type': 'text'}, - 'field2': {'type': 'keyword'}, - 'nested_field': { - 'properties': { - 'subfield1': {'type': 'long'} - } - } - } - } - } - } - mock_get_connection.return_value = mock_es - - fields = get_elasticsearch_field_mappings(mock_connection.id, 'test-index') - - assert 'field1' in fields - assert 'field2' in fields - assert 'nested_field' in fields - assert 'nested_field.subfield1' in fields - assert len(fields) == 4 - - @patch('Common.elastic_utils.get_elastic_connection') - def test_get_field_mappings_sorted(self, mock_get_connection, mock_connection): - """Test that field mappings are sorted""" - mock_es = Mock() - mock_es.indices.get_mapping.return_value = { - 'test-index': { - 'mappings': { - 'properties': { - 'zebra': {'type': 'text'}, - 'alpha': {'type': 'keyword'}, - 'beta': {'type': 'long'} - } - } - } - } - mock_get_connection.return_value = mock_es - - fields = get_elasticsearch_field_mappings(mock_connection.id, 'test-index') - - assert fields == ['alpha', 'beta', 'zebra'] - - @patch('Common.elastic_utils.get_elastic_connection') - def test_get_field_mappings_deduplication(self, mock_get_connection, mock_connection): - """Test that duplicate fields are removed""" - mock_es = Mock() - mock_es.indices.get_mapping.return_value = { - 'test-index-1': { - 'mappings': { - 'properties': { - 'field1': {'type': 'text'} - } - } - }, - 'test-index-2': { - 'mappings': { - 'properties': { - 'field1': {'type': 'text'} - } - } - } - } - mock_get_connection.return_value = mock_es - - fields = get_elasticsearch_field_mappings(mock_connection.id, 'test-index-*') - - assert fields.count('field1') == 1 - - @patch('Common.elastic_utils.get_elastic_connection') - def test_get_field_mappings_error_handling(self, mock_get_connection, mock_connection): - """Test error handling when fetching mappings fails""" - mock_es = Mock() - mock_es.indices.get_mapping.side_effect = Exception("Index not found") - mock_get_connection.return_value = mock_es - - fields = get_elasticsearch_field_mappings(mock_connection.id, 'nonexistent-index') - - assert fields == [] - - -class TestExtractFieldNames: - """Test _extract_field_names function""" - - def test_extract_simple_fields(self): - """Test extracting simple field names""" - properties = { - 'field1': {'type': 'text'}, - 'field2': {'type': 'keyword'} - } - - fields = _extract_field_names(properties) - - assert 'field1' in fields - assert 'field2' in fields - assert len(fields) == 2 - - def test_extract_nested_fields(self): - """Test extracting nested field names""" - properties = { - 'parent': { - 'properties': { - 'child1': {'type': 'text'}, - 'child2': {'type': 'keyword'} - } - } - } - - fields = _extract_field_names(properties) - - assert 'parent' in fields - assert 'parent.child1' in fields - assert 'parent.child2' in fields - assert len(fields) == 3 - - def test_extract_deeply_nested_fields(self): - """Test extracting deeply nested field names""" - properties = { - 'level1': { - 'properties': { - 'level2': { - 'properties': { - 'level3': {'type': 'text'} - } - } - } - } - } - - fields = _extract_field_names(properties) - - assert 'level1' in fields - assert 'level1.level2' in fields - assert 'level1.level2.level3' in fields - - def test_extract_with_prefix(self): - """Test extracting field names with prefix""" - properties = { - 'field1': {'type': 'text'} - } - - fields = _extract_field_names(properties, prefix='parent') - - assert 'parent.field1' in fields - - def test_extract_empty_properties(self): - """Test extracting from empty properties""" - fields = _extract_field_names({}) - - assert fields == [] - - -class TestQueryElasticsearchDocuments: - """Test query_elasticsearch_documents function""" - - @patch('Common.elastic_utils.get_elastic_connection') - def test_query_by_document_ids(self, mock_get_connection, mock_connection): - """Test querying documents by IDs""" - mock_es = Mock() - mock_es.mget.return_value = { - 'docs': [ - {'_source': {'field': 'value1'}, 'found': True}, - {'_source': {'field': 'value2'}, 'found': True} - ] - } - mock_get_connection.return_value = mock_es - - docs = query_elasticsearch_documents( - mock_connection.id, - 'test-index', - doc_ids=['id1', 'id2'] - ) - - assert len(docs) == 2 - assert docs[0] == {'field': 'value1'} - assert docs[1] == {'field': 'value2'} - mock_es.mget.assert_called_once_with(index='test-index', ids=['id1', 'id2']) - - @patch('Common.elastic_utils.get_elastic_connection') - def test_query_by_document_ids_not_found(self, mock_get_connection, mock_connection): - """Test querying documents by IDs with some not found""" - mock_es = Mock() - mock_es.mget.return_value = { - 'docs': [ - {'_source': {'field': 'value1'}, 'found': True}, - {'found': False} - ] - } - mock_get_connection.return_value = mock_es - - docs = query_elasticsearch_documents( - mock_connection.id, - 'test-index', - doc_ids=['id1', 'id2'] - ) - - assert len(docs) == 1 - assert docs[0] == {'field': 'value1'} - - @patch('Common.elastic_utils.get_elastic_connection') - def test_query_by_field(self, mock_get_connection, mock_connection): - """Test querying documents by field""" - mock_es = Mock() - mock_es.search.return_value = { - 'hits': { - 'hits': [ - {'_source': {'field1': 'value1'}}, - {'_source': {'field1': 'value2'}} - ] - } - } - mock_get_connection.return_value = mock_es - - docs = query_elasticsearch_documents( - mock_connection.id, - 'test-index', - field='field1', - size=10 - ) - - assert len(docs) == 2 - mock_es.search.assert_called_once() - - @patch('Common.elastic_utils.get_elastic_connection') - def test_query_with_query_string(self, mock_get_connection, mock_connection): - """Test querying documents with query string""" - mock_es = Mock() - mock_es.search.return_value = { - 'hits': { - 'hits': [ - {'_source': {'field': 'value'}} - ] - } - } - mock_get_connection.return_value = mock_es - - docs = query_elasticsearch_documents( - mock_connection.id, - 'test-index', - query_string='field:value', - size=5 - ) - - assert len(docs) == 1 - call_kwargs = mock_es.search.call_args[1] - assert call_kwargs['query']['query_string']['query'] == 'field:value' - - @patch('Common.elastic_utils.get_elastic_connection') - def test_query_match_all(self, mock_get_connection, mock_connection): - """Test querying documents with match_all""" - mock_es = Mock() - mock_es.search.return_value = { - 'hits': { - 'hits': [ - {'_source': {'field': 'value'}} - ] - } - } - mock_get_connection.return_value = mock_es - - docs = query_elasticsearch_documents( - mock_connection.id, - 'test-index', - size=10 - ) - - call_kwargs = mock_es.search.call_args[1] - assert 'match_all' in call_kwargs['query'] - - @patch('Common.elastic_utils.get_elastic_connection') - def test_query_error_handling(self, mock_get_connection, mock_connection): - """Test error handling when query fails""" - mock_es = Mock() - mock_es.search.side_effect = Exception("Query error") - mock_get_connection.return_value = mock_es - - docs = query_elasticsearch_documents( - mock_connection.id, - 'test-index' - ) - - assert docs == [] - - @patch('Common.elastic_utils.get_elastic_connection') - def test_query_with_specific_field_source(self, mock_get_connection, mock_connection): - """Test querying with specific field in _source""" - mock_es = Mock() - mock_es.search.return_value = { - 'hits': { - 'hits': [ - {'_source': {'field1': 'value1'}} - ] - } - } - mock_get_connection.return_value = mock_es - - docs = query_elasticsearch_documents( - mock_connection.id, - 'test-index', - field='field1', - size=10 - ) - - call_kwargs = mock_es.search.call_args[1] - assert call_kwargs['source'] == ['field1'] - - -class TestTestElasticConnectivity: - """Tests for test_elastic_connectivity function""" - - def test_returns_json_string(self): - """Test that test_elastic_connectivity returns a JSON-formatted string""" - import json - - mock_connection = Mock() - mock_connection.info.return_value = { - 'name': 'node-1', - 'cluster_name': 'my-cluster', - 'version': {'number': '8.0.0'} - } - - result = es_test_connectivity(mock_connection) - - assert isinstance(result, str) - # Should be valid JSON - parsed = json.loads(result) - assert parsed['name'] == 'node-1' - assert parsed['cluster_name'] == 'my-cluster' - - def test_json_is_pretty_printed(self): - """Test that the JSON output is indented (pretty-printed)""" - mock_connection = Mock() - mock_connection.info.return_value = {'name': 'node-1'} - - result = es_test_connectivity(mock_connection) - - # Pretty-printed JSON contains newlines and spaces for indentation - assert '\n' in result - assert ' ' in result # 4-space indent - - def test_calls_info_on_connection(self): - """Test that .info() is called on the provided connection object""" - mock_connection = Mock() - mock_connection.info.return_value = {'name': 'node-1'} - - es_test_connectivity(mock_connection) - - mock_connection.info.assert_called_once() - - def test_returns_all_cluster_info_fields(self): - """Test that all fields from cluster info are returned in JSON""" - import json - - cluster_info = { - 'name': 'node-1', - 'cluster_name': 'test-cluster', - 'cluster_uuid': 'abc-123', - 'version': { - 'number': '8.12.0', - 'build_flavor': 'default' - }, - 'tagline': 'You Know, for Search' - } - mock_connection = Mock() - mock_connection.info.return_value = cluster_info - - result = es_test_connectivity(mock_connection) - parsed = json.loads(result) - - # All top-level keys should be present - for key in cluster_info: - assert key in parsed - - def test_propagates_exception_from_info(self): - """Test that exceptions from .info() are propagated (not swallowed)""" - mock_connection = Mock() - mock_connection.info.side_effect = ConnectionError("Cannot reach Elasticsearch") - - with pytest.raises(ConnectionError): - es_test_connectivity(mock_connection) - - def test_empty_info_response(self): - """Test behavior when info() returns an empty dict""" - import json - - mock_connection = Mock() - mock_connection.info.return_value = {} - - result = es_test_connectivity(mock_connection) - parsed = json.loads(result) - assert parsed == {} - - -class TestNormalizeKibanaUrl: - """URL-based connections often store the ES endpoint; Agent Builder needs Kibana.""" - - def test_rewrites_es_infix_to_kb(self): - assert normalize_kibana_url( - 'https://proj.es.us-east-1.aws.elastic.cloud' - ) == 'https://proj.kb.us-east-1.aws.elastic.cloud' - - def test_inserts_kb_for_alias_without_es_or_kb(self): - assert normalize_kibana_url( - 'https://logstashuiserverless-ae1d5d3b.us-east-1.aws.elastic.cloud' - ) == 'https://logstashuiserverless-ae1d5d3b.kb.us-east-1.aws.elastic.cloud' - - def test_leaves_kibana_host_unchanged(self): - url = 'https://proj.kb.us-east-1.aws.elastic.cloud' - assert normalize_kibana_url(url) == url - - def test_strips_kibana_app_path(self): - assert normalize_kibana_url( - 'https://proj.kb.us-east-1.aws.elastic.cloud/app/home' - ) == 'https://proj.kb.us-east-1.aws.elastic.cloud' - - def test_drops_es_port_when_rewriting_elastic_cloud(self): - assert normalize_kibana_url( - 'https://proj.es.us-east-1.aws.elastic.cloud:9243' - ) == 'https://proj.kb.us-east-1.aws.elastic.cloud' - - def test_preserves_self_managed_kibana_port(self): - assert normalize_kibana_url('https://localhost:5601') == 'https://localhost:5601' - - def test_adds_https_when_scheme_missing(self): - assert normalize_kibana_url( - 'proj.es.eu-west-1.gcp.elastic.cloud' - ) == 'https://proj.kb.eu-west-1.gcp.elastic.cloud' - - def test_cloud_id_found_io_host_passthrough(self): - url = 'https://abc123.us-east-1.aws.found.io' - assert normalize_kibana_url(url) == url - - def test_empty_and_none(self): - assert normalize_kibana_url('') == '' - assert normalize_kibana_url(None) is None - diff --git a/src/logstashui/Common/tests/test_encryption.py b/src/logstashui/Common/tests/test_encryption.py deleted file mode 100644 index 9885ea7..0000000 --- a/src/logstashui/Common/tests/test_encryption.py +++ /dev/null @@ -1,267 +0,0 @@ -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. - -import pytest -import os -import tempfile -from pathlib import Path -from unittest.mock import patch, mock_open, Mock -from cryptography.fernet import Fernet, InvalidToken - -from Common.encryption import ( - get_encryption_key, - encrypt_credential, - decrypt_credential, - get_django_secret_key -) - - -@pytest.fixture -def temp_data_dir(tmp_path, monkeypatch): - """Isolated data directory (LOGSTASHUI_DATA_DIR).""" - data_dir = tmp_path / "data" - data_dir.mkdir() - monkeypatch.setenv("LOGSTASHUI_DATA_DIR", str(data_dir)) - return data_dir - - -class TestGetEncryptionKey: - """Test get_encryption_key function""" - - def test_key_from_environment_variable(self, monkeypatch, temp_data_dir): - """Test loading key from CREDENTIAL_KEY environment variable""" - valid_key = Fernet.generate_key() - monkeypatch.setenv('CREDENTIAL_KEY', valid_key.decode()) - - with patch('Common.encryption.Path') as mock_path: - mock_path.return_value.resolve.return_value.parent.parent = temp_data_dir.parent - key = get_encryption_key() - - assert key == valid_key - assert isinstance(key, bytes) - - def test_invalid_key_in_environment_variable(self, monkeypatch, temp_data_dir): - """Test that invalid CREDENTIAL_KEY raises RuntimeError""" - monkeypatch.setenv('CREDENTIAL_KEY', 'invalid-key-format') - - with patch('Common.encryption.Path') as mock_path: - mock_path.return_value.resolve.return_value.parent.parent = temp_data_dir.parent - with pytest.raises(RuntimeError, match="Invalid CREDENTIAL_KEY format"): - get_encryption_key() - - def test_key_from_file(self, temp_data_dir): - """Test loading key from file""" - key_file = temp_data_dir / ".secret_key" - valid_key = Fernet.generate_key() - key_file.write_bytes(valid_key) - - with patch('Common.encryption.Path') as mock_path: - mock_path.return_value.resolve.return_value.parent.parent = temp_data_dir.parent - key = get_encryption_key() - - assert key == valid_key - - def test_invalid_key_in_file(self, temp_data_dir): - """Test that invalid key in file raises RuntimeError""" - key_file = temp_data_dir / ".secret_key" - key_file.write_bytes(b'invalid-key-data') - - with patch('Common.encryption.Path') as mock_path: - mock_path.return_value.resolve.return_value.parent.parent = temp_data_dir.parent - with pytest.raises(RuntimeError, match="Invalid encryption key in file"): - get_encryption_key() - - def test_generate_new_key_and_persist(self, temp_data_dir): - """Test generating new key and persisting to file""" - key_file = temp_data_dir / ".secret_key" - - with patch('Common.encryption.Path') as mock_path: - mock_path.return_value.resolve.return_value.parent.parent = temp_data_dir.parent - key = get_encryption_key() - - # Verify key was generated - assert isinstance(key, bytes) - assert len(key) > 0 - - # Verify key was saved to file - assert key_file.exists() - saved_key = key_file.read_bytes() - assert saved_key == key - - # Verify key is valid Fernet key - fernet = Fernet(key) - assert fernet is not None - - def test_permission_error_reading_key_file(self, temp_data_dir): - """Test handling of permission errors when reading key file""" - key_file = temp_data_dir / ".secret_key" - key_file.write_bytes(Fernet.generate_key()) - - with patch('Common.encryption.Path') as mock_path: - mock_path.return_value.resolve.return_value.parent.parent = temp_data_dir.parent - with patch('builtins.open', side_effect=PermissionError("Access denied")): - with pytest.raises(RuntimeError, match="Cannot read encryption key file: Permission denied"): - get_encryption_key() - - -class TestEncryptDecryptCredential: - """Test encrypt_credential and decrypt_credential functions""" - - def test_encrypt_decrypt_round_trip(self, temp_data_dir): - """Test that encryption and decryption work correctly""" - plaintext = "my-secret-password" - - with patch('Common.encryption.Path') as mock_path: - mock_path.return_value.resolve.return_value.parent.parent = temp_data_dir.parent - - encrypted = encrypt_credential(plaintext) - assert encrypted != plaintext - assert isinstance(encrypted, str) - - decrypted = decrypt_credential(encrypted) - assert decrypted == plaintext - - def test_encrypt_empty_string_passthrough(self): - """Test that empty string is passed through without encryption""" - assert encrypt_credential("") == "" - assert encrypt_credential(None) is None - - def test_decrypt_empty_string_passthrough(self): - """Test that empty string is passed through without decryption""" - assert decrypt_credential("") == "" - assert decrypt_credential(None) is None - - def test_encrypt_non_string_raises_error(self): - """Test that encrypting non-string raises ValueError""" - with pytest.raises(ValueError, match="plaintext must be a string"): - encrypt_credential(123) - - with pytest.raises(ValueError, match="plaintext must be a string"): - encrypt_credential(['list']) - - def test_decrypt_non_string_raises_error(self): - """Test that decrypting non-string raises ValueError""" - with pytest.raises(ValueError, match="encrypted_text must be a string"): - decrypt_credential(123) - - with pytest.raises(ValueError, match="encrypted_text must be a string"): - decrypt_credential(['list']) - - def test_decrypt_invalid_token(self, temp_data_dir): - """Test that decrypting with invalid token raises ValueError""" - with patch('Common.encryption.Path') as mock_path: - mock_path.return_value.resolve.return_value.parent.parent = temp_data_dir.parent - - with pytest.raises(ValueError, match="Cannot decrypt credential: Invalid token"): - decrypt_credential("invalid-encrypted-data") - - def test_decrypt_with_wrong_key(self, temp_data_dir): - """Test that decrypting with wrong key raises ValueError""" - # Encrypt with one key - key1 = Fernet.generate_key() - fernet1 = Fernet(key1) - encrypted = fernet1.encrypt(b"secret").decode() - - # Try to decrypt with different key - key_file = temp_data_dir / ".secret_key" - key2 = Fernet.generate_key() - key_file.write_bytes(key2) - - with patch('Common.encryption.Path') as mock_path: - mock_path.return_value.resolve.return_value.parent.parent = temp_data_dir.parent - - with pytest.raises(ValueError, match="Cannot decrypt credential: Invalid token"): - decrypt_credential(encrypted) - - def test_encrypt_unicode_characters(self, temp_data_dir): - """Test encrypting and decrypting unicode characters""" - plaintext = "🔒 Secret with émojis and spëcial çhars 中文" - - with patch('Common.encryption.Path') as mock_path: - mock_path.return_value.resolve.return_value.parent.parent = temp_data_dir.parent - - encrypted = encrypt_credential(plaintext) - decrypted = decrypt_credential(encrypted) - assert decrypted == plaintext - - -class TestGetDjangoSecretKey: - """Test get_django_secret_key function""" - - def test_key_from_environment_variable(self, monkeypatch, temp_data_dir): - """Test loading Django secret key from environment variable""" - secret_key = "test-secret-key-from-environment-variable-long-enough" - monkeypatch.setenv('SECRET_KEY', secret_key) - - with patch('Common.encryption.Path') as mock_path: - mock_path.return_value.resolve.return_value.parent.parent = temp_data_dir.parent - key = get_django_secret_key() - - assert key == secret_key - - def test_short_key_warning(self, monkeypatch, temp_data_dir, caplog): - """Test that short SECRET_KEY generates warning""" - short_key = "short" - monkeypatch.setenv('SECRET_KEY', short_key) - - with patch('Common.encryption.Path') as mock_path: - mock_path.return_value.resolve.return_value.parent.parent = temp_data_dir.parent - key = get_django_secret_key() - - assert key == short_key - assert "SECRET_KEY from environment is short" in caplog.text - - def test_key_from_file(self, temp_data_dir): - """Test loading Django secret key from file""" - key_file = temp_data_dir / ".django_secret_key" - secret_key = "test-secret-key-from-file-should-be-long-enough-now" - key_file.write_text(secret_key) - - with patch('Common.encryption.Path') as mock_path: - mock_path.return_value.resolve.return_value.parent.parent = temp_data_dir.parent - key = get_django_secret_key() - - assert key == secret_key - - def test_empty_key_file_raises_error(self, temp_data_dir): - """Test that empty key file raises RuntimeError""" - key_file = temp_data_dir / ".django_secret_key" - key_file.write_text("") - - with patch('Common.encryption.Path') as mock_path: - mock_path.return_value.resolve.return_value.parent.parent = temp_data_dir.parent - with pytest.raises(RuntimeError, match="Django secret key file is empty"): - get_django_secret_key() - - def test_generate_new_key_and_persist(self, temp_data_dir): - """Test generating new Django secret key and persisting to file""" - key_file = temp_data_dir / ".django_secret_key" - - with patch('Common.encryption.Path') as mock_path: - mock_path.return_value.resolve.return_value.parent.parent = temp_data_dir.parent - key = get_django_secret_key() - - # Verify key was generated - assert isinstance(key, str) - assert len(key) == 50 - - # Verify key contains expected characters - valid_chars = set('abcdefghijklmnopqrstuvwxyz0123456789!@#$%^&*(-_=+)') - assert all(c in valid_chars for c in key) - - # Verify key was saved to file - assert key_file.exists() - saved_key = key_file.read_text().strip() - assert saved_key == key - - def test_permission_error_reading_key_file(self, temp_data_dir): - """Test handling of permission errors when reading Django secret key file""" - key_file = temp_data_dir / ".django_secret_key" - key_file.write_text("test-key") - - with patch('Common.encryption.Path') as mock_path: - mock_path.return_value.resolve.return_value.parent.parent = temp_data_dir.parent - with patch('builtins.open', side_effect=PermissionError("Access denied")): - with pytest.raises(RuntimeError, match="Cannot read Django secret key: Permission denied"): - get_django_secret_key() diff --git a/src/logstashui/Common/tests/test_error_handlers.py b/src/logstashui/Common/tests/test_error_handlers.py deleted file mode 100644 index bf80652..0000000 --- a/src/logstashui/Common/tests/test_error_handlers.py +++ /dev/null @@ -1,263 +0,0 @@ -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. - -import pytest -from django.test import RequestFactory - -from Common.error_handlers import handler400, handler403, handler404, handler500 - -# All tests in this file require DB access because Django's render() triggers -# context processors (navigation_highlight) that query Connection/Device models. -pytestmark = pytest.mark.django_db - - -@pytest.fixture -def request_factory(): - """Django RequestFactory for creating mock requests""" - return RequestFactory() - - -@pytest.fixture -def mock_request(request_factory): - """Create a basic GET mock request""" - return request_factory.get('/some/path/') - - -class TestHandler400: - """Tests for handler400 (Bad Request)""" - - def test_returns_400_status(self, mock_request): - """Test that handler400 returns a 400 status code""" - response = handler400(mock_request) - assert response.status_code == 400 - - def test_correct_error_code_in_context(self, mock_request): - """Test that context contains correct error code""" - response = handler400(mock_request) - assert response.context_data['error_code'] == '400' - - def test_correct_error_title(self, mock_request): - """Test that context contains correct error title""" - response = handler400(mock_request) - assert response.context_data['error_title'] == 'Bad Request' - - def test_correct_error_message(self, mock_request): - """Test that context contains appropriate error message""" - response = handler400(mock_request) - assert 'could not understand' in response.context_data['error_message'] - - def test_with_exception(self, mock_request): - """Test that exception class name is included when exception is provided""" - exc = ValueError("bad input") - response = handler400(mock_request, exception=exc) - assert response.context_data['exception'] == 'ValueError' - - def test_without_exception(self, mock_request): - """Test that exception is None when no exception is provided""" - response = handler400(mock_request) - assert response.context_data['exception'] is None - - def test_with_none_exception(self, mock_request): - """Test that explicitly passing None exception gives None in context""" - response = handler400(mock_request, exception=None) - assert response.context_data['exception'] is None - - def test_uses_error_template(self, mock_request): - """Test that the error.html template is used""" - response = handler400(mock_request) - assert response.template_name == 'error.html' - - -class TestHandler403: - """Tests for handler403 (Access Denied)""" - - def test_returns_403_status(self, mock_request): - """Test that handler403 returns a 403 status code""" - response = handler403(mock_request) - assert response.status_code == 403 - - def test_correct_error_code(self, mock_request): - """Test correct error code in context""" - response = handler403(mock_request) - assert response.context_data['error_code'] == '403' - - def test_correct_error_title(self, mock_request): - """Test correct error title""" - response = handler403(mock_request) - assert response.context_data['error_title'] == 'Access Denied' - - def test_correct_error_message(self, mock_request): - """Test permission-related error message""" - response = handler403(mock_request) - assert 'permission' in response.context_data['error_message'].lower() - - def test_with_exception(self, mock_request): - """Test exception class name in context""" - exc = PermissionError("Access denied") - response = handler403(mock_request, exception=exc) - assert response.context_data['exception'] == 'PermissionError' - - def test_without_exception(self, mock_request): - """Test that exception is None when not provided""" - response = handler403(mock_request) - assert response.context_data['exception'] is None - - def test_uses_error_template(self, mock_request): - """Test that the error.html template is used""" - response = handler403(mock_request) - assert response.template_name == 'error.html' - - -class TestHandler404: - """Tests for handler404 (Page Not Found)""" - - def test_returns_404_status(self, mock_request): - """Test that handler404 returns a 404 status code""" - response = handler404(mock_request) - assert response.status_code == 404 - - def test_correct_error_code(self, mock_request): - """Test correct error code in context""" - response = handler404(mock_request) - assert response.context_data['error_code'] == '404' - - def test_correct_error_title(self, mock_request): - """Test correct error title""" - response = handler404(mock_request) - assert response.context_data['error_title'] == 'Page Not Found' - - def test_correct_error_message(self, mock_request): - """Test not-found error message""" - response = handler404(mock_request) - assert 'does not exist' in response.context_data['error_message'] - - def test_path_included_in_context(self, request_factory): - """Test that request path is included in context for 404""" - request = request_factory.get('/missing/page/') - response = handler404(request) - assert response.context_data['path'] == '/missing/page/' - - def test_with_exception(self, mock_request): - """Test exception class name in context""" - exc = LookupError("Not found") - response = handler404(mock_request, exception=exc) - assert response.context_data['exception'] == 'LookupError' - - def test_without_exception(self, mock_request): - """Test that exception is None when not provided""" - response = handler404(mock_request) - assert response.context_data['exception'] is None - - def test_uses_error_template(self, mock_request): - """Test that the error.html template is used""" - response = handler404(mock_request) - assert response.template_name == 'error.html' - - def test_path_reflects_actual_request(self, request_factory): - """Test that path value matches the actual request path""" - request = request_factory.get('/admin/nonexistent/') - response = handler404(request) - assert response.context_data['path'] == '/admin/nonexistent/' - - -class TestHandler500: - """Tests for handler500 (Server Error)""" - - def test_returns_500_status(self, mock_request): - """Test that handler500 returns a 500 status code""" - response = handler500(mock_request) - assert response.status_code == 500 - - def test_correct_error_code(self, mock_request): - """Test correct error code in context""" - response = handler500(mock_request) - assert response.context_data['error_code'] == '500' - - def test_correct_error_title(self, mock_request): - """Test correct error title""" - response = handler500(mock_request) - assert response.context_data['error_title'] == 'Server Error' - - def test_correct_error_message(self, mock_request): - """Test server error message""" - response = handler500(mock_request) - assert 'Something went wrong' in response.context_data['error_message'] - - def test_path_included_in_context(self, request_factory): - """Test that request path is included in context for 500""" - request = request_factory.get('/api/some-endpoint/') - response = handler500(request) - assert response.context_data['path'] == '/api/some-endpoint/' - - def test_with_exception(self, mock_request): - """Test exception class name in context""" - exc = RuntimeError("Something blew up") - response = handler500(mock_request, exception=exc) - assert response.context_data['exception'] == 'RuntimeError' - - def test_without_exception(self, mock_request): - """Test that exception is None when not provided""" - response = handler500(mock_request) - assert response.context_data['exception'] is None - - def test_uses_error_template(self, mock_request): - """Test that the error.html template is used""" - response = handler500(mock_request) - assert response.template_name == 'error.html' - - def test_path_reflects_actual_request(self, request_factory): - """Test that path value matches the actual request path""" - request = request_factory.get('/pipeline/1/deploy/') - response = handler500(request) - assert response.context_data['path'] == '/pipeline/1/deploy/' - - -class TestErrorHandlerEdgeCases: - """Edge case tests across all error handlers""" - - def test_all_handlers_use_same_template(self, mock_request): - """Test that all four handlers use the same error.html template""" - for handler in [handler400, handler403, handler404, handler500]: - response = handler(mock_request) - assert response.template_name == 'error.html', \ - f"{handler.__name__} should use 'error.html' template" - - def test_exception_class_name_not_message(self, mock_request): - """Test that context stores class name, not exception message""" - exc = ValueError("This is the message, not the class name") - response = handler400(mock_request, exception=exc) - # Should be class name, not the message - assert response.context_data['exception'] == 'ValueError' - assert 'message' not in response.context_data['exception'] - - def test_handler404_and_500_include_path_not_400_403(self, mock_request): - """Test that only 404 and 500 include path in context""" - # 400 and 403 should NOT have path - assert 'path' not in handler400(mock_request).context_data - assert 'path' not in handler403(mock_request).context_data - # 404 and 500 SHOULD have path - assert 'path' in handler404(mock_request).context_data - assert 'path' in handler500(mock_request).context_data - - @pytest.mark.parametrize("handler,expected_status", [ - (handler400, 400), - (handler403, 403), - (handler404, 404), - (handler500, 500), - ]) - def test_status_codes(self, mock_request, handler, expected_status): - """Parametrized test verifying each handler returns the correct status code""" - response = handler(mock_request) - assert response.status_code == expected_status - - @pytest.mark.parametrize("handler,expected_code", [ - (handler400, '400'), - (handler403, '403'), - (handler404, '404'), - (handler500, '500'), - ]) - def test_error_codes_in_context(self, mock_request, handler, expected_code): - """Parametrized test verifying error_code in context matches HTTP status""" - response = handler(mock_request) - assert response.context_data['error_code'] == expected_code diff --git a/src/logstashui/Common/tests/test_formatters.py b/src/logstashui/Common/tests/test_formatters.py deleted file mode 100644 index c0c2a4f..0000000 --- a/src/logstashui/Common/tests/test_formatters.py +++ /dev/null @@ -1,407 +0,0 @@ -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. - -import pytest -from Common.formatters import ( - _safe_get_numeric, - _safe_extract_value, - _format_uptime, - _sanitize_pipeline_name_component -) - - -class TestSafeGetNumeric: - """Test _safe_get_numeric function with edge cases""" - - def test_integer_value(self): - """Test integer value is returned as-is""" - assert _safe_get_numeric(42) == 42 - assert isinstance(_safe_get_numeric(42), int) - - def test_float_value(self): - """Test float value is returned as-is""" - assert _safe_get_numeric(3.14) == 3.14 - assert isinstance(_safe_get_numeric(3.14), float) - - def test_string_integer(self): - """Test string integer is converted to int""" - assert _safe_get_numeric("123") == 123 - assert isinstance(_safe_get_numeric("123"), int) - - def test_string_float(self): - """Test string float is converted to float""" - assert _safe_get_numeric("3.14") == 3.14 - assert isinstance(_safe_get_numeric("3.14"), float) - - def test_none_returns_default(self): - """Test None returns default value""" - assert _safe_get_numeric(None) == 0 - assert _safe_get_numeric(None, default=100) == 100 - - def test_empty_list_returns_default(self): - """Test empty list returns default value""" - assert _safe_get_numeric([]) == 0 - assert _safe_get_numeric([], default=50) == 50 - - def test_list_with_integer(self): - """Test list with integer returns first element""" - assert _safe_get_numeric([42]) == 42 - assert _safe_get_numeric([42, 100]) == 42 - - def test_list_with_float(self): - """Test list with float returns first element""" - assert _safe_get_numeric([3.14]) == 3.14 - assert _safe_get_numeric([3.14, 2.71]) == 3.14 - - def test_list_with_string_number(self): - """Test list with string number converts first element""" - assert _safe_get_numeric(["123"]) == 123 - assert _safe_get_numeric(["3.14"]) == 3.14 - - def test_invalid_string_returns_default(self): - """Test invalid string returns default value""" - assert _safe_get_numeric("not a number") == 0 - assert _safe_get_numeric("abc", default=99) == 99 - - def test_list_with_invalid_string_returns_default(self): - """Test list with invalid string returns default""" - assert _safe_get_numeric(["invalid"]) == 0 - assert _safe_get_numeric(["invalid"], default=77) == 77 - - def test_boolean_value(self): - """Test boolean values (True=1, False=0)""" - assert _safe_get_numeric(True) == 1 - assert _safe_get_numeric(False) == 0 - - def test_zero_value(self): - """Test zero is returned correctly""" - assert _safe_get_numeric(0) == 0 - assert _safe_get_numeric("0") == 0 - assert _safe_get_numeric([0]) == 0 - - def test_negative_numbers(self): - """Test negative numbers are handled correctly""" - assert _safe_get_numeric(-42) == -42 - assert _safe_get_numeric("-42") == -42 - assert _safe_get_numeric([-3.14]) == -3.14 - - def test_custom_default_value(self): - """Test custom default values work correctly""" - assert _safe_get_numeric(None, default=-1) == -1 - assert _safe_get_numeric([], default=999) == 999 - assert _safe_get_numeric("invalid", default=42) == 42 - - def test_list_with_none_returns_default(self): - """Test list containing None returns default""" - assert _safe_get_numeric([None]) == 0 - assert _safe_get_numeric([None], default=10) == 10 - - def test_scientific_notation(self): - """Test scientific notation strings - not supported, returns default""" - # The function uses '.' check for float detection, so "1e3" is treated as invalid - assert _safe_get_numeric("1e3") == 0 - # "1.5e2" has a period so it tries float() which works - assert _safe_get_numeric("1.5e2") == 150.0 - - def test_whitespace_in_string(self): - """Test strings with whitespace""" - assert _safe_get_numeric(" 123 ") == 123 - assert _safe_get_numeric(" 3.14 ") == 3.14 - - def test_dict_returns_default(self): - """Test dict returns default value""" - assert _safe_get_numeric({"value": 123}) == 0 - assert _safe_get_numeric({"value": 123}, default=5) == 5 - - -class TestSafeExtractValue: - """Test _safe_extract_value function with edge cases""" - - def test_simple_value(self): - """Test simple values are returned as-is""" - assert _safe_extract_value("test") == "test" - assert _safe_extract_value(123) == 123 - assert _safe_extract_value(3.14) == 3.14 - - def test_none_returns_default(self): - """Test None returns default value""" - assert _safe_extract_value(None) == 0 - assert _safe_extract_value(None, default="default") == "default" - - def test_empty_list_returns_default(self): - """Test empty list returns default value""" - assert _safe_extract_value([]) == 0 - assert _safe_extract_value([], default="empty") == "empty" - - def test_list_with_value(self): - """Test list with value returns first element""" - assert _safe_extract_value(["test"]) == "test" - assert _safe_extract_value([123]) == 123 - assert _safe_extract_value(["first", "second"]) == "first" - - def test_list_with_none_returns_default(self): - """Test list with None values returns default""" - assert _safe_extract_value([None]) == 0 - assert _safe_extract_value([None, None]) == 0 - assert _safe_extract_value([None], default="none") == "none" - - def test_list_with_empty_string_returns_default(self): - """Test list with empty strings returns default""" - assert _safe_extract_value([""]) == 0 - assert _safe_extract_value(["", ""]) == 0 - assert _safe_extract_value([""], default="empty") == "empty" - - def test_list_with_mixed_none_and_empty(self): - """Test list with mix of None and empty strings returns default""" - assert _safe_extract_value([None, "", None]) == 0 - assert _safe_extract_value(["", None, ""], default=99) == 99 - - def test_list_with_valid_after_invalid(self): - """Test list returns first non-null, non-empty value""" - assert _safe_extract_value([None, "valid"]) == "valid" - assert _safe_extract_value(["", "valid"]) == "valid" - assert _safe_extract_value([None, "", "valid"]) == "valid" - - def test_list_with_zero(self): - """Test list with zero value (zero is valid, not empty)""" - assert _safe_extract_value([0]) == 0 - assert _safe_extract_value([None, 0]) == 0 - - def test_list_with_false(self): - """Test list with False value (False is valid, not empty)""" - assert _safe_extract_value([False]) is False - assert _safe_extract_value([None, False]) is False - - def test_custom_default_value(self): - """Test custom default values""" - assert _safe_extract_value(None, default="custom") == "custom" - assert _safe_extract_value([], default=999) == 999 - - def test_dict_value(self): - """Test dict values are returned as-is""" - test_dict = {"key": "value"} - assert _safe_extract_value(test_dict) == test_dict - - def test_boolean_values(self): - """Test boolean values are returned correctly""" - assert _safe_extract_value(True) is True - assert _safe_extract_value(False) is False - - def test_list_with_whitespace_string(self): - """Test list with whitespace-only string (treated as non-empty)""" - assert _safe_extract_value([" "]) == " " - assert _safe_extract_value([" "]) == " " - - -class TestFormatUptime: - """Test _format_uptime function with edge cases""" - - def test_zero_milliseconds(self): - """Test zero milliseconds""" - assert _format_uptime(0) == "0s" - - def test_seconds_only(self): - """Test uptime in seconds only""" - assert _format_uptime(5000) == "5s" - assert _format_uptime(59000) == "59s" - - def test_minutes_and_seconds(self): - """Test uptime in minutes and seconds""" - assert _format_uptime(60000) == "1m 0s" - assert _format_uptime(90000) == "1m 30s" - assert _format_uptime(3599000) == "59m 59s" - - def test_hours_and_minutes(self): - """Test uptime in hours and minutes""" - assert _format_uptime(3600000) == "1h 0m" - assert _format_uptime(3660000) == "1h 1m" - assert _format_uptime(7200000) == "2h 0m" - assert _format_uptime(86399000) == "23h 59m" - - def test_days_and_hours(self): - """Test uptime in days and hours""" - assert _format_uptime(86400000) == "1d 0h" - assert _format_uptime(90000000) == "1d 1h" - assert _format_uptime(172800000) == "2d 0h" - assert _format_uptime(176400000) == "2d 1h" - - def test_one_millisecond(self): - """Test one millisecond rounds to 0 seconds""" - assert _format_uptime(1) == "0s" - - def test_999_milliseconds(self): - """Test 999 milliseconds rounds to 0 seconds""" - assert _format_uptime(999) == "0s" - - def test_exactly_one_minute(self): - """Test exactly one minute""" - assert _format_uptime(60000) == "1m 0s" - - def test_exactly_one_hour(self): - """Test exactly one hour""" - assert _format_uptime(3600000) == "1h 0m" - - def test_exactly_one_day(self): - """Test exactly one day""" - assert _format_uptime(86400000) == "1d 0h" - - def test_large_uptime(self): - """Test large uptime values""" - # 30 days - assert _format_uptime(2592000000) == "30d 0h" - # 365 days - assert _format_uptime(31536000000) == "365d 0h" - - def test_complex_uptime(self): - """Test complex uptime with all components""" - # 1 day, 2 hours, 3 minutes, 4 seconds, 500 milliseconds - ms = (1 * 86400000) + (2 * 3600000) + (3 * 60000) + (4 * 1000) + 500 - assert _format_uptime(ms) == "1d 2h" - - def test_uptime_priority_days_over_hours(self): - """Test that days format takes priority over hours""" - # 1 day, 23 hours - ms = (1 * 86400000) + (23 * 3600000) - assert _format_uptime(ms) == "1d 23h" - - def test_uptime_priority_hours_over_minutes(self): - """Test that hours format takes priority over minutes""" - # 1 hour, 59 minutes - ms = (1 * 3600000) + (59 * 60000) - assert _format_uptime(ms) == "1h 59m" - - def test_uptime_priority_minutes_over_seconds(self): - """Test that minutes format takes priority over seconds""" - # 1 minute, 59 seconds - ms = (1 * 60000) + (59 * 1000) - assert _format_uptime(ms) == "1m 59s" - - def test_negative_uptime(self): - """Test negative uptime (edge case, should handle gracefully)""" - # Negative values will result in negative calculations - result = _format_uptime(-1000) - assert "s" in result - - def test_fractional_seconds(self): - """Test that fractional seconds are truncated""" - # 1500ms = 1.5 seconds, should show as 1s - assert _format_uptime(1500) == "1s" - # 2999ms = 2.999 seconds, should show as 2s - assert _format_uptime(2999) == "2s" - - @pytest.mark.parametrize("milliseconds,expected", [ - (0, "0s"), - (1000, "1s"), - (60000, "1m 0s"), - (3600000, "1h 0m"), - (86400000, "1d 0h"), - (90061000, "1d 1h"), # 1 day, 1 hour, 1 minute, 1 second - (5000, "5s"), - (125000, "2m 5s"), - (7325000, "2h 2m"), - (90000000, "1d 1h"), - ]) - def test_parametrized_uptime_formats(self, milliseconds, expected): - """Test various uptime formats with parametrized inputs""" - assert _format_uptime(milliseconds) == expected - - -class TestSanitizePipelineNameComponent: - """Test _sanitize_pipeline_name_component function""" - - def test_simple_valid_name(self): - """Test simple valid name is returned as-is (lowercased)""" - assert _sanitize_pipeline_name_component('myname') == 'myname' - - def test_uppercase_is_lowercased(self): - """Test that uppercase letters are lowercased""" - assert _sanitize_pipeline_name_component('MyName') == 'myname' - assert _sanitize_pipeline_name_component('UPPERCASE') == 'uppercase' - - def test_numbers_are_preserved(self): - """Test that numbers are kept""" - assert _sanitize_pipeline_name_component('name123') == 'name123' - assert _sanitize_pipeline_name_component('abc456def') == 'abc456def' - - def test_underscores_are_preserved(self): - """Test that underscores are kept""" - assert _sanitize_pipeline_name_component('my_name') == 'my_name' - assert _sanitize_pipeline_name_component('a_b_c') == 'a_b_c' - - def test_hyphens_are_preserved(self): - """Test that hyphens are kept""" - assert _sanitize_pipeline_name_component('my-name') == 'my-name' - assert _sanitize_pipeline_name_component('a-b-c') == 'a-b-c' - - def test_spaces_replaced_with_underscore(self): - """Test that spaces are replaced with underscores""" - assert _sanitize_pipeline_name_component('my name') == 'my_name' - assert _sanitize_pipeline_name_component('hello world foo') == 'hello_world_foo' - - def test_special_characters_replaced_with_underscore(self): - """Test that special characters are replaced with underscores""" - assert _sanitize_pipeline_name_component('name@host') == 'name_host' - assert _sanitize_pipeline_name_component('name.value') == 'name_value' - assert _sanitize_pipeline_name_component('name/path') == 'name_path' - assert _sanitize_pipeline_name_component('name:port') == 'name_port' - - def test_consecutive_underscores_collapsed(self): - """Test that consecutive underscores are collapsed to a single one""" - assert _sanitize_pipeline_name_component('a__b') == 'a_b' - assert _sanitize_pipeline_name_component('a___b') == 'a_b' - # Multiple special chars in a row → multiple underscores → collapsed - assert _sanitize_pipeline_name_component('a@#b') == 'a_b' - - def test_leading_underscores_stripped(self): - """Test that leading underscores are stripped""" - assert _sanitize_pipeline_name_component('_name') == 'name' - assert _sanitize_pipeline_name_component('__name') == 'name' - - def test_trailing_underscores_stripped(self): - """Test that trailing underscores are stripped""" - assert _sanitize_pipeline_name_component('name_') == 'name' - assert _sanitize_pipeline_name_component('name__') == 'name' - - def test_leading_special_chars_stripped(self): - """Test that leading special characters (→ underscores) are stripped""" - # '@name' → '_name' (special char → underscore) → 'name' (stripped leading) - assert _sanitize_pipeline_name_component('@name') == 'name' - - def test_all_allowed_chars(self): - """Test a name using all allowed character types""" - result = _sanitize_pipeline_name_component('MyName-123_test') - assert result == 'myname-123_test' - - def test_empty_string(self): - """Test that empty string returns empty string""" - assert _sanitize_pipeline_name_component('') == '' - - def test_only_special_chars(self): - """Test that a string of only special chars results in empty string""" - result = _sanitize_pipeline_name_component('@@@@') - # All replaced with underscores, collapsed, then stripped → empty - assert result == '' - - def test_unicode_replaced_with_underscore(self): - """Test that unicode/emoji chars are replaced with underscores""" - result = _sanitize_pipeline_name_component('name_中文') - # Non-ASCII chars each become '_', consecutive collapsed, trailing stripped - assert result == 'name' - - def test_already_clean_name(self): - """Test that a clean name passes through unchanged (except lowercasing)""" - assert _sanitize_pipeline_name_component('good-name_123') == 'good-name_123' - - @pytest.mark.parametrize('input_name,expected', [ - ('Hello World', 'hello_world'), - ('My-Pipeline_Name', 'my-pipeline_name'), - (' spaces ', 'spaces'), - ('a.b.c', 'a_b_c'), - ('test!@#$%', 'test'), - ('UPPER_LOWER', 'upper_lower'), - ('123abc', '123abc'), - ]) - def test_parametrized_sanitization(self, input_name, expected): - """Parametrized test for various sanitization scenarios""" - assert _sanitize_pipeline_name_component(input_name) == expected diff --git a/src/logstashui/Common/tests/test_logstash_config_parse.py b/src/logstashui/Common/tests/test_logstash_config_parse.py deleted file mode 100644 index 9f279b2..0000000 --- a/src/logstashui/Common/tests/test_logstash_config_parse.py +++ /dev/null @@ -1,542 +0,0 @@ -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. - -import pytest -import json - -from Common.logstash_config_parse import ( - _extract_error_context, - _strip_inline_comments, - parse_logstash_config, - logstash_config_to_components, - ComponentToPipeline, -) - - -# ───────────────────────────────────────────────────────────── -# Helpers / shared fixtures -# ───────────────────────────────────────────────────────────── - -MINIMAL_INPUT = 'input {\n stdin {}\n}\n' -MINIMAL_FILTER = 'filter {\n mutate {}\n}\n' -MINIMAL_OUTPUT = 'output {\n stdout {}\n}\n' -FULL_PIPELINE = MINIMAL_INPUT + MINIMAL_FILTER + MINIMAL_OUTPUT - - -# ───────────────────────────────────────────────────────────── -# _extract_error_context -# ───────────────────────────────────────────────────────────── - -class TestExtractErrorContext: - """Unit tests for _extract_error_context""" - - def test_contains_separator_lines(self): - """Result includes the === separator lines""" - config = "input {\n stdin {}\n}\n" - result = _extract_error_context(config, line=1, column=1) - assert '=' * 60 in result - - def test_contains_problematic_code_header(self): - """Result contains the PROBLEMATIC CODE label""" - config = "input {\n stdin {}\n}\n" - result = _extract_error_context(config, line=1, column=1) - assert 'PROBLEMATIC CODE' in result - - def test_error_line_marked_with_arrow(self): - """The error line is prefixed with '>>>'""" - config = "input {\n bad_line\n}\n" - result = _extract_error_context(config, line=2, column=1) - assert '>>>' in result - - def test_non_error_lines_not_marked(self): - """Non-error lines have normal indentation, not '>>>'""" - config = "input {\n bad_line\n}\n" - result = _extract_error_context(config, line=2, column=1) - lines = result.split('\n') - # Lines that aren't the error line should start with spaces, not >>> - non_arrow_lines = [l for l in lines if l.strip() and not l.strip().startswith(('=', 'P', '>')) and '|' in l] - for line in non_arrow_lines: - assert not line.startswith('>>>') - - def test_column_pointer_added(self): - """A '^' pointer is added at the correct column position""" - config = "input {\n bad_line\n}\n" - result = _extract_error_context(config, line=2, column=3) - assert '^' in result - - def test_no_pointer_for_column_zero(self): - """No '^' pointer when column is 0""" - config = "input {\n bad_line\n}\n" - result = _extract_error_context(config, line=2, column=0) - assert '^' not in result - - def test_line_number_appears_in_output(self): - """The error line number appears in the formatted output""" - config = "line1\nline2\nline3\n" - result = _extract_error_context(config, line=2, column=1) - assert '2' in result - - def test_context_lines_parameter_controls_range(self): - """context_lines parameter controls how many surrounding lines are shown""" - config = "\n".join([f"line{i}" for i in range(1, 21)]) - # With 1 context line around line 10, we should see lines 9, 10, 11 - result_1 = _extract_error_context(config, line=10, column=1, context_lines=1) - result_5 = _extract_error_context(config, line=10, column=1, context_lines=5) - # Larger context means more lines shown - assert len(result_5) > len(result_1) - - def test_first_line_error_no_index_error(self): - """Error at line 1 should not cause an IndexError""" - config = "bad config\nmore\n" - result = _extract_error_context(config, line=1, column=1) - assert '>>>' in result - - def test_last_line_error_no_index_error(self): - """Error at last line should not cause an IndexError""" - config = "good\nbad" - result = _extract_error_context(config, line=2, column=1) - assert '>>>' in result - - def test_returns_string(self): - """Function always returns a string""" - config = "input {}" - result = _extract_error_context(config, line=1, column=1) - assert isinstance(result, str) - - -# ───────────────────────────────────────────────────────────── -# _strip_inline_comments -# ───────────────────────────────────────────────────────────── - -class TestStripInlineComments: - """Unit tests for _strip_inline_comments""" - - def test_removes_inline_comment_after_value(self): - """Inline comments inside plugin blocks are stripped from the value line - and re-injected as standalone comment lines before the plugin's closing }""" - config = 'input {\n beats {\n port => 5044 # this is a comment\n }\n}\n' - result = _strip_inline_comments(config) - # The comment is moved, not discarded — it should appear somewhere in the output - assert '# this is a comment' in result - # The original value line must have the comment stripped - lines = result.split('\n') - port_line = next(l for l in lines if 'port' in l) - assert '# this is a comment' not in port_line - assert 'port => 5044' in port_line - - def test_removes_inline_comment_after_closing_brace(self): - """Inline comments on a plugin's closing } line are emitted as a - standalone comment line at the outer scope after the }""" - config = 'input {\n beats {\n port => 5044\n } # end beats\n}\n' - result = _strip_inline_comments(config) - # Comment is re-emitted after the closing }, not discarded - assert '# end beats' in result - # The } line itself must be clean - lines = result.split('\n') - close_line = next(l for l in lines if l.strip() == '}') - assert '# end beats' not in close_line - - def test_preserves_hash_in_string_value(self): - """# inside a quoted string is NOT treated as a comment""" - config = 'input {\n beats {\n path => "/var/log/#file"\n }\n}\n' - result = _strip_inline_comments(config) - assert '/var/log/#file' in result - - def test_preserves_standalone_comment_at_section_level(self): - """Standalone comment lines at the top/section level are preserved""" - config = '# This is a top-level comment\ninput {\n stdin {}\n}\n' - result = _strip_inline_comments(config) - assert '# This is a top-level comment' in result - - def test_preserves_standalone_comment_inside_plugin_block(self): - """Standalone comment lines INSIDE a plugin block are preserved in place - (the grammar now supports them and attaches them as plugin.comments[])""" - config = 'input {\n beats {\n # comment inside plugin\n port => 5044\n }\n}\n' - result = _strip_inline_comments(config) - assert '# comment inside plugin' in result - assert 'port => 5044' in result - - def test_preserves_standalone_comment_inside_conditional(self): - """Standalone comment lines inside an if/else block are preserved""" - config = ( - 'filter {\n' - ' if [type] == "syslog" {\n' - ' # comment in conditional\n' - ' mutate {}\n' - ' }\n' - '}\n' - ) - result = _strip_inline_comments(config) - assert '# comment in conditional' in result - - def test_returns_string(self): - """Function returns a string""" - assert isinstance(_strip_inline_comments('input { stdin {} }'), str) - - def test_no_comments_unchanged(self): - """Config with no comments is returned unchanged""" - config = 'input {\n stdin {}\n}\n' - result = _strip_inline_comments(config) - assert result == config - - def test_trailing_whitespace_before_comment_removed(self): - """Trailing whitespace before the inline comment is also stripped""" - config = 'input {\n beats {\n port => 5044 # trailing spaces before comment\n }\n}\n' - result = _strip_inline_comments(config) - # The value should be present without trailing spaces - lines_with_port = [l for l in result.split('\n') if 'port' in l] - assert len(lines_with_port) == 1 - assert lines_with_port[0].endswith('5044') - - -# ───────────────────────────────────────────────────────────── -# parse_logstash_config -# ───────────────────────────────────────────────────────────── - -class TestParseLogstashConfig: - """Unit tests for parse_logstash_config - - Note on return type: Lark's LALR transformer with a single section returns - a bare dict (not a list). With multiple sections it returns a list of dicts. - This matches the defensive check inside logstash_config_to_components: - if not isinstance(parsed, list): parsed = [parsed] - The helper _as_list() below mirrors that normalization. - """ - - @staticmethod - def _as_list(result): - """Normalize to list regardless of whether one or multiple sections came back.""" - if isinstance(result, list): - return result - return [result] if result else [] - - def test_parses_single_section_as_dict_or_list(self): - """Single-section configs are returned as a dict (Lark unwraps single items)""" - result = parse_logstash_config(MINIMAL_INPUT) - assert isinstance(result, (dict, list)) - - def test_parses_all_three_sections(self): - """All three sections (input, filter, output) are parsed into a list""" - result = parse_logstash_config(FULL_PIPELINE) - assert isinstance(result, list) - types = [s['type'] for s in result] - assert 'input' in types - assert 'filter' in types - assert 'output' in types - - def test_single_section_has_correct_type(self): - """A single input section has type == 'input'""" - result = self._as_list(parse_logstash_config(MINIMAL_INPUT)) - assert len(result) == 1 - assert result[0]['type'] == 'input' - - def test_each_section_has_statements(self): - """Each parsed section has a 'statements' key""" - result = self._as_list(parse_logstash_config(MINIMAL_INPUT)) - for section in result: - assert 'statements' in section - - def test_plugin_name_captured(self): - """The plugin name ('stdin') is captured correctly""" - result = self._as_list(parse_logstash_config(MINIMAL_INPUT)) - statements = result[0]['statements'] - plugin_names = [str(s.get('name', '')) for s in statements] - assert 'stdin' in plugin_names - - def test_invalid_config_raises_value_error(self): - """Invalid config raises a ValueError""" - bad_config = "this is not valid logstash config {{{" - with pytest.raises(ValueError): - parse_logstash_config(bad_config) - - def test_error_message_includes_line_info(self): - """ValueError message contains line number information""" - bad_config = "input {\n ??? invalid\n}\n" - with pytest.raises(ValueError, match=r'line'): - parse_logstash_config(bad_config) - - def test_plugin_with_settings(self): - """Settings on a plugin are parsed into the settings dict""" - config = 'input {\n beats {\n port => 5044\n }\n}\n' - result = self._as_list(parse_logstash_config(config)) - statements = result[0]['statements'] - beats = next(s for s in statements if str(s.get('name', '')) == 'beats') - assert beats['settings']['port'] == 5044 - - def test_plugin_with_string_setting(self): - """String settings are unquoted during parsing""" - config = 'input {\n file {\n path => "/var/log/syslog"\n }\n}\n' - result = self._as_list(parse_logstash_config(config)) - statements = result[0]['statements'] - file_plugin = next(s for s in statements if str(s.get('name', '')) == 'file') - assert file_plugin['settings']['path'] == '/var/log/syslog' - - def test_plugin_with_numeric_setting(self): - """Numeric settings are parsed correctly (not checked as list here)""" - config = 'input {\n file {\n sincedb_clean_after => 0\n start_position => "beginning"\n }\n}\n' - result = self._as_list(parse_logstash_config(config)) - assert len(result) == 1 - assert result[0]['type'] == 'input' - - def test_empty_plugin_block(self): - """A plugin with no settings produces an empty settings dict""" - result = self._as_list(parse_logstash_config(MINIMAL_INPUT)) - statements = result[0]['statements'] - stdin = next(s for s in statements if str(s.get('name', '')) == 'stdin') - assert stdin['settings'] == {} - - def test_strips_inline_comments_before_parsing(self): - """Inline comments that would break parsing are stripped first""" - config = 'input {\n beats {\n port => 5044 # inline comment\n }\n}\n' - # Should not raise — comments are stripped before parsing - result = self._as_list(parse_logstash_config(config)) - assert len(result) == 1 - - -# ───────────────────────────────────────────────────────────── -# logstash_config_to_components -# ───────────────────────────────────────────────────────────── - -class TestLogstashConfigToComponents: - """Unit tests for logstash_config_to_components""" - - def test_returns_json_string(self): - """Returns a JSON string""" - result = logstash_config_to_components(FULL_PIPELINE) - assert isinstance(result, str) - # Should be valid JSON - parsed = json.loads(result) - assert isinstance(parsed, dict) - - def test_output_has_three_sections(self): - """Output JSON has input, filter, output keys""" - result = json.loads(logstash_config_to_components(FULL_PIPELINE)) - assert 'input' in result - assert 'filter' in result - assert 'output' in result - - def test_all_sections_present_even_when_missing(self): - """All three sections are present even when only some exist in config""" - result = json.loads(logstash_config_to_components(MINIMAL_INPUT)) - assert 'input' in result - assert 'filter' in result - assert 'output' in result - assert result['filter'] == [] - assert result['output'] == [] - - def test_plugin_in_correct_section(self): - """Plugins end up in their correct section""" - result = json.loads(logstash_config_to_components(FULL_PIPELINE)) - input_plugins = [c['plugin'] for c in result['input']] - filter_plugins = [c['plugin'] for c in result['filter']] - output_plugins = [c['plugin'] for c in result['output']] - assert 'stdin' in input_plugins - assert 'mutate' in filter_plugins - assert 'stdout' in output_plugins - - def test_component_has_required_keys(self): - """Each component has id, type, plugin, and config keys""" - result = json.loads(logstash_config_to_components(FULL_PIPELINE)) - for section_components in result.values(): - for component in section_components: - assert 'id' in component - assert 'type' in component - assert 'plugin' in component - assert 'config' in component - - def test_component_type_matches_section(self): - """Each component's 'type' matches the section it belongs to""" - result = json.loads(logstash_config_to_components(FULL_PIPELINE)) - for section_name, section_components in result.items(): - for component in section_components: - assert component['type'] == section_name - - def test_component_id_includes_plugin_name(self): - """Component IDs include the plugin name""" - result = json.loads(logstash_config_to_components(MINIMAL_INPUT)) - stdin_component = result['input'][0] - assert 'stdin' in stdin_component['id'] - - def test_plugin_settings_in_config(self): - """Plugin settings appear in the component's config dict""" - config = 'input {\n beats {\n port => 5044\n }\n}\n' - result = json.loads(logstash_config_to_components(config)) - beats = result['input'][0] - assert beats['plugin'] == 'beats' - assert beats['config']['port'] == 5044 - - def test_invalid_config_raises_exception(self): - """Invalid config raises an Exception""" - with pytest.raises(Exception): - logstash_config_to_components("this is complete garbage {{{}}") - - def test_multiple_plugins_same_section(self): - """Multiple plugins in the same section are all captured""" - config = ( - 'input {\n' - ' stdin {}\n' - ' beats { port => 5044 }\n' - '}\n' - 'output { stdout {} }\n' - ) - result = json.loads(logstash_config_to_components(config)) - assert len(result['input']) == 2 - plugin_names = [c['plugin'] for c in result['input']] - assert 'stdin' in plugin_names - assert 'beats' in plugin_names - - def test_components_have_unique_ids(self): - """All component IDs are unique within the output""" - config = ( - 'input { stdin {} }\n' - 'filter { mutate {} grok {} }\n' - 'output { stdout {} }\n' - ) - result = json.loads(logstash_config_to_components(config)) - all_ids = [] - for section_components in result.values(): - all_ids.extend(c['id'] for c in section_components) - assert len(all_ids) == len(set(all_ids)), "Component IDs should be unique" - - def test_output_is_pretty_printed_json(self): - """Output JSON is indented (pretty-printed)""" - result = logstash_config_to_components(MINIMAL_INPUT) - assert '\n' in result - assert ' ' in result # 4-space indent - - -# ───────────────────────────────────────────────────────────── -# ComponentToPipeline -# ───────────────────────────────────────────────────────────── - -class TestComponentToPipeline: - """Unit tests for ComponentToPipeline helper methods""" - - def _make_parser(self, components=None): - if components is None: - components = {'input': [], 'filter': [], 'output': []} - return ComponentToPipeline(components) - - # _format_string_value tests - - def test_format_string_simple(self): - """Simple strings are quoted with double quotes""" - parser = self._make_parser() - result = parser._format_string_value('hello') - assert result == '"hello"' - - def test_format_string_non_string_passthrough(self): - """Non-string values are returned as-is""" - parser = self._make_parser() - assert parser._format_string_value(42) == 42 - assert parser._format_string_value(True) is True - assert parser._format_string_value(3.14) == 3.14 - - def test_format_string_with_double_quotes_uses_single(self): - """String containing double quotes is wrapped in single quotes""" - parser = self._make_parser() - result = parser._format_string_value('say "hello"') - assert result.startswith("'") - assert result.endswith("'") - - def test_format_string_with_single_quotes_uses_double(self): - """String containing single quotes is wrapped in double quotes""" - parser = self._make_parser() - result = parser._format_string_value("it's fine") - assert result.startswith('"') - assert result.endswith('"') - - def test_format_multiline_string_uses_single_quotes(self): - """Multiline string uses single quotes""" - parser = self._make_parser() - result = parser._format_string_value('line1\nline2') - assert result.startswith("'") - - # _generate_plugin_id tests - - def test_generate_plugin_id_format(self): - """Generated ID follows section_plugin_count format""" - parser = self._make_parser() - plugin_id = parser._generate_plugin_id('beats', 'input') - assert plugin_id == 'input_beats_1' - - def test_generate_plugin_id_increments(self): - """Each call increments the counter for the same plugin type""" - parser = self._make_parser() - id1 = parser._generate_plugin_id('beats', 'input') - id2 = parser._generate_plugin_id('beats', 'input') - assert id1 == 'input_beats_1' - assert id2 == 'input_beats_2' - - def test_generate_plugin_id_separate_counters_per_section(self): - """Different sections have independent counters""" - parser = self._make_parser() - input_id = parser._generate_plugin_id('stdout', 'input') - output_id = parser._generate_plugin_id('stdout', 'output') - assert input_id == 'input_stdout_1' - assert output_id == 'output_stdout_1' - - # components_to_logstash_config tests - - def test_empty_components_returns_empty_sections(self): - """Empty component lists produce an empty pipeline string""" - parser = self._make_parser({'input': [], 'filter': [], 'output': []}) - result = parser.components_to_logstash_config() - assert isinstance(result, str) - - def test_simple_stdin_plugin(self): - """A simple stdin plugin is rendered correctly""" - components = { - 'input': [ - {'id': 'input_stdin_0', 'type': 'input', 'plugin': 'stdin', 'config': {}} - ], - 'filter': [], - 'output': [] - } - parser = ComponentToPipeline(components) - result = parser.components_to_logstash_config() - assert 'input {' in result - assert 'stdin {' in result - - def test_plugin_with_string_setting(self): - """String settings are rendered with quotes""" - components = { - 'input': [ - {'id': 'input_file_0', 'type': 'input', 'plugin': 'file', - 'config': {'path': '/var/log/syslog'}} - ], - 'filter': [], - 'output': [] - } - parser = ComponentToPipeline(components) - result = parser.components_to_logstash_config() - assert 'path => "/var/log/syslog"' in result - - def test_plugin_with_numeric_setting(self): - """Numeric settings are rendered without quotes""" - components = { - 'input': [ - {'id': 'input_beats_0', 'type': 'input', 'plugin': 'beats', - 'config': {'port': 5044}} - ], - 'filter': [], - 'output': [] - } - parser = ComponentToPipeline(components) - result = parser.components_to_logstash_config() - assert 'port => 5044' in result - assert 'port => "5044"' not in result - - def test_all_three_sections_rendered(self): - """All three sections (input, filter, output) appear in the output""" - components = { - 'input': [{'id': 'i', 'type': 'input', 'plugin': 'stdin', 'config': {}}], - 'filter': [{'id': 'f', 'type': 'filter', 'plugin': 'mutate', 'config': {}}], - 'output': [{'id': 'o', 'type': 'output', 'plugin': 'stdout', 'config': {}}] - } - parser = ComponentToPipeline(components) - result = parser.components_to_logstash_config() - assert 'input {' in result - assert 'filter {' in result - assert 'output {' in result diff --git a/src/logstashui/Common/tests/test_logstash_utils.py b/src/logstashui/Common/tests/test_logstash_utils.py deleted file mode 100644 index b3a86c7..0000000 --- a/src/logstashui/Common/tests/test_logstash_utils.py +++ /dev/null @@ -1,163 +0,0 @@ -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. - -import pytest -from unittest.mock import Mock, patch, MagicMock - -from Common.logstash_utils import get_logstash_pipeline - - -class TestGetLogstashPipeline: - """Tests for get_logstash_pipeline function""" - - @patch('Common.logstash_utils.get_elastic_connection') - def test_returns_pipeline_document_on_success(self, mock_get_connection): - """Test that the pipeline document is returned when found""" - pipeline_name = 'my_pipeline' - expected_doc = { - 'pipeline': 'input { stdin {} } output { stdout {} }', - 'last_modified': '2024-01-01T00:00:00Z', - 'username': 'elastic' - } - - mock_es = Mock() - mock_es.logstash.get_pipeline.return_value = {pipeline_name: expected_doc} - mock_get_connection.return_value = mock_es - - result = get_logstash_pipeline(es_id=1, pipeline_name=pipeline_name) - - assert result == expected_doc - - @patch('Common.logstash_utils.get_elastic_connection') - def test_calls_get_elastic_connection_with_correct_id(self, mock_get_connection): - """Test that get_elastic_connection is called with the provided es_id""" - mock_es = Mock() - mock_es.logstash.get_pipeline.return_value = {'pipe': {'pipeline': 'input {}'}} - mock_get_connection.return_value = mock_es - - get_logstash_pipeline(es_id=42, pipeline_name='pipe') - - mock_get_connection.assert_called_once_with(42) - - @patch('Common.logstash_utils.get_elastic_connection') - def test_calls_get_pipeline_with_correct_name(self, mock_get_connection): - """Test that logstash.get_pipeline is called with the correct pipeline name""" - pipeline_name = 'target_pipeline' - mock_es = Mock() - mock_es.logstash.get_pipeline.return_value = {pipeline_name: {'pipeline': ''}} - mock_get_connection.return_value = mock_es - - get_logstash_pipeline(es_id=1, pipeline_name=pipeline_name) - - mock_es.logstash.get_pipeline.assert_called_once_with(id=pipeline_name) - - @patch('Common.logstash_utils.get_elastic_connection') - def test_returns_none_on_key_error(self, mock_get_connection): - """Test that None is returned when pipeline name not found in response (KeyError)""" - mock_es = Mock() - # Pipeline name not in response dict → KeyError - mock_es.logstash.get_pipeline.return_value = {'other_pipeline': {}} - mock_get_connection.return_value = mock_es - - result = get_logstash_pipeline(es_id=1, pipeline_name='nonexistent_pipeline') - - assert result is None - - @patch('Common.logstash_utils.get_elastic_connection') - def test_returns_none_on_connection_error(self, mock_get_connection): - """Test that None is returned when an exception occurs during connection""" - mock_get_connection.side_effect = Exception("Connection refused") - - result = get_logstash_pipeline(es_id=1, pipeline_name='my_pipeline') - - assert result is None - - @patch('Common.logstash_utils.get_elastic_connection') - def test_returns_none_on_api_error(self, mock_get_connection): - """Test that None is returned when the ES API call raises an exception""" - mock_es = Mock() - mock_es.logstash.get_pipeline.side_effect = Exception("API Error: 500") - mock_get_connection.return_value = mock_es - - result = get_logstash_pipeline(es_id=1, pipeline_name='my_pipeline') - - assert result is None - - @patch('Common.logstash_utils.get_elastic_connection') - def test_logs_error_on_key_error(self, mock_get_connection, caplog): - """Test that an error is logged when pipeline is not found (KeyError)""" - pipeline_name = 'missing_pipeline' - mock_es = Mock() - mock_es.logstash.get_pipeline.return_value = {} # Missing key - mock_get_connection.return_value = mock_es - - get_logstash_pipeline(es_id=5, pipeline_name=pipeline_name) - - assert pipeline_name in caplog.text - - @patch('Common.logstash_utils.get_elastic_connection') - def test_logs_error_on_generic_exception(self, mock_get_connection, caplog): - """Test that an error is logged when a generic exception occurs""" - pipeline_name = 'error_pipeline' - mock_es = Mock() - mock_es.logstash.get_pipeline.side_effect = ConnectionError("Timeout") - mock_get_connection.return_value = mock_es - - get_logstash_pipeline(es_id=3, pipeline_name=pipeline_name) - - assert pipeline_name in caplog.text - - @patch('Common.logstash_utils.get_elastic_connection') - def test_returns_full_pipeline_document_structure(self, mock_get_connection): - """Test that the full pipeline document structure is preserved""" - pipeline_name = 'complex_pipeline' - expected_doc = { - 'pipeline': 'input { beats { port => 5044 } } output { elasticsearch {} }', - 'last_modified': '2024-03-01T12:00:00Z', - 'username': 'kibana_system', - 'metadata': { - 'type': 'logstash_pipeline', - 'version': 1 - } - } - - mock_es = Mock() - mock_es.logstash.get_pipeline.return_value = {pipeline_name: expected_doc} - mock_get_connection.return_value = mock_es - - result = get_logstash_pipeline(es_id=1, pipeline_name=pipeline_name) - - assert result == expected_doc - assert result['pipeline'] == expected_doc['pipeline'] - assert result['username'] == 'kibana_system' - - @patch('Common.logstash_utils.get_elastic_connection') - def test_handles_multiple_pipelines_in_response(self, mock_get_connection): - """Test correct pipeline is returned when response contains multiple pipelines""" - target_name = 'target' - target_doc = {'pipeline': 'input { stdin {} }'} - other_doc = {'pipeline': 'input { beats {} }'} - - mock_es = Mock() - mock_es.logstash.get_pipeline.return_value = { - target_name: target_doc, - 'other': other_doc - } - mock_get_connection.return_value = mock_es - - result = get_logstash_pipeline(es_id=1, pipeline_name=target_name) - - assert result == target_doc - assert result != other_doc - - @patch('Common.logstash_utils.get_elastic_connection') - def test_handles_empty_pipeline_response(self, mock_get_connection): - """Test returns None when response is completely empty dict""" - mock_es = Mock() - mock_es.logstash.get_pipeline.return_value = {} - mock_get_connection.return_value = mock_es - - result = get_logstash_pipeline(es_id=1, pipeline_name='any_pipeline') - - assert result is None diff --git a/src/logstashui/Common/tests/test_middleware.py b/src/logstashui/Common/tests/test_middleware.py deleted file mode 100644 index 81e5fd0..0000000 --- a/src/logstashui/Common/tests/test_middleware.py +++ /dev/null @@ -1,242 +0,0 @@ -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. - -import pytest -from django.test import RequestFactory -from django.http import HttpResponse - -from Common.middleware import SecurityHeadersMiddleware - - -@pytest.fixture -def request_factory(): - """Django RequestFactory for creating mock requests""" - return RequestFactory() - - -@pytest.fixture -def mock_request(request_factory): - """Create a basic GET mock request""" - return request_factory.get('/') - - -@pytest.fixture -def middleware(): - """Create middleware instance with a simple get_response callable""" - def get_response(request): - return HttpResponse("OK", status=200) - - return SecurityHeadersMiddleware(get_response) - - -@pytest.fixture -def middleware_with_custom_response(): - """Factory fixture to create middleware with a custom response""" - def factory(response): - return SecurityHeadersMiddleware(lambda r: response) - return factory - - -class TestSecurityHeadersMiddlewareInit: - """Tests for SecurityHeadersMiddleware initialization""" - - def test_middleware_stores_get_response(self): - """Test that middleware stores the get_response callable""" - def get_response(request): - return HttpResponse("OK") - - mw = SecurityHeadersMiddleware(get_response) - assert mw.get_response is get_response - - def test_middleware_callable(self, middleware, mock_request): - """Test that middleware is callable and returns a response""" - response = middleware(mock_request) - assert response is not None - assert response.status_code == 200 - - -class TestContentSecurityPolicy: - """Tests for the Content-Security-Policy header""" - - def test_csp_header_is_set(self, middleware, mock_request): - """Test that CSP header is present in the response""" - response = middleware(mock_request) - assert 'Content-Security-Policy' in response - - def test_csp_default_src_self(self, middleware, mock_request): - """Test that default-src is restricted to self""" - response = middleware(mock_request) - csp = response['Content-Security-Policy'] - assert "default-src 'self'" in csp - - def test_csp_script_src_includes_self(self, middleware, mock_request): - """Test that script-src includes self""" - response = middleware(mock_request) - csp = response['Content-Security-Policy'] - assert "script-src 'self'" in csp - - def test_csp_script_src_allows_unsafe_inline(self, middleware, mock_request): - """Test that script-src allows unsafe-inline (required for htmx)""" - response = middleware(mock_request) - csp = response['Content-Security-Policy'] - # Need unsafe-inline for htmx and dynamic JS - assert "'unsafe-inline'" in csp - - def test_csp_style_src_allows_unsafe_inline(self, middleware, mock_request): - """Test that style-src allows unsafe-inline (required for Tailwind)""" - response = middleware(mock_request) - csp = response['Content-Security-Policy'] - assert "style-src 'self' 'unsafe-inline'" in csp - - def test_csp_img_src_allows_data_and_https(self, middleware, mock_request): - """Test that img-src allows data URIs and HTTPS sources""" - response = middleware(mock_request) - csp = response['Content-Security-Policy'] - assert "img-src 'self' data: https:" in csp - - def test_csp_font_src_allows_data(self, middleware, mock_request): - """Test that font-src allows data URIs""" - response = middleware(mock_request) - csp = response['Content-Security-Policy'] - assert "font-src 'self' data:" in csp - - def test_csp_connect_src_self_only(self, middleware, mock_request): - """Test that connect-src is restricted to self""" - response = middleware(mock_request) - csp = response['Content-Security-Policy'] - assert "connect-src 'self'" in csp - - def test_csp_frame_src_includes_elastic(self, middleware, mock_request): - """Test that frame-src includes elastic.co documentation domains""" - response = middleware(mock_request) - csp = response['Content-Security-Policy'] - assert "https://www.elastic.co" in csp - assert "https://elastic.co" in csp - - def test_csp_frame_src_includes_github(self, middleware, mock_request): - """Test that frame-src includes github.com for plugin docs""" - response = middleware(mock_request) - csp = response['Content-Security-Policy'] - assert "https://github.com" in csp - - def test_csp_frame_src_includes_rubydoc(self, middleware, mock_request): - """Test that frame-src includes rubydoc.info for plugin docs""" - response = middleware(mock_request) - csp = response['Content-Security-Policy'] - assert "https://rubydoc.info" in csp - - def test_csp_frame_ancestors_self_only(self, middleware, mock_request): - """Test that frame-ancestors restricts framing to same origin (anti-clickjacking)""" - response = middleware(mock_request) - csp = response['Content-Security-Policy'] - assert "frame-ancestors 'self'" in csp - - def test_csp_directives_separated_by_semicolons(self, middleware, mock_request): - """Test that CSP directives are joined with semicolons""" - response = middleware(mock_request) - csp = response['Content-Security-Policy'] - assert '; ' in csp - - -class TestAdditionalSecurityHeaders: - """Tests for X-Content-Type-Options and Referrer-Policy headers""" - - def test_x_content_type_options_nosniff(self, middleware, mock_request): - """Test that X-Content-Type-Options is set to nosniff""" - response = middleware(mock_request) - assert response['X-Content-Type-Options'] == 'nosniff' - - def test_referrer_policy_set(self, middleware, mock_request): - """Test that Referrer-Policy header is set""" - response = middleware(mock_request) - assert 'Referrer-Policy' in response - - def test_referrer_policy_value(self, middleware, mock_request): - """Test Referrer-Policy has the correct value""" - response = middleware(mock_request) - assert response['Referrer-Policy'] == 'no-referrer-when-downgrade' - - -class TestMiddlewarePassthrough: - """Tests ensuring middleware doesn't break normal response behavior""" - - def test_middleware_preserves_response_status(self, request_factory): - """Test that middleware preserves the original response status code""" - def get_response(request): - return HttpResponse("Created", status=201) - - mw = SecurityHeadersMiddleware(get_response) - request = request_factory.get('/') - response = mw(request) - assert response.status_code == 201 - - def test_middleware_preserves_response_body(self, request_factory): - """Test that middleware preserves the original response body""" - def get_response(request): - return HttpResponse("Hello World") - - mw = SecurityHeadersMiddleware(get_response) - request = request_factory.get('/') - response = mw(request) - assert b'Hello World' in response.content - - def test_middleware_does_not_remove_existing_headers(self, request_factory): - """Test that middleware does not remove headers already set by the view""" - def get_response(request): - resp = HttpResponse("OK") - resp['X-Custom-Header'] = 'custom-value' - return resp - - mw = SecurityHeadersMiddleware(get_response) - request = request_factory.get('/') - response = mw(request) - assert response['X-Custom-Header'] == 'custom-value' - - def test_middleware_works_with_post_request(self, request_factory): - """Test that middleware applies headers to POST requests too""" - def get_response(request): - return HttpResponse("Posted", status=200) - - mw = SecurityHeadersMiddleware(get_response) - request = request_factory.post('/submit/') - response = mw(request) - assert 'Content-Security-Policy' in response - assert response['X-Content-Type-Options'] == 'nosniff' - - def test_middleware_works_with_json_response(self, request_factory): - """Test that middleware works with JSON responses""" - from django.http import JsonResponse - - def get_response(request): - return JsonResponse({'key': 'value'}) - - mw = SecurityHeadersMiddleware(get_response) - request = request_factory.get('/api/data/') - response = mw(request) - assert 'Content-Security-Policy' in response - assert response['X-Content-Type-Options'] == 'nosniff' - - def test_middleware_applies_to_error_responses(self, request_factory): - """Test that security headers are added even to error responses""" - def get_response(request): - return HttpResponse("Not Found", status=404) - - mw = SecurityHeadersMiddleware(get_response) - request = request_factory.get('/missing/') - response = mw(request) - assert response.status_code == 404 - assert 'Content-Security-Policy' in response - - def test_headers_applied_on_every_request(self, request_factory): - """Test that headers are applied on every call (not just first)""" - def get_response(request): - return HttpResponse("OK") - - mw = SecurityHeadersMiddleware(get_response) - - for _ in range(3): - request = request_factory.get('/') - response = mw(request) - assert 'Content-Security-Policy' in response - assert response['X-Content-Type-Options'] == 'nosniff' diff --git a/src/logstashui/Common/tests/test_pipeline_to_components.py b/src/logstashui/Common/tests/test_pipeline_to_components.py deleted file mode 100644 index 7b61c58..0000000 --- a/src/logstashui/Common/tests/test_pipeline_to_components.py +++ /dev/null @@ -1,51 +0,0 @@ -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. - -from Common.logstash_config_parse import logstash_config_to_components -import pytest -import json -import os - -# Load test cases from external files -def load_test_cases(): - """Load test cases from conversion_data directory.""" - base_dir = os.path.dirname(os.path.abspath(__file__)) - pipelines_dir = os.path.join(base_dir, "conversion_data", "pipelines") - components_dir = os.path.join(base_dir, "conversion_data", "components") - - test_cases = [] - - # Get all .conf files - for filename in sorted(os.listdir(pipelines_dir)): - if filename.endswith('.conf'): - name = filename[:-5] # Remove .conf extension - - # Load pipeline config - pipeline_file = os.path.join(pipelines_dir, filename) - with open(pipeline_file, 'r', encoding='utf-8') as f: - pipeline = f.read() - - # Load components JSON - components_file = os.path.join(components_dir, f"{name}.json") - with open(components_file, 'r', encoding='utf-8') as f: - components_json = json.load(f) - # Convert back to JSON string for comparison - components = json.dumps(components_json, indent=4) - - test_cases.append((name, pipeline, components)) - - return test_cases - -test_cases = load_test_cases() - -@pytest.mark.parametrize( - "name, pipeline, components", - test_cases, - ids=[case[0] for case in test_cases] -) -def test_pipeline_to_components(name, pipeline, components): - assert logstash_config_to_components(pipeline) == components - - - diff --git a/src/logstashui/Common/tests/test_product_ca.py b/src/logstashui/Common/tests/test_product_ca.py deleted file mode 100644 index 933ff36..0000000 --- a/src/logstashui/Common/tests/test_product_ca.py +++ /dev/null @@ -1,334 +0,0 @@ -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. - -"""Tests for product CA generation and enrollment token payload.""" - -import hashlib -import ipaddress -import socket - -import pytest -from cryptography import x509 -from cryptography.hazmat.primitives import serialization -from django.test import override_settings - - -@pytest.mark.django_db -def test_ensure_product_ca_generates_and_fingerprints(tmp_path, settings): - from Common import product_ca - - # Isolate CA storage - product_ca._cached_cert_pem = None - product_ca._cached_fingerprint = None - settings.BASE_DIR = tmp_path - settings.DATA_DIR = tmp_path / "data" - settings.DATA_DIR.mkdir(exist_ok=True) - - pem1, fp1 = product_ca.ensure_product_ca() - assert b"BEGIN CERTIFICATE" in pem1 - assert len(fp1) == 64 - cert = x509.load_pem_x509_certificate(pem1) - der = cert.public_bytes(serialization.Encoding.DER) - assert hashlib.sha256(der).hexdigest() == fp1 - - # Second call is cached / reloads same files - product_ca._cached_cert_pem = None - product_ca._cached_fingerprint = None - pem2, fp2 = product_ca.ensure_product_ca() - assert fp1 == fp2 - assert pem1 == pem2 - - -@pytest.mark.django_db -def test_build_enrollment_token_payload_includes_fingerprint(tmp_path, settings): - from Common import product_ca - - product_ca._cached_cert_pem = None - product_ca._cached_fingerprint = None - settings.BASE_DIR = tmp_path - settings.DATA_DIR = tmp_path / "data" - settings.DATA_DIR.mkdir(exist_ok=True) - settings.LOGSTASHUI_CONFIG = { - "agent": {"include_ca_fingerprint": True}, - } - - payload = product_ca.build_enrollment_token_payload("secret-token") - assert payload["enrollment_token"] == "secret-token" - assert payload["token_version"] == 2 - assert "fingerprint" in payload - assert len(payload["fingerprint"]) == 64 - assert "ui_url" not in payload - - -@pytest.mark.django_db -def test_build_enrollment_token_payload_omits_fingerprint(tmp_path, settings): - from Common import product_ca - - product_ca._cached_cert_pem = None - product_ca._cached_fingerprint = None - settings.BASE_DIR = tmp_path - settings.DATA_DIR = tmp_path / "data" - settings.DATA_DIR.mkdir(exist_ok=True) - settings.LOGSTASHUI_CONFIG = { - "agent": {"include_ca_fingerprint": False}, - } - - payload = product_ca.build_enrollment_token_payload("secret-token") - assert "fingerprint" not in payload - - -def test_product_ca_endpoint(client, tmp_path, settings): - from Common import product_ca - - product_ca._cached_cert_pem = None - product_ca._cached_fingerprint = None - settings.BASE_DIR = tmp_path - settings.DATA_DIR = tmp_path / "data" - settings.DATA_DIR.mkdir(exist_ok=True) - - resp = client.get("/.well-known/logstashui/ca.crt") - assert resp.status_code == 200 - assert b"BEGIN CERTIFICATE" in resp.content - - -@pytest.mark.django_db -def test_default_ui_server_cert_includes_compose_sans(tmp_path, settings, monkeypatch): - """Product default leaf must cover localhost and logstashui service name.""" - from Common import product_ca - - product_ca._cached_cert_pem = None - product_ca._cached_fingerprint = None - settings.BASE_DIR = tmp_path - settings.DATA_DIR = tmp_path / "data" - settings.DATA_DIR.mkdir(exist_ok=True) - monkeypatch.delenv("LOGSTASHUI_TLS_SANS", raising=False) - monkeypatch.delenv("LOGSTASHUI_HOST_HOSTNAME", raising=False) - monkeypatch.delenv("LOGSTASHUI_HOST_IPS", raising=False) - - cert_path, key_path = product_ca.ensure_default_ui_server_cert() - assert cert_path.is_file() - assert key_path.is_file() - leaf = x509.load_pem_x509_certificate(cert_path.read_bytes()) - ext = leaf.extensions.get_extension_for_class(x509.SubjectAlternativeName) - dns = {n.value for n in ext.value if isinstance(n, x509.DNSName)} - assert "localhost" in dns - assert "logstashui" in dns - assert product_ca.get_ui_server_mode() == "product" - - -@pytest.mark.django_db -def test_ui_cert_includes_host_ips_and_reissues_on_san_change(tmp_path, settings, monkeypatch): - from Common import product_ca - import ipaddress - - product_ca._cached_cert_pem = None - product_ca._cached_fingerprint = None - settings.BASE_DIR = tmp_path - settings.DATA_DIR = tmp_path / "data" - settings.DATA_DIR.mkdir(exist_ok=True) - settings.LOGSTASHUI_CONFIG = {} - - monkeypatch.setenv("LOGSTASHUI_HOST_HOSTNAME", "docker-host.example") - monkeypatch.setenv("LOGSTASHUI_HOST_IPS", "10.20.30.40,10.20.30.41") - monkeypatch.delenv("LOGSTASHUI_TLS_SANS", raising=False) - - product_ca.ensure_default_ui_server_cert() - leaf = x509.load_pem_x509_certificate(product_ca.ui_server_cert_path().read_bytes()) - ext = leaf.extensions.get_extension_for_class(x509.SubjectAlternativeName) - dns = {n.value for n in ext.value if isinstance(n, x509.DNSName)} - ips = {n.value for n in ext.value if isinstance(n, x509.IPAddress)} - assert "docker-host.example" in dns - assert ipaddress.IPv4Address("10.20.30.40") in ips - assert ipaddress.IPv4Address("10.20.30.41") in ips - fp1 = product_ca.fingerprint_sha256_der(leaf) - - # New host IP → must re-issue - monkeypatch.setenv("LOGSTASHUI_HOST_IPS", "10.20.30.40,10.20.30.41,10.20.30.99") - assert product_ca.product_ui_cert_needs_reissue() is True - product_ca.ensure_default_ui_server_cert() - leaf2 = x509.load_pem_x509_certificate(product_ca.ui_server_cert_path().read_bytes()) - ips2 = { - n.value - for n in leaf2.extensions.get_extension_for_class(x509.SubjectAlternativeName).value - if isinstance(n, x509.IPAddress) - } - assert ipaddress.IPv4Address("10.20.30.99") in ips2 - fp2 = product_ca.fingerprint_sha256_der(leaf2) - assert fp1 != fp2 - - -def test_reverse_lookup_fqdns_filters_short_and_ip(monkeypatch): - from Common import product_ca - - def _fake(ip): - if ip == "10.0.0.5": - return ("host.example.com.", ["alias.example.com"], ["10.0.0.5"]) - if ip == "10.0.0.6": - return ("shortname", [], ["10.0.0.6"]) - raise socket.herror("no PTR") - - monkeypatch.setattr(product_ca.socket, "gethostbyaddr", _fake) - assert product_ca._reverse_lookup_fqdns("10.0.0.5") == [ - "host.example.com", - "alias.example.com", - ] - assert product_ca._reverse_lookup_fqdns("10.0.0.6") == [] - assert product_ca._reverse_lookup_fqdns("10.0.0.7") == [] - - -def test_prefer_ptr_fqdns_over_short_hostnames(monkeypatch): - from Common import product_ca - - def _fake(ip): - if ip == "10.9.5.31": - return ("mac.untergeek.dev", [], [ip]) - raise socket.herror("no PTR") - - monkeypatch.setattr(product_ca.socket, "gethostbyaddr", _fake) - - dns = ["localhost", "logstashui", "Palpatine"] - ips = [ - ipaddress.IPv4Address("127.0.0.1"), - ipaddress.IPv4Address("10.9.5.31"), - ] - product_ca._prefer_ptr_fqdns_over_short_hostnames(dns, ips) - assert "mac.untergeek.dev" in dns - assert "localhost" in dns - assert "logstashui" in dns - assert "Palpatine" not in dns - - -def test_collect_desired_ui_sans_uses_ptr_over_bare_hostname(monkeypatch, settings): - from Common import product_ca - - settings.LOGSTASHUI_CONFIG = {} - monkeypatch.setenv("LOGSTASHUI_HOST_HOSTNAME", "Palpatine") - monkeypatch.setenv("LOGSTASHUI_HOST_IPS", "172.19.7.21") - monkeypatch.delenv("LOGSTASHUI_TLS_SANS", raising=False) - - def _fake(ip): - if ip == "172.19.7.21": - return ("palpatine.untergeek.net", [], [ip]) - raise socket.herror("no PTR") - - monkeypatch.setattr(product_ca.socket, "gethostbyaddr", _fake) - # Avoid noise from local interface discovery / getfqdn - monkeypatch.setattr(product_ca.socket, "gethostname", lambda: "Palpatine") - monkeypatch.setattr(product_ca.socket, "getfqdn", lambda: "Palpatine") - monkeypatch.setattr(product_ca.socket, "getaddrinfo", lambda *a, **k: []) - # Skip UDP outbound-IP trick (would discover real LAN IPs and PTR them) - monkeypatch.setattr( - product_ca.socket, - "socket", - lambda *a, **k: (_ for _ in ()).throw(OSError("skip")), - ) - - dns, ips = product_ca.collect_desired_ui_sans() - assert "palpatine.untergeek.net" in dns - assert "Palpatine" not in dns - assert "localhost" in dns - assert "logstashui" in dns - assert ipaddress.IPv4Address("172.19.7.21") in ips - - -@pytest.mark.django_db -def test_sign_agent_csr(tmp_path, settings): - from Common import product_ca - from cryptography.hazmat.primitives.asymmetric import rsa - from cryptography.hazmat.primitives import hashes, serialization - from cryptography.x509.oid import NameOID - from cryptography import x509 - from datetime import datetime, timedelta, timezone - - product_ca._cached_cert_pem = None - product_ca._cached_fingerprint = None - settings.BASE_DIR = tmp_path - settings.DATA_DIR = tmp_path / "data" - settings.DATA_DIR.mkdir(exist_ok=True) - - key = rsa.generate_private_key(public_exponent=65537, key_size=2048) - csr = ( - x509.CertificateSigningRequestBuilder() - .subject_name(x509.Name([x509.NameAttribute(NameOID.COMMON_NAME, "agent1")])) - .add_extension( - x509.SubjectAlternativeName([ - x509.DNSName("agent1"), - x509.DNSName("localhost"), - ]), - critical=False, - ) - .sign(key, hashes.SHA256()) - ) - signed = product_ca.sign_agent_csr(csr.public_bytes(serialization.Encoding.PEM)) - assert "BEGIN CERTIFICATE" in signed["certificate_pem"] - assert "BEGIN CERTIFICATE" in signed["ca_pem"] - leaf = x509.load_pem_x509_certificate(signed["certificate_pem"].encode()) - assert leaf.subject.get_attributes_for_oid(NameOID.COMMON_NAME)[0].value == "agent1" - - -@pytest.mark.django_db -def test_custom_ui_cert_and_revert(tmp_path, settings): - from Common import product_ca - from cryptography.hazmat.primitives.asymmetric import rsa - from cryptography import x509 - from cryptography.hazmat.primitives import hashes, serialization - from cryptography.x509.oid import NameOID - from datetime import datetime, timedelta, timezone - - product_ca._cached_cert_pem = None - product_ca._cached_fingerprint = None - settings.BASE_DIR = tmp_path - settings.DATA_DIR = tmp_path / "data" - settings.DATA_DIR.mkdir(exist_ok=True) - - # Self-signed standalone cert (simulates public/custom CA leaf) - key = rsa.generate_private_key(public_exponent=65537, key_size=2048) - subject = x509.Name([x509.NameAttribute(NameOID.COMMON_NAME, "custom.example")]) - now = datetime.now(timezone.utc) - cert = ( - x509.CertificateBuilder() - .subject_name(subject) - .issuer_name(subject) - .public_key(key.public_key()) - .serial_number(x509.random_serial_number()) - .not_valid_before(now - timedelta(minutes=1)) - .not_valid_after(now + timedelta(days=30)) - .add_extension( - x509.SubjectAlternativeName([x509.DNSName("custom.example")]), - critical=False, - ) - .sign(key, hashes.SHA256()) - ) - cert_pem = cert.public_bytes(serialization.Encoding.PEM) - key_pem = key.private_bytes( - encoding=serialization.Encoding.PEM, - format=serialization.PrivateFormat.TraditionalOpenSSL, - encryption_algorithm=serialization.NoEncryption(), - ) - - info = product_ca.save_custom_ui_certificate(cert_pem, key_pem) - assert info["mode"] == "custom" - assert info["subject_cn"] == "custom.example" - assert product_ca.get_ui_server_mode() == "custom" - - status = product_ca.revert_ui_certificate_to_product_default() - assert status["mode"] == "product" - assert product_ca.ui_server_cert_path().is_file() - - -@pytest.mark.django_db -def test_settings_saves_agent_ui_url(admin_client): - from Management.models import Settings - - # Ensure admin has profile role admin (signal creates admin by default) - resp = admin_client.post( - "/Management/Settings/", - {"experimental_mode": "on", "agent_ui_url": "https://10.0.0.5:8443"}, - ) - assert resp.status_code == 200 - data = resp.json() - assert data["success"] is True - s = Settings.get_settings() - assert s.agent_ui_url == "https://10.0.0.5:8443" - assert s.experimental_mode is True diff --git a/src/logstashui/Common/tests/test_validators.py b/src/logstashui/Common/tests/test_validators.py deleted file mode 100644 index 3c003a1..0000000 --- a/src/logstashui/Common/tests/test_validators.py +++ /dev/null @@ -1,175 +0,0 @@ -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. - -import pytest -from Common.validators import validate_pipeline_name - - -class TestValidatePipelineName: - """Test validate_pipeline_name function with parametrized tests""" - - @pytest.mark.parametrize("pipeline_name,expected_valid", [ - ("valid_pipeline", True), - ("ValidPipeline", True), - ("_underscore_start", True), - ("pipeline123", True), - ("pipeline_with_dash-123", True), - ("a", True), - ("_", True), - ("Pipeline_Name_123", True), - ("my-pipeline-name", True), - ("my_pipeline_name", True), - ("ABC123_test-pipeline", True), - ]) - def test_valid_pipeline_names(self, pipeline_name, expected_valid): - """Test valid pipeline names""" - is_valid, error_message = validate_pipeline_name(pipeline_name) - assert is_valid == expected_valid - assert error_message is None - - @pytest.mark.parametrize("pipeline_name,expected_error_fragment", [ - ("", "cannot be empty"), - ("123pipeline", "must begin with a letter or underscore"), - ("-pipeline", "must begin with a letter or underscore"), - ("pipeline name", "must begin with a letter or underscore"), - ("pipeline@name", "must begin with a letter or underscore"), - ("pipeline.name", "must begin with a letter or underscore"), - ("pipeline$name", "must begin with a letter or underscore"), - ("pipeline#name", "must begin with a letter or underscore"), - ("pipeline!name", "must begin with a letter or underscore"), - ("pipeline*name", "must begin with a letter or underscore"), - ("pipeline(name)", "must begin with a letter or underscore"), - ("pipeline[name]", "must begin with a letter or underscore"), - ("pipeline{name}", "must begin with a letter or underscore"), - ("pipeline/name", "must begin with a letter or underscore"), - ("pipeline\\name", "must begin with a letter or underscore"), - ("pipeline:name", "must begin with a letter or underscore"), - ("pipeline;name", "must begin with a letter or underscore"), - ("pipeline,name", "must begin with a letter or underscore"), - ("pipeline", "must begin with a letter or underscore"), - ("pipeline?name", "must begin with a letter or underscore"), - ("pipeline|name", "must begin with a letter or underscore"), - ("pipeline~name", "must begin with a letter or underscore"), - ("pipeline`name", "must begin with a letter or underscore"), - ("pipeline'name", "must begin with a letter or underscore"), - ('pipeline"name', "must begin with a letter or underscore"), - ]) - def test_invalid_pipeline_names(self, pipeline_name, expected_error_fragment): - """Test invalid pipeline names""" - is_valid, error_message = validate_pipeline_name(pipeline_name) - assert is_valid is False - assert error_message is not None - assert expected_error_fragment in error_message.lower() - - def test_empty_string(self): - """Test empty string returns specific error""" - is_valid, error_message = validate_pipeline_name("") - assert is_valid is False - assert error_message == "Pipeline name cannot be empty" - - def test_none_value(self): - """Test None value is treated as empty""" - is_valid, error_message = validate_pipeline_name(None) - assert is_valid is False - assert error_message == "Pipeline name cannot be empty" - - def test_starts_with_number(self): - """Test pipeline name starting with number is invalid""" - is_valid, error_message = validate_pipeline_name("1pipeline") - assert is_valid is False - assert "must begin with a letter or underscore" in error_message - assert "[1pipeline]" in error_message - - def test_starts_with_dash(self): - """Test pipeline name starting with dash is invalid""" - is_valid, error_message = validate_pipeline_name("-pipeline") - assert is_valid is False - assert "must begin with a letter or underscore" in error_message - - def test_contains_space(self): - """Test pipeline name with spaces is invalid""" - is_valid, error_message = validate_pipeline_name("my pipeline") - assert is_valid is False - assert "must begin with a letter or underscore" in error_message - - def test_contains_special_characters(self): - """Test pipeline name with special characters is invalid""" - is_valid, error_message = validate_pipeline_name("pipeline@test") - assert is_valid is False - assert "must begin with a letter or underscore" in error_message - - def test_valid_with_all_allowed_characters(self): - """Test pipeline name with all allowed character types""" - is_valid, error_message = validate_pipeline_name("aZ_09-") - assert is_valid is True - assert error_message is None - - def test_single_letter(self): - """Test single letter is valid""" - is_valid, error_message = validate_pipeline_name("a") - assert is_valid is True - assert error_message is None - - def test_single_underscore(self): - """Test single underscore is valid""" - is_valid, error_message = validate_pipeline_name("_") - assert is_valid is True - assert error_message is None - - def test_long_pipeline_name(self): - """Test very long pipeline name is valid if format is correct""" - long_name = "a" * 1000 - is_valid, error_message = validate_pipeline_name(long_name) - assert is_valid is True - assert error_message is None - - def test_error_message_includes_pipeline_name(self): - """Test that error message includes the invalid pipeline name""" - invalid_name = "123invalid" - is_valid, error_message = validate_pipeline_name(invalid_name) - assert is_valid is False - assert invalid_name in error_message - - def test_unicode_characters_invalid(self): - """Test that unicode characters are invalid""" - is_valid, error_message = validate_pipeline_name("pipeline_中文") - assert is_valid is False - assert "must begin with a letter or underscore" in error_message - - def test_emoji_invalid(self): - """Test that emoji characters are invalid""" - is_valid, error_message = validate_pipeline_name("pipeline_🔥") - assert is_valid is False - assert "must begin with a letter or underscore" in error_message - - @pytest.mark.parametrize("valid_start", ["a", "Z", "_"]) - def test_valid_starting_characters(self, valid_start): - """Test all valid starting characters""" - is_valid, error_message = validate_pipeline_name(f"{valid_start}pipeline") - assert is_valid is True - assert error_message is None - - def test_consecutive_dashes_and_underscores(self): - """Test pipeline name with consecutive dashes and underscores""" - is_valid, error_message = validate_pipeline_name("pipeline__--__name") - assert is_valid is True - assert error_message is None - - def test_ends_with_dash(self): - """Test pipeline name ending with dash is valid""" - is_valid, error_message = validate_pipeline_name("pipeline-") - assert is_valid is True - assert error_message is None - - def test_ends_with_underscore(self): - """Test pipeline name ending with underscore is valid""" - is_valid, error_message = validate_pipeline_name("pipeline_") - assert is_valid is True - assert error_message is None - - def test_ends_with_number(self): - """Test pipeline name ending with number is valid""" - is_valid, error_message = validate_pipeline_name("pipeline123") - assert is_valid is True - assert error_message is None diff --git a/src/logstashui/LogstashUI/tests/test_cli.py b/src/logstashui/LogstashUI/tests/test_cli.py deleted file mode 100644 index ba63103..0000000 --- a/src/logstashui/LogstashUI/tests/test_cli.py +++ /dev/null @@ -1,230 +0,0 @@ -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. - -from argparse import Namespace - -from django.core.management.base import CommandError - -from LogstashUI.cli import _exec_gunicorn, build_parser, cmd_serve, install_systemd - - -def test_parser_defaults_to_serve(): - parser = build_parser() - ns = parser.parse_args([]) - assert ns.command == "serve" - - -def test_parser_manage_passthrough(): - parser = build_parser() - ns = parser.parse_args(["manage", "migrate", "--noinput"]) - assert ns.command == "manage" - assert ns.manage_args == ["migrate", "--noinput"] - - -def test_systemd_dry_run_writes_unit_and_default(tmp_path): - result = install_systemd( - output_dir=tmp_path, - exec_start="/usr/bin/logstashui serve", - user="logstashui", - group="logstashui", - data_dir="/var/lib/logstashui", - bind="0.0.0.0:8443", - workers=2, - allowed_hosts="ui.example", - csrf_trusted_origins="https://ui.example:8443", - tls="true", - host_hostname="ui.example", - host_ips="10.0.0.5", - tls_sans="ui.example,10.0.0.5", - agent_ui_url="https://ui.example:8443", - no_auth="false", - dry_run=True, - ) - unit = tmp_path / "logstashui.service" - envf = tmp_path / "logstashui.default" - assert unit.is_file() - assert envf.is_file() - unit_text = unit.read_text() - env_text = envf.read_text() - assert "EnvironmentFile=-/etc/default/logstashui" in unit_text - assert "ExecStart=/usr/bin/logstashui serve" in unit_text - assert "User=logstashui" in unit_text - assert "LOGSTASHUI_DATA_DIR=/var/lib/logstashui" in env_text - assert "LOGSTASHUI_BIND=0.0.0.0:8443" in env_text - assert "LOGSTASHUI_NO_AUTH=false" in env_text - assert result["unit"] == unit - assert result["default"] == envf - assert "LOGSTASHUI_DB_ENGINE=postgresql" not in env_text - assert "# LOGSTASHUI_DB_ENGINE=sqlite" in env_text - - -def test_systemd_env_includes_postgres_when_passed(tmp_path): - result = install_systemd( - output_dir=tmp_path, - exec_start="/usr/bin/logstashui serve", - user="logstashui", - group="logstashui", - data_dir="/var/lib/logstashui", - bind="0.0.0.0:8443", - workers=2, - allowed_hosts="*", - csrf_trusted_origins="", - tls="true", - host_hostname="", - host_ips="", - tls_sans="", - agent_ui_url="", - no_auth="false", - dry_run=True, - db_engine="postgresql", - db_host="db.example", - db_port="5432", - db_name="logstashui", - db_user="lsui", - ) - text = (tmp_path / "logstashui.default").read_text() - assert "LOGSTASHUI_DB_ENGINE=postgresql" in text - assert "LOGSTASHUI_DB_HOST=db.example" in text - assert "LOGSTASHUI_DB_PORT=5432" in text - assert "LOGSTASHUI_DB_NAME=logstashui" in text - assert "LOGSTASHUI_DB_USER=lsui" in text - assert result["default"] == tmp_path / "logstashui.default" - - -def test_serve_snmp_commanderror_does_not_abort(monkeypatch): - """execute_from_command_line sys.exits on CommandError; call_command must not.""" - from LogstashUI import cli - - monkeypatch.setattr(cli, "_manage", lambda argv: None) - monkeypatch.setattr(cli, "_check_db_floor", lambda: None) - monkeypatch.setenv("LOGSTASHUI_TLS", "false") - - def fake_call(name, *args, **kwargs): - if name == "sync_snmp_official_data": - raise CommandError("sync failed") - return None - - monkeypatch.setattr("django.core.management.call_command", fake_call) - - exec_called = {} - - def fake_execvp(file, args): - exec_called["file"] = file - raise SystemExit(0) - - monkeypatch.setattr(cli.os, "execvp", fake_execvp) - - ns = Namespace(skip_migrate=False, no_tls=True, bind="127.0.0.1:8443", workers=1) - try: - cmd_serve(ns) - except SystemExit as exc: - assert exc.code == 0 - assert exec_called.get("file") == "gunicorn" - - -def test_parser_migrate_engine_requires_backup_flag(): - parser = build_parser() - ns = parser.parse_args(["migrate-engine", "--to", "postgresql"]) - assert ns.command == "migrate-engine" - assert ns.to == "postgresql" - assert ns.i_have_a_backup is False - - -def test_parser_migrate_engine_accepts_mariadb_alias(): - parser = build_parser() - ns = parser.parse_args(["migrate-engine", "--to", "mariadb", "--i-have-a-backup"]) - assert ns.to == "mariadb" - assert ns.i_have_a_backup is True - - -def test_serve_checks_version_before_migrate(monkeypatch): - from LogstashUI import cli - - order = [] - monkeypatch.setattr(cli, "_check_db_floor", lambda: order.append("check")) - monkeypatch.setattr(cli, "_manage", lambda argv: order.append(argv[0])) - monkeypatch.setattr(cli, "_best_effort_call", lambda *a, **k: None) - monkeypatch.setenv("LOGSTASHUI_TLS", "false") - - def fake_execvp(file, args): - raise SystemExit(0) - - monkeypatch.setattr(cli.os, "execvp", fake_execvp) - ns = Namespace(skip_migrate=False, no_tls=True, bind="127.0.0.1:8443", workers=1) - try: - cmd_serve(ns) - except SystemExit: - pass - assert order[0] == "check" - assert "migrate" in order - - -def test_serve_checks_version_when_skip_migrate(monkeypatch, tmp_path): - from LogstashUI import cli - - called = [] - monkeypatch.setattr(cli, "_check_db_floor", lambda: called.append(True)) - monkeypatch.setattr(cli, "_manage", lambda argv: None) - monkeypatch.setenv("LOGSTASHUI_TLS", "false") - monkeypatch.setenv("LOGSTASHUI_DATA_DIR", str(tmp_path)) - - def fake_execvp(file, args): - raise SystemExit(0) - - monkeypatch.setattr(cli.os, "execvp", fake_execvp) - ns = Namespace(skip_migrate=True, no_tls=True, bind="127.0.0.1:8443", workers=1) - try: - cmd_serve(ns) - except SystemExit: - pass - assert called == [True] - - -def test_serve_adds_pidfile_and_warns_sqlite(monkeypatch, tmp_path, capsys): - from LogstashUI import cli - - monkeypatch.setattr(cli, "_manage", lambda argv: None) - monkeypatch.setattr(cli, "_check_db_floor", lambda: None) - monkeypatch.setenv("LOGSTASHUI_TLS", "false") - monkeypatch.setenv("LOGSTASHUI_DATA_DIR", str(tmp_path)) - monkeypatch.delenv("LOGSTASHUI_DB_ENGINE", raising=False) - - captured = {} - - def fake_execvp(file, args): - captured["file"] = file - captured["args"] = list(args) - raise SystemExit(0) - - monkeypatch.setattr(cli.os, "execvp", fake_execvp) - ns = Namespace(skip_migrate=True, no_tls=True, bind="127.0.0.1:8443", workers=2) - try: - cmd_serve(ns) - except SystemExit: - pass - assert "--pid" in captured["args"] - pid_idx = captured["args"].index("--pid") - assert captured["args"][pid_idx + 1].endswith("gunicorn.pid") - err = capsys.readouterr().err - assert "SQLite is the small-install default" in err - - -def test_exec_gunicorn_frozen_runs_in_process(monkeypatch): - import sys - - monkeypatch.setattr(sys, "frozen", True, raising=False) - seen = {} - - def fake_run(): - seen["argv"] = list(sys.argv) - return 0 - - monkeypatch.setattr("gunicorn.app.wsgiapp.run", fake_run) - rc = _exec_gunicorn( - ["gunicorn", "LogstashUI.wsgi:application", "--bind", "0.0.0.0:8443"] - ) - assert rc == 0 - assert seen["argv"][0] == "gunicorn" - assert "LogstashUI.wsgi:application" in seen["argv"] - diff --git a/src/logstashui/LogstashUI/tests/test_config.py b/src/logstashui/LogstashUI/tests/test_config.py deleted file mode 100644 index 3c8e862..0000000 --- a/src/logstashui/LogstashUI/tests/test_config.py +++ /dev/null @@ -1,61 +0,0 @@ -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. - -from LogstashUI.config import load_config, merge_allowed_hosts - - -def test_merge_allowed_hosts_wildcard_unchanged(): - assert merge_allowed_hosts(allowed="*", host_ips="10.11.3.107") == ["*"] - - -def test_merge_allowed_hosts_appends_pod_ip(): - hosts = merge_allowed_hosts( - allowed="logstashui.example.com,logstashui", - host_ips="10.11.3.107", - pod_ip="", - ) - assert hosts == ["logstashui.example.com", "logstashui", "10.11.3.107"] - - -def test_merge_allowed_hosts_pod_ip_env_and_no_dupes(monkeypatch): - monkeypatch.setenv("ALLOWED_HOSTS", "logstashui") - monkeypatch.setenv("LOGSTASHUI_HOST_IPS", "10.11.3.107") - monkeypatch.setenv("POD_IP", "10.11.3.107") - assert merge_allowed_hosts() == ["logstashui", "10.11.3.107"] - - -def test_load_config_defaults(monkeypatch): - monkeypatch.delenv("LOGSTASHUI_NO_AUTH", raising=False) - monkeypatch.delenv("LOGSTASHUI_AGENT_UI_URL", raising=False) - monkeypatch.delenv("LOGSTASHUI_INCLUDE_CA_FINGERPRINT", raising=False) - cfg = load_config() - assert cfg["no_auth"]["enabled"] is False - assert cfg["agent"]["ui_url"] == "" - assert cfg["agent"]["include_ca_fingerprint"] is True - - -def test_load_config_no_auth_env(monkeypatch): - monkeypatch.setenv("LOGSTASHUI_NO_AUTH", "true") - cfg = load_config() - assert cfg["no_auth"]["enabled"] is True - monkeypatch.setenv("LOGSTASHUI_NO_AUTH", "0") - cfg = load_config() - assert cfg["no_auth"]["enabled"] is False - - -def test_load_config_agent_env(monkeypatch): - monkeypatch.setenv("LOGSTASHUI_AGENT_UI_URL", "https://ui.example:8443/") - monkeypatch.setenv("LOGSTASHUI_INCLUDE_CA_FINGERPRINT", "false") - cfg = load_config() - assert cfg["agent"]["ui_url"] == "https://ui.example:8443" - assert cfg["agent"]["include_ca_fingerprint"] is False - - -def test_load_config_ignores_yaml_env(monkeypatch, tmp_path): - yml = tmp_path / "logstashui.yml" - yml.write_text("no_auth:\n enabled: true\n") - monkeypatch.setenv("LOGSTASHUI_CONFIG", str(yml)) - monkeypatch.delenv("LOGSTASHUI_NO_AUTH", raising=False) - cfg = load_config() - assert cfg["no_auth"]["enabled"] is False diff --git a/src/logstashui/LogstashUI/tests/test_logging_config.py b/src/logstashui/LogstashUI/tests/test_logging_config.py deleted file mode 100644 index b798d5a..0000000 --- a/src/logstashui/LogstashUI/tests/test_logging_config.py +++ /dev/null @@ -1,50 +0,0 @@ -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. - -import pytest - -from LogstashUI.logging_config import resolve_django_log_levels, resolve_log_level - - -def test_log_level_defaults_follow_debug_flag(monkeypatch): - monkeypatch.delenv("LOGSTASHUI_LOG_LEVEL", raising=False) - assert resolve_log_level("LOGSTASHUI_LOG_LEVEL", default="INFO") == "INFO" - assert resolve_log_level("LOGSTASHUI_LOG_LEVEL", default="DEBUG") == "DEBUG" - - -def test_log_level_env_override(monkeypatch): - monkeypatch.setenv("LOGSTASHUI_LOG_LEVEL", "warning") - assert resolve_log_level("LOGSTASHUI_LOG_LEVEL", default="INFO") == "WARNING" - monkeypatch.setenv("LOGSTASHUI_LOG_LEVEL", "WARN") - assert resolve_log_level("LOGSTASHUI_LOG_LEVEL", default="INFO") == "WARNING" - - -def test_log_level_invalid(monkeypatch): - monkeypatch.setenv("LOGSTASHUI_LOG_LEVEL", "verbose") - with pytest.raises(RuntimeError, match="LOGSTASHUI_LOG_LEVEL"): - resolve_log_level("LOGSTASHUI_LOG_LEVEL", default="INFO") - - -def test_django_levels_default(monkeypatch): - monkeypatch.delenv("LOGSTASHUI_DJANGO_LOG_LEVEL", raising=False) - monkeypatch.delenv("DJANGO_LOG_LEVEL", raising=False) - django_level, request_level = resolve_django_log_levels() - assert django_level == "INFO" - assert request_level == "ERROR" - - -def test_django_levels_prefixed_env(monkeypatch): - monkeypatch.setenv("LOGSTASHUI_DJANGO_LOG_LEVEL", "debug") - monkeypatch.setenv("DJANGO_LOG_LEVEL", "error") - django_level, request_level = resolve_django_log_levels() - assert django_level == "DEBUG" - assert request_level == "DEBUG" - - -def test_django_levels_alias(monkeypatch): - monkeypatch.delenv("LOGSTASHUI_DJANGO_LOG_LEVEL", raising=False) - monkeypatch.setenv("DJANGO_LOG_LEVEL", "warning") - django_level, request_level = resolve_django_log_levels() - assert django_level == "WARNING" - assert request_level == "WARNING" diff --git a/src/logstashui/LogstashUI/tests/test_paths.py b/src/logstashui/LogstashUI/tests/test_paths.py deleted file mode 100644 index 86af809..0000000 --- a/src/logstashui/LogstashUI/tests/test_paths.py +++ /dev/null @@ -1,91 +0,0 @@ -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. - -from pathlib import Path - -from LogstashUI.paths import ( - PROJECT_ROOT, - maybe_migrate_legacy_data, - resolve_data_dir, - resolve_logs_dir, -) - - -def test_env_data_dir_wins(tmp_path, monkeypatch): - dest = tmp_path / "from-env" - dest.mkdir() - monkeypatch.setenv("LOGSTASHUI_DATA_DIR", str(dest)) - monkeypatch.delenv("LOGSTASHUI_LOGS_DIR", raising=False) - assert resolve_data_dir(migrate_legacy=False) == dest - assert resolve_logs_dir(dest) == dest / "logs" - - -def test_env_logs_dir_wins(tmp_path, monkeypatch): - data = tmp_path / "data" - logs = tmp_path / "logs" - data.mkdir() - logs.mkdir() - monkeypatch.setenv("LOGSTASHUI_DATA_DIR", str(data)) - monkeypatch.setenv("LOGSTASHUI_LOGS_DIR", str(logs)) - assert resolve_logs_dir() == logs - - -def test_relative_env_path_is_absolute(tmp_path, monkeypatch): - monkeypatch.chdir(tmp_path) - monkeypatch.setenv("LOGSTASHUI_DATA_DIR", "relative-data") - resolved = resolve_data_dir(migrate_legacy=False) - assert resolved.is_absolute() - assert resolved == (tmp_path / "relative-data").resolve() - - -def test_migrate_legacy_copies_sqlite(tmp_path): - legacy = tmp_path / "legacy" - dest = tmp_path / "dest" - legacy.mkdir() - (legacy / "db.sqlite3").write_bytes(b"sqlite") - (legacy / "tls").mkdir() - (legacy / "tls" / "product-ca.crt").write_text("ca") - # Point LEGACY_DATA_DIR by copying into maybe_migrate with patched constant - from LogstashUI import paths as paths_mod - - original = paths_mod.LEGACY_DATA_DIR - try: - paths_mod.LEGACY_DATA_DIR = legacy - maybe_migrate_legacy_data(dest) - assert (dest / "db.sqlite3").read_bytes() == b"sqlite" - assert (dest / "tls" / "product-ca.crt").read_text() == "ca" - maybe_migrate_legacy_data(dest) # idempotent - assert (dest / "db.sqlite3").read_bytes() == b"sqlite" - finally: - paths_mod.LEGACY_DATA_DIR = original - - -def test_pytest_default_is_legacy_not_checkout_bind(): - # Under pytest, default must not be /logstashui_data - resolved = resolve_data_dir(migrate_legacy=False) - assert resolved != PROJECT_ROOT / "logstashui_data" - assert resolved.name == "data" - - -def test_native_default_is_cwd_logstashui_data(tmp_path, monkeypatch): - """Installed / CLI default is $(pwd)/logstashui_data, not site-packages.""" - from LogstashUI import paths as paths_mod - - monkeypatch.chdir(tmp_path) - monkeypatch.delenv("LOGSTASHUI_DATA_DIR", raising=False) - monkeypatch.setattr(paths_mod, "_is_pytest", lambda: False) - resolved = paths_mod.resolve_data_dir(migrate_legacy=False) - assert resolved == (tmp_path / "logstashui_data").resolve() - - -def test_packaged_docs_dir_expects_content_images(tmp_path, monkeypatch): - """Wheel install: docs root is Documentation/content (images live under that).""" - from LogstashUI import paths as paths_mod - from LogstashUI.paths import resolve_docs_dir - - monkeypatch.delenv("LOGSTASHUI_DOCS_DIR", raising=False) - monkeypatch.setattr(paths_mod, "PROJECT_ROOT", tmp_path / "not-a-checkout") - resolved = resolve_docs_dir() - assert resolved == paths_mod.BASE_DIR / "Documentation" / "content" - assert resolved / "images" == paths_mod.BASE_DIR / "Documentation" / "content" / "images" diff --git a/src/logstashui/Management/tests/__init__.py b/src/logstashui/Management/tests/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/src/logstashui/Management/tests/test_views.py b/src/logstashui/Management/tests/test_views.py deleted file mode 100644 index e26bc9d..0000000 --- a/src/logstashui/Management/tests/test_views.py +++ /dev/null @@ -1,1067 +0,0 @@ -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. - -import pytest -from django.contrib.auth.models import User -from Common.test_resources import authenticated_client, test_user - - -# ============================================================================ -# FIXTURES -# ============================================================================ - -@pytest.fixture -def readonly_user(db): - """Create a readonly-role test user""" - user = User.objects.create_user( - username='readonlyuser', - password='testpass123', - email='readonly@example.com' - ) - user.is_superuser = False - user.is_staff = False - user.save() - # The signal creates a profile with role='admin' by default; override it. - user.profile.role = 'readonly' - user.profile.save() - return user - - -@pytest.fixture -def readonly_client(client, readonly_user): - """Client authenticated as a readonly user""" - client.login(username='readonlyuser', password='testpass123') - return client - - -# ============================================================================ -# SECTION 1: BootstrapLoginView — First-Run & Login Tests -# ============================================================================ - -@pytest.mark.django_db -class TestFirstRunLogin: - """ - Tests for BootstrapLoginView: the first-user registration flow and - the normal login flow (finding 4e). - """ - - def test_first_run_shows_registration_form(self, client): - """ - With no users in the database, the login page should render the - UserCreationForm (first-run registration mode). - """ - assert not User.objects.exists() - response = client.get('/Management/Login/') - assert response.status_code == 200 - # Context flag must be True to drive the template - assert response.context['is_first_run'] is True - # Registration fields should be present - assert b'password1' in response.content or b'Create Your Account' in response.content - - def test_first_run_creates_admin_user(self, client): - """ - POSTing valid credentials on first run should create a user with - is_superuser=True and role='admin'. - """ - assert not User.objects.exists() - - response = client.post('/Management/Login/', { - 'username': 'firstadmin', - 'password1': 'StrongPass123!', - 'password2': 'StrongPass123!', - }) - - # Should redirect after successful creation - assert response.status_code in (200, 302) - assert User.objects.filter(username='firstadmin').exists() - - user = User.objects.get(username='firstadmin') - assert user.is_superuser, "First user must be is_superuser" - assert user.is_staff, "First user must be is_staff" - assert user.profile.role == 'admin', "First user must have role='admin'" - - def test_first_run_weak_password_rejected(self, client): - """ - A weak password on the first-run form should be rejected and no user - should be created. - """ - assert not User.objects.exists() - - response = client.post('/Management/Login/', { - 'username': 'firstadmin', - 'password1': '123', - 'password2': '123', - }) - - # Form should re-render with errors, not redirect - assert response.status_code == 200 - assert not User.objects.exists(), "No user should be created with a weak password" - - def test_normal_login_shows_auth_form(self, db): - """ - When at least one user exists, the login page should render the - AuthenticationForm (normal login mode). - """ - from django.test import Client - User.objects.create_user(username='existinguser', password='pass123') - client = Client() - response = client.get('/Management/Login/') - assert response.status_code == 200 - assert response.context['is_first_run'] is False - assert b'Sign In' in response.content or b'password' in response.content - - def test_normal_login_success(self, db): - """ - Correct credentials on the standard login form should log the user in - and redirect to the home page. - """ - from django.test import Client - User.objects.create_user(username='loginuser', password='ValidPass123!') - client = Client() - response = client.post('/Management/Login/', { - 'username': 'loginuser', - 'password': 'ValidPass123!', - }) - # Successful login redirects - assert response.status_code == 302 - - def test_normal_login_wrong_password(self, db): - """ - Wrong credentials should re-render the form with errors and not log in. - """ - from django.test import Client - User.objects.create_user(username='loginuser', password='ValidPass123!') - client = Client() - response = client.post('/Management/Login/', { - 'username': 'loginuser', - 'password': 'WrongPassword!', - }) - # Should re-render the page (200), not redirect - assert response.status_code == 200 - assert response.context['form'].errors - - def test_first_run_form_not_shown_after_user_exists(self, db): - """ - Once a user exists, a second visitor should NOT see the registration - form — even if they know to hit /Management/Login/ on a fresh browser. - This tests the first-run guard doesn't leak to after setup. - """ - from django.test import Client - User.objects.create_user(username='alreadysetup', password='pass123') - client = Client() - response = client.get('/Management/Login/') - assert response.context['is_first_run'] is False - # Registration-only fields should not appear - assert b'password1' not in response.content - - -# ============================================================================ -# SECTION 2: User Management CRUD Tests -# ============================================================================ - -@pytest.mark.django_db -class TestUserManagementCRUD: - """Test User Create, Read, Update, Delete operations""" - - def test_create_user_success(self, authenticated_client): - """Test successful user creation""" - response = authenticated_client.post('/Management/Users/', { - 'action': 'add', - 'username': 'newuser', - 'password': 'SecurePass123!', - 'password2': 'SecurePass123!', - 'email': 'newuser@example.com', - 'role': 'admin' - }) - - assert response.status_code == 200 - assert b'window.location.reload()' in response.content - - # Verify user was created - assert User.objects.filter(username='newuser').exists() - new_user = User.objects.get(username='newuser') - assert new_user.is_superuser - assert new_user.is_staff - - def test_create_user_duplicate_username(self, authenticated_client, test_user): - """Test creating user with duplicate username""" - response = authenticated_client.post('/Management/Users/', { - 'action': 'add', - 'username': 'testuser', # Already exists - 'password': 'SecurePass123!', - 'password2': 'SecurePass123!', - 'email': 'duplicate@example.com' - }) - - assert response.status_code == 200 - assert b'Username already exists' in response.content - - def test_create_user_password_mismatch(self, authenticated_client): - """Test creating user with mismatched passwords""" - response = authenticated_client.post('/Management/Users/', { - 'action': 'add', - 'username': 'newuser', - 'password': 'SecurePass123!', - 'password2': 'DifferentPass123!', - 'email': 'newuser@example.com' - }) - - assert response.status_code == 200 - assert b"didn't match" in response.content - - def test_create_user_weak_password(self, authenticated_client): - """Test creating user with weak password""" - response = authenticated_client.post('/Management/Users/', { - 'action': 'add', - 'username': 'newuser', - 'password': '123', # Too short - 'password2': '123', - 'email': 'newuser@example.com' - }) - - assert response.status_code == 200 - # Should contain password validation error - assert b'red-500' in response.content - - def test_update_user_password_success(self, authenticated_client, db): - """Test successful user password update""" - # Create a second user to update - other_user = User.objects.create_user( - username='otheruser', - password='oldpass123', - email='other@example.com' - ) - - response = authenticated_client.post('/Management/Users/', { - 'action': 'update_password', - 'user_id': other_user.id, - 'new_password': 'NewSecurePass123!', - 'new_password2': 'NewSecurePass123!' - }) - - assert response.status_code == 200 - assert b'window.location.reload()' in response.content - - # Verify password was updated - other_user.refresh_from_db() - assert other_user.check_password('NewSecurePass123!') - - def test_update_user_password_mismatch(self, authenticated_client, db): - """Test updating user password with mismatch""" - other_user = User.objects.create_user( - username='otheruser', - password='oldpass123', - email='other@example.com' - ) - - response = authenticated_client.post('/Management/Users/', { - 'action': 'update_password', - 'user_id': other_user.id, - 'new_password': 'NewPass123!', - 'new_password2': 'DifferentPass123!' - }) - - assert response.status_code == 200 - assert b"didn't match" in response.content - - def test_delete_user_success(self, authenticated_client, db): - """Test successful user deletion""" - # Create a second user to delete - other_user = User.objects.create_user( - username='deleteuser', - password='pass123', - email='delete@example.com' - ) - user_id = other_user.id - - response = authenticated_client.post('/Management/Users/', { - 'action': 'delete', - 'user_id': user_id - }) - - assert response.status_code == 200 - - # Verify user was deleted - assert not User.objects.filter(id=user_id).exists() - - def test_delete_last_user_prevented(self, authenticated_client, test_user): - """Test that deleting the last user is prevented""" - response = authenticated_client.post('/Management/Users/', { - 'action': 'delete', - 'user_id': test_user.id - }) - - assert response.status_code == 200 - assert b'Cannot delete the last user' in response.content - - # Verify user still exists - assert User.objects.filter(id=test_user.id).exists() - - def test_delete_own_account_prevented(self, authenticated_client, test_user, db): - """Test that users cannot delete their own account""" - # Create a second user so we're not the last user - User.objects.create_user( - username='otheruser', - password='pass123', - email='other@example.com' - ) - - response = authenticated_client.post('/Management/Users/', { - 'action': 'delete', - 'user_id': test_user.id - }) - - assert response.status_code == 200 - assert b'cannot delete your own account' in response.content - - # Verify user still exists - assert User.objects.filter(id=test_user.id).exists() - - -# ============================================================================ -# SECTION 3: Authorization / Role Enforcement Tests -# ============================================================================ - -@pytest.mark.django_db -class TestReadonlyUserBlocked: - """ - Verify that a user with role='readonly' cannot perform any write - operations on the Users management endpoint (finding 4a / issue 2c). - """ - - def test_readonly_cannot_add_user(self, readonly_client): - """Readonly user should receive 403 when attempting to add a user""" - response = readonly_client.post('/Management/Users/', { - 'action': 'add', - 'username': 'shouldnotexist', - 'password': 'SecurePass123!', - 'password2': 'SecurePass123!', - 'email': 'nope@example.com' - }) - - assert response.status_code == 403 - assert b'Access denied' in response.content - assert not User.objects.filter(username='shouldnotexist').exists() - - def test_readonly_cannot_delete_user(self, readonly_client, test_user): - """Readonly user should receive 403 when attempting to delete a user""" - response = readonly_client.post('/Management/Users/', { - 'action': 'delete', - 'user_id': test_user.id - }) - - assert response.status_code == 403 - assert b'Access denied' in response.content - # Confirm the user was NOT deleted - assert User.objects.filter(id=test_user.id).exists() - - def test_readonly_cannot_update_password(self, readonly_client, test_user): - """Readonly user should receive 403 when attempting to update a password""" - response = readonly_client.post('/Management/Users/', { - 'action': 'update_password', - 'user_id': test_user.id, - 'new_password': 'HackedPass123!', - 'new_password2': 'HackedPass123!' - }) - - assert response.status_code == 403 - assert b'Access denied' in response.content - # Confirm original password still works - test_user.refresh_from_db() - assert test_user.check_password('testpass123') - - def test_readonly_cannot_update_role(self, readonly_client, test_user): - """Readonly user should receive 403 when attempting to change a user role""" - response = readonly_client.post('/Management/Users/', { - 'action': 'update_role', - 'user_id': test_user.id, - 'role': 'readonly' - }) - - assert response.status_code == 403 - assert b'Access denied' in response.content - - def test_readonly_can_view_users_page(self, readonly_client): - """Readonly user should still be able to GET the users page""" - response = readonly_client.get('/Management/Users/') - assert response.status_code == 200 - - -@pytest.mark.django_db -class TestRoleValidation: - """ - Confirm that only valid role values ('admin', 'readonly') are accepted - server-side in add and update_role actions (finding 2c). - """ - - def test_add_user_with_invalid_role_rejected(self, authenticated_client): - """Submitting an invalid role string should return an error and not create the user""" - response = authenticated_client.post('/Management/Users/', { - 'action': 'add', - 'username': 'rolebreaker', - 'password': 'SecurePass123!', - 'password2': 'SecurePass123!', - 'email': 'rolebreaker@example.com', - 'role': 'superadmin' # Not a valid choice - }) - # Should return error response - assert response.status_code == 200 - assert b'Invalid role' in response.content - # User should not be created - assert not User.objects.filter(username='rolebreaker').exists() - - def test_update_role_with_invalid_role_rejected(self, authenticated_client, test_user): - """Submitting an invalid role to update_role should not persist it""" - response = authenticated_client.post('/Management/Users/', { - 'action': 'update_role', - 'user_id': test_user.id, - 'role': 'god' # Not a valid choice - }) - test_user.refresh_from_db() - assert test_user.profile.role in ('admin', 'readonly'), ( - f"Invalid role '{test_user.profile.role}' was saved to the database" - ) - - -# ============================================================================ -# SECTION 4: Update Role Tests -# ============================================================================ - -@pytest.mark.django_db -class TestUpdateRole: - """Test the update_role action on the Users management endpoint (finding 4b)""" - - def test_update_role_admin_to_readonly(self, authenticated_client, test_user, db): - """Successfully change a user's role from admin to readonly""" - # Create a second user with admin role (signal default) - other_user = User.objects.create_user( - username='otheradmin', - password='pass123', - email='other@example.com' - ) - assert other_user.profile.role == 'admin' - - response = authenticated_client.post('/Management/Users/', { - 'action': 'update_role', - 'user_id': other_user.id, - 'role': 'readonly' - }) - - assert response.status_code == 200 - other_user.refresh_from_db() - assert other_user.profile.role == 'readonly' - # Verify Django permissions were synced - assert not other_user.is_superuser - assert not other_user.is_staff - - def test_update_role_readonly_to_admin(self, authenticated_client, db): - """Successfully change a user's role from readonly to admin""" - user = User.objects.create_user( - username='readonlyuser', - password='pass123', - email='ro@example.com' - ) - user.profile.role = 'readonly' - user.profile.save() - - response = authenticated_client.post('/Management/Users/', { - 'action': 'update_role', - 'user_id': user.id, - 'role': 'admin' - }) - - assert response.status_code == 200 - user.refresh_from_db() - assert user.profile.role == 'admin' - # Verify Django permissions were synced - assert user.is_superuser - assert user.is_staff - - def test_update_role_no_change_returns_message(self, authenticated_client, db): - """Submitting the same role that already exists should return a message, not reload""" - user = User.objects.create_user( - username='sameroleuser', - password='pass123', - email='same@example.com' - ) - # Default role is 'admin' - assert user.profile.role == 'admin' - - response = authenticated_client.post('/Management/Users/', { - 'action': 'update_role', - 'user_id': user.id, - 'role': 'admin' - }) - - assert response.status_code == 200 - # Should say no changes made, NOT trigger a reload - assert b'No changes made' in response.content - assert b'window.location.reload()' not in response.content - - def test_update_role_user_not_found(self, authenticated_client): - """Passing a non-existent user_id should return a friendly error""" - response = authenticated_client.post('/Management/Users/', { - 'action': 'update_role', - 'user_id': 999999, - 'role': 'readonly' - }) - - assert response.status_code == 200 - assert b'User not found' in response.content - - -# ============================================================================ -# SECTION 5: Logs Endpoint Tests -# ============================================================================ - -@pytest.mark.django_db -class TestLogsEndpoints: - """Tests for the Logs view, LogsFilter, and LogsDownload (finding 4d)""" - - def test_logs_page_loads(self, authenticated_client): - """The Logs page should render successfully""" - response = authenticated_client.get('/Management/Logs/') - assert response.status_code == 200 - assert b'Log Entries' in response.content - - def test_logs_page_no_file_shows_empty(self, authenticated_client, settings, tmp_path): - """ - When the log file does not exist, the page should still render and - show zero entries rather than raising an error. - """ - # Point LOGS_DIR to a temp directory that has NO log file - settings.LOGS_DIR = tmp_path - response = authenticated_client.get('/Management/Logs/') - # Should still render (not 500) - assert response.status_code == 200 - - def test_logs_filter_returns_fragment(self, authenticated_client): - """LogsFilter should return an HTML fragment, not a full page""" - response = authenticated_client.get('/Management/Logs/filter') - assert response.status_code == 200 - # Should be an HTML fragment, not a full Django page with base template - assert b'' not in response.content - assert b'alert("xss") logged in\n', - encoding='utf-8' - ) - settings.LOGS_DIR = tmp_path - - response = authenticated_client.get('/Management/Logs/filter') - - assert response.status_code == 200 - content = response.content.decode() - # The raw " - mock_test_connectivity.return_value = (False, xss_payload) - - response = authenticated_client.get(f'/ConnectionManager/TestConnectivity?test={test_connection.id}') - - assert response.status_code == 200 - # Verify the script tag is escaped, not executed - content = response.content.decode('utf-8') - assert '<script>' in content - assert '", - "run_id": "test-run-xss", - "malicious_field": "'; DROP TABLE users; --" - } - - response = client.post( - '/ConnectionManager/StreamSimulate/', - data=json.dumps(event_data), - content_type='application/json' - ) - - assert response.status_code == 200 - - # Verify data is stored as-is (will be escaped when rendered) - with simulation_lock: - assert len(simulation_results) == 1 - stored_event = simulation_results[0] - # Data should be stored but will be escaped during rendering - assert stored_event['message'] == "" - - def test_stream_simulate_method_not_allowed(self, client): - """Test StreamSimulate with GET request""" - response = client.get('/ConnectionManager/StreamSimulate/') - - assert response.status_code == 405 - data = json.loads(response.content) - assert 'error' in data - - -# ============================================================================ -# GetSimulationResults Tests -# ============================================================================ - -@pytest.mark.django_db -class TestGetSimulationResults: - """Test GetSimulationResults view""" - - def test_get_simulation_results_success(self, authenticated_client): - """Test successful retrieval of simulation results""" - # Clear and populate queue - with simulation_lock: - simulation_results.clear() - simulation_results.append({ - "message": "event 1", - "run_id": "test-run-1" - }) - simulation_results.append({ - "message": "event 2", - "run_id": "test-run-1" - }) - simulation_results.append({ - "message": "event 3", - "run_id": "test-run-2" - }) - - response = authenticated_client.get('/ConnectionManager/GetSimulationResults/?run_id=test-run-1') - - assert response.status_code == 200 - data = json.loads(response.content) - assert 'results' in data - assert len(data['results']) == 2 - assert all(r['run_id'] == 'test-run-1' for r in data['results']) - - # Verify only run-1 events were removed from queue - with simulation_lock: - assert len(simulation_results) == 1 - assert simulation_results[0]['run_id'] == 'test-run-2' - - def test_get_simulation_results_no_run_id(self, authenticated_client): - """Test GetSimulationResults without run_id parameter""" - response = authenticated_client.get('/ConnectionManager/GetSimulationResults/') - - assert response.status_code == 400 - data = json.loads(response.content) - assert 'error' in data - assert 'run_id' in data['error'] - - def test_get_simulation_results_race_condition_safety(self, authenticated_client): - """Test GetSimulationResults handles concurrent access safely""" - # Populate queue - with simulation_lock: - simulation_results.clear() - for i in range(100): - simulation_results.append({ - "message": f"event {i}", - "run_id": "test-run-race" - }) - - # Make multiple concurrent-like requests - response1 = authenticated_client.get('/ConnectionManager/GetSimulationResults/?run_id=test-run-race') - response2 = authenticated_client.get('/ConnectionManager/GetSimulationResults/?run_id=test-run-race') - - assert response1.status_code == 200 - assert response2.status_code == 200 - - data1 = json.loads(response1.content) - data2 = json.loads(response2.content) - - # First request should get all events, second should get none - assert len(data1['results']) == 100 - assert len(data2['results']) == 0 - - def test_get_simulation_results_empty_queue(self, authenticated_client): - """Test GetSimulationResults when no results exist""" - with simulation_lock: - simulation_results.clear() - - response = authenticated_client.get('/ConnectionManager/GetSimulationResults/?run_id=nonexistent') - - assert response.status_code == 200 - data = json.loads(response.content) - assert data['results'] == [] - - -# ============================================================================ -# CheckIfPipelineLoaded Tests -# ============================================================================ - -@pytest.mark.django_db -class TestCheckIfPipelineLoaded: - """Test CheckIfPipelineLoaded view""" - - @patch('PipelineManager.simulation.requests.get') - def test_check_pipeline_loaded_running(self, mock_get, authenticated_client): - """Test checking a running pipeline""" - mock_response = Mock() - mock_response.status_code = 200 - mock_response.json.return_value = { - 'running_pipelines': ['slot1-filter1', 'slot2-filter1', 'main'] - } - mock_get.return_value = mock_response - - response = authenticated_client.get('/ConnectionManager/CheckIfPipelineLoaded/?pipeline_name=slot1-filter1') - - assert response.status_code == 200 - data = json.loads(response.content) - assert data['is_running'] is True - assert data['pipeline_name'] == 'slot1-filter1' - - @patch('PipelineManager.simulation.requests.get') - def test_check_pipeline_loaded_not_running(self, mock_get, authenticated_client): - """Test checking a non-running pipeline""" - mock_response = Mock() - mock_response.status_code = 200 - mock_response.json.return_value = { - 'running_pipelines': ['slot1-filter1', 'main'] - } - mock_get.return_value = mock_response - - response = authenticated_client.get('/ConnectionManager/CheckIfPipelineLoaded/?pipeline_name=slot2-filter1') - - assert response.status_code == 200 - data = json.loads(response.content) - assert data['is_running'] is False - assert data['pipeline_name'] == 'slot2-filter1' - - def test_check_pipeline_loaded_no_pipeline_name(self, authenticated_client): - """Test CheckIfPipelineLoaded without pipeline_name parameter""" - response = authenticated_client.get('/ConnectionManager/CheckIfPipelineLoaded/') - - assert response.status_code == 400 - data = json.loads(response.content) - assert 'error' in data - assert 'pipeline_name' in data['error'] - - @patch('PipelineManager.simulation.requests.get') - def test_check_pipeline_loaded_service_unavailable(self, mock_get, authenticated_client): - """Test CheckIfPipelineLoaded when logstashagent is unavailable""" - mock_get.side_effect = Exception("Connection refused") - - response = authenticated_client.get('/ConnectionManager/CheckIfPipelineLoaded/?pipeline_name=slot1-filter1') - - assert response.status_code == 500 - data = json.loads(response.content) - assert 'error' in data - assert data['is_running'] is False - - -# ============================================================================ -# GetRelatedLogs Tests -# ============================================================================ - -@pytest.mark.django_db -class TestGetRelatedLogs: - """Test GetRelatedLogs view""" - - @patch('PipelineManager.simulation.requests.get') - def test_get_related_logs_success(self, mock_get, authenticated_client): - """Test successful log retrieval""" - # Mock slots endpoint - mock_slots_response = Mock() - mock_slots_response.status_code = 200 - mock_slots_response.json.return_value = { - '1': { - 'created_at_millis': 1609459200000, - 'pipeline_name': 'slot1-filter1' - } - } - - # Mock logs endpoint - mock_logs_response = Mock() - mock_logs_response.status_code = 200 - mock_logs_response.json.return_value = { - 'pipeline_id': 'slot1-filter1', - 'log_count': 2, - 'logs': [ - {'level': 'INFO', 'message': 'Pipeline started', 'timeMillis': 1609459201000}, - {'level': 'DEBUG', 'message': 'Processing event', 'timeMillis': 1609459202000} - ] - } - - mock_get.side_effect = [mock_slots_response, mock_logs_response] - - response = authenticated_client.get('/ConnectionManager/GetRelatedLogs/?slot_id=1') - - assert response.status_code == 200 - data = json.loads(response.content) - assert data['log_count'] == 2 - assert len(data['logs']) == 2 - - def test_get_related_logs_no_slot_id(self, authenticated_client): - """Test GetRelatedLogs without slot_id parameter""" - response = authenticated_client.get('/ConnectionManager/GetRelatedLogs/') - - assert response.status_code == 400 - data = json.loads(response.content) - assert 'error' in data - assert 'slot_id' in data['error'] - - @patch('PipelineManager.simulation.requests.get') - def test_get_related_logs_with_filters(self, mock_get, authenticated_client): - """Test GetRelatedLogs with max_entries and min_level filters""" - mock_slots_response = Mock() - mock_slots_response.status_code = 200 - mock_slots_response.json.return_value = { - '1': {'created_at_millis': 1609459200000} - } - - mock_logs_response = Mock() - mock_logs_response.status_code = 200 - mock_logs_response.json.return_value = { - 'pipeline_id': 'slot1-filter1', - 'log_count': 1, - 'logs': [ - {'level': 'ERROR', 'message': 'Error occurred', 'timeMillis': 1609459201000} - ] - } - - mock_get.side_effect = [mock_slots_response, mock_logs_response] - - response = authenticated_client.get( - '/ConnectionManager/GetRelatedLogs/?slot_id=1&max_entries=50&min_level=ERROR' - ) - - assert response.status_code == 200 - data = json.loads(response.content) - assert data['log_count'] == 1 - - @patch('PipelineManager.simulation.requests.get') - def test_get_related_logs_service_unavailable(self, mock_get, authenticated_client): - """Test GetRelatedLogs when logstashagent is unavailable""" - mock_get.side_effect = Exception("Connection refused") - - response = authenticated_client.get('/ConnectionManager/GetRelatedLogs/?slot_id=1') - - assert response.status_code == 500 - data = json.loads(response.content) - assert 'error' in data - assert data['log_count'] == 0 - - -# ============================================================================ -# UploadFile Tests -# ============================================================================ - -@pytest.mark.django_db -class TestUploadFile: - """Test UploadFile view""" - - @patch('PipelineManager.simulation.requests.post') - def test_upload_file_success(self, mock_post, authenticated_client): - """Test successful file upload""" - mock_response = Mock() - mock_response.status_code = 200 - mock_response.json.return_value = {'status': 'ok'} - mock_post.return_value = mock_response - - file_content = b'test file content' - uploaded_file = SimpleUploadedFile("test.txt", file_content, content_type="text/plain") - - response = authenticated_client.post('/ConnectionManager/UploadFile/', { - 'file': uploaded_file, - 'filename': 'test.txt' - }) - - assert response.status_code == 200 - data = json.loads(response.content) - assert data['status'] == 'ok' - assert data['filename'] == 'test.txt' - - # Verify base64 encoding was used - mock_post.assert_called_once() - call_args = mock_post.call_args - posted_data = call_args[1]['json'] - assert 'content' in posted_data - assert 'filename' in posted_data - # Verify content is base64 encoded - decoded = base64.b64decode(posted_data['content']) - assert decoded == file_content - - def test_upload_file_no_file(self, authenticated_client): - """Test UploadFile with no file provided""" - response = authenticated_client.post('/ConnectionManager/UploadFile/', { - 'filename': 'test.txt' - }) - - assert response.status_code == 400 - data = json.loads(response.content) - assert 'error' in data - assert 'No file provided' in data['error'] - - def test_upload_file_no_filename(self, authenticated_client): - """Test UploadFile with no filename provided""" - file_content = b'test content' - uploaded_file = SimpleUploadedFile("test.txt", file_content) - - response = authenticated_client.post('/ConnectionManager/UploadFile/', { - 'file': uploaded_file - }) - - assert response.status_code == 400 - data = json.loads(response.content) - assert 'error' in data - assert 'No filename provided' in data['error'] - - @patch('PipelineManager.simulation.requests.post') - def test_upload_file_oversized(self, mock_post, authenticated_client): - """Test UploadFile with large file""" - # Create a 10MB file - large_content = b'x' * (10 * 1024 * 1024) - uploaded_file = SimpleUploadedFile("large.txt", large_content) - - mock_response = Mock() - mock_response.status_code = 200 - mock_post.return_value = mock_response - - response = authenticated_client.post('/ConnectionManager/UploadFile/', { - 'file': uploaded_file, - 'filename': 'large.txt' - }) - - # Should handle large files (or return appropriate error if size limit exists) - assert response.status_code in [200, 400, 413, 500] - - @patch('PipelineManager.simulation.requests.post') - def test_upload_file_agent_failure(self, mock_post, authenticated_client): - """Test UploadFile when logstashagent fails""" - mock_post.side_effect = Exception("Connection refused") - - file_content = b'test content' - uploaded_file = SimpleUploadedFile("test.txt", file_content) - - response = authenticated_client.post('/ConnectionManager/UploadFile/', { - 'file': uploaded_file, - 'filename': 'test.txt' - }) - - assert response.status_code == 500 - data = json.loads(response.content) - assert 'error' in data - # The actual error message is just the exception message - assert 'Connection refused' in data['error'] - - def test_upload_file_requires_admin(self, client): - """Test that UploadFile requires admin role""" - from django.contrib.auth.models import User - from Management.models import UserProfile - - readonly_user = User.objects.create_user( - username='readonly_upload', - password='testpass123', - is_staff=False - ) - readonly_user.profile.role = 'readonly' - readonly_user.profile.save() - client.login(username='readonly_upload', password='testpass123') - - file_content = b'test content' - uploaded_file = SimpleUploadedFile("test.txt", file_content) - - response = client.post('/ConnectionManager/UploadFile/', { - 'file': uploaded_file, - 'filename': 'test.txt' - }) - - assert response.status_code == 403 - - @patch('PipelineManager.simulation.requests.post') - def test_upload_file_binary_content(self, mock_post, authenticated_client): - """Test UploadFile with binary file content""" - mock_response = Mock() - mock_response.status_code = 200 - mock_post.return_value = mock_response - - # Binary content (e.g., image file) - binary_content = bytes([0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A]) - uploaded_file = SimpleUploadedFile("image.png", binary_content, content_type="image/png") - - response = authenticated_client.post('/ConnectionManager/UploadFile/', { - 'file': uploaded_file, - 'filename': 'image.png' - }) - - assert response.status_code == 200 - - # Verify binary content was properly encoded - call_args = mock_post.call_args - posted_data = call_args[1]['json'] - decoded = base64.b64decode(posted_data['content']) - assert decoded == binary_content diff --git a/src/logstashui/SNMP/tests/__init__.py b/src/logstashui/SNMP/tests/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/src/logstashui/SNMP/tests/test_commands.py b/src/logstashui/SNMP/tests/test_commands.py deleted file mode 100644 index ec797c2..0000000 --- a/src/logstashui/SNMP/tests/test_commands.py +++ /dev/null @@ -1,776 +0,0 @@ -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. - -""" -Tests for the sync_snmp_official_data management command and the -sync_official_profiles / sync_official_device_templates snmp_crud helpers -it delegates to. - -Test categories: - - sync_official_profiles() individual function behaviour - - sync_official_device_templates() individual function behaviour - - call_command('sync_snmp_official_data') end-to-end command behaviour - - --cleanup path (delete / orphan stale official records) -""" - -import json -import os - -import pytest -from django.core.management import call_command -from io import StringIO - -from SNMP.snmp_crud import sync_official_profiles, sync_official_device_templates -from SNMP.models import Profile, DeviceTemplate - - -# --------------------------------------------------------------------------- -# Shared data -# --------------------------------------------------------------------------- - -MINIMAL_PROFILE = { - "official_key": "test_profile_key", - "description": "A test profile", - "vendor": "Generic", - "product": "", -} - -MINIMAL_TEMPLATE = { - "official_key": "test_template_key", - "name": "Test Template", - "description": "A test template", - "vendor": "Generic", - "model": "", - "product": "", - "matching_rules": [], - "profiles": [], -} - - -# --------------------------------------------------------------------------- -# Helpers -# --------------------------------------------------------------------------- - -def _write_json(directory, filename, data): - with open(os.path.join(directory, filename), "w", encoding="utf-8") as f: - json.dump(data, f) - - -# --------------------------------------------------------------------------- -# Fixtures -# --------------------------------------------------------------------------- - -@pytest.fixture -def profile_dir(tmp_path, settings): - """ - Create the official_profiles directory and point settings.BASE_DIR to - the temp root. Tests that only exercise sync_official_profiles() use this. - """ - dirpath = tmp_path / "SNMP" / "data" / "official_profiles" - dirpath.mkdir(parents=True) - settings.BASE_DIR = str(tmp_path) - return str(dirpath) - - -@pytest.fixture -def template_dir(tmp_path, settings): - """ - Create BOTH data directories and point settings.BASE_DIR to the temp root. - Tests that exercise sync_official_device_templates() use this. - """ - (tmp_path / "SNMP" / "data" / "official_profiles").mkdir(parents=True) - dirpath = tmp_path / "SNMP" / "data" / "official_device_templates" - dirpath.mkdir(parents=True) - settings.BASE_DIR = str(tmp_path) - return str(dirpath) - - -@pytest.fixture -def both_dirs(tmp_path, settings): - """ - Creates both directories and returns (profile_dir_str, template_dir_str). - Used by command-level tests that exercise the full pipeline. - """ - p = tmp_path / "SNMP" / "data" / "official_profiles" - t = tmp_path / "SNMP" / "data" / "official_device_templates" - p.mkdir(parents=True) - t.mkdir(parents=True) - settings.BASE_DIR = str(tmp_path) - return str(p), str(t) - - -# --------------------------------------------------------------------------- -# sync_official_profiles — new record creation -# --------------------------------------------------------------------------- - -@pytest.mark.django_db -def test_sync_profiles_creates_new_profile(profile_dir): - _write_json(profile_dir, "test_profile.json", MINIMAL_PROFILE) - sync_official_profiles() - assert Profile.objects.filter(official_key="test_profile_key").exists() - - -@pytest.mark.django_db -def test_sync_profiles_stores_name_with_json_extension(profile_dir): - _write_json(profile_dir, "test_profile.json", MINIMAL_PROFILE) - sync_official_profiles() - profile = Profile.objects.get(official_key="test_profile_key") - assert profile.name == "test_profile.json" - - -@pytest.mark.django_db -def test_sync_profiles_sets_placeholder_flag(profile_dir): - _write_json(profile_dir, "test_profile.json", MINIMAL_PROFILE) - sync_official_profiles() - profile = Profile.objects.get(official_key="test_profile_key") - assert profile.profile_data == {"is_official_placeholder": True} - - -@pytest.mark.django_db -def test_sync_profiles_stores_vendor_and_product(profile_dir): - data = {**MINIMAL_PROFILE, "vendor": "Cisco", "product": "Catalyst"} - _write_json(profile_dir, "test_profile.json", data) - sync_official_profiles() - profile = Profile.objects.get(official_key="test_profile_key") - assert profile.vendor == "Cisco" - assert profile.product == "Catalyst" - - -# --------------------------------------------------------------------------- -# sync_official_profiles — update existing record -# --------------------------------------------------------------------------- - -@pytest.mark.django_db -def test_sync_profiles_updates_existing_record(profile_dir): - _write_json(profile_dir, "test_profile.json", MINIMAL_PROFILE) - sync_official_profiles() - - updated = {**MINIMAL_PROFILE, "description": "Updated description", "vendor": "Cisco"} - _write_json(profile_dir, "test_profile.json", updated) - sync_official_profiles() - - profile = Profile.objects.get(official_key="test_profile_key") - assert profile.description == "Updated description" - assert profile.vendor == "Cisco" - assert Profile.objects.filter(official_key="test_profile_key").count() == 1 - - -@pytest.mark.django_db -def test_sync_profiles_idempotent(profile_dir): - _write_json(profile_dir, "test_profile.json", MINIMAL_PROFILE) - sync_official_profiles() - sync_official_profiles() - assert Profile.objects.filter(official_key="test_profile_key").count() == 1 - - -# --------------------------------------------------------------------------- -# sync_official_profiles — fix #1: is_orphaned cleared on restoration -# --------------------------------------------------------------------------- - -@pytest.mark.django_db -def test_sync_profiles_clears_is_orphaned_on_restoration(profile_dir): - """ - Regression test for fix #1. - A profile previously marked is_orphaned=True must have the flag removed - the next time its backing JSON is present during sync. - """ - _write_json(profile_dir, "test_profile.json", MINIMAL_PROFILE) - sync_official_profiles() - - # Simulate cleanup having orphaned the profile - profile = Profile.objects.get(official_key="test_profile_key") - profile.profile_data = {"is_official_placeholder": True, "is_orphaned": True} - profile.save() - assert profile.profile_data.get("is_orphaned") is True - - # Re-sync with the JSON still present - sync_official_profiles() - profile.refresh_from_db() - - assert "is_orphaned" not in profile.profile_data - assert profile.profile_data == {"is_official_placeholder": True} - - -@pytest.mark.django_db -def test_sync_profiles_clears_arbitrary_stale_flags(profile_dir): - """profile_data is always reset to a clean placeholder on sync.""" - _write_json(profile_dir, "test_profile.json", MINIMAL_PROFILE) - sync_official_profiles() - - profile = Profile.objects.get(official_key="test_profile_key") - profile.profile_data = {"is_official_placeholder": True, "custom_flag": "leftover"} - profile.save() - - sync_official_profiles() - profile.refresh_from_db() - assert profile.profile_data == {"is_official_placeholder": True} - - -# --------------------------------------------------------------------------- -# sync_official_profiles — fix #6: missing vendor defaults to 'Any' -# --------------------------------------------------------------------------- - -@pytest.mark.django_db -def test_sync_profiles_missing_vendor_defaults_to_any(profile_dir): - """ - Regression test for fix #6. - A JSON file that omits the vendor key must still produce a DB record - (vendor='Any') rather than being silently skipped by a full_clean() failure. - """ - no_vendor = {k: v for k, v in MINIMAL_PROFILE.items() if k != "vendor"} - _write_json(profile_dir, "no_vendor.json", no_vendor) - sync_official_profiles() - - profile = Profile.objects.get(official_key="test_profile_key") - assert profile.vendor == "Any" - - -# --------------------------------------------------------------------------- -# sync_official_profiles — skip / guard cases -# --------------------------------------------------------------------------- - -@pytest.mark.django_db -def test_sync_profiles_skips_file_without_official_key(profile_dir): - no_key = {k: v for k, v in MINIMAL_PROFILE.items() if k != "official_key"} - _write_json(profile_dir, "no_key.json", no_key) - sync_official_profiles() - assert Profile.objects.count() == 0 - - -@pytest.mark.django_db -def test_sync_profiles_ignores_non_json_files(profile_dir): - with open(os.path.join(profile_dir, "readme.txt"), "w") as f: - f.write("not a profile") - sync_official_profiles() - assert Profile.objects.count() == 0 - - -@pytest.mark.django_db -def test_sync_profiles_handles_empty_directory(profile_dir): - sync_official_profiles() - assert Profile.objects.count() == 0 - - -@pytest.mark.django_db -def test_sync_profiles_handles_malformed_json_gracefully(profile_dir): - with open(os.path.join(profile_dir, "bad.json"), "w") as f: - f.write("{ not valid json }") - sync_official_profiles() - assert Profile.objects.count() == 0 - - -@pytest.mark.django_db -def test_sync_profiles_handles_multiple_files(profile_dir): - for i in range(5): - _write_json(profile_dir, f"profile_{i}.json", { - **MINIMAL_PROFILE, - "official_key": f"key_{i}", - }) - sync_official_profiles() - assert Profile.objects.filter(official_key__startswith="key_").count() == 5 - - -# --------------------------------------------------------------------------- -# sync_official_profiles — legacy backfill path -# --------------------------------------------------------------------------- - -@pytest.mark.django_db -def test_sync_profiles_backfills_official_key_for_legacy_record(profile_dir): - """ - A pre-existing DB record that lacks official_key should have it backfilled - when a JSON file with the matching name is found. - """ - legacy = Profile.objects.create( - name="legacy_profile.json", - official_key=None, - vendor="Generic", - profile_data={"is_official_placeholder": True}, - ) - _write_json(profile_dir, "legacy_profile.json", { - **MINIMAL_PROFILE, - "official_key": "legacy_key", - }) - sync_official_profiles() - - legacy.refresh_from_db() - assert legacy.official_key == "legacy_key" - assert Profile.objects.filter(official_key="legacy_key").count() == 1 - - -# --------------------------------------------------------------------------- -# sync_official_device_templates — new record creation -# --------------------------------------------------------------------------- - -@pytest.mark.django_db -def test_sync_templates_creates_new_template(template_dir): - _write_json(template_dir, "test_template.json", MINIMAL_TEMPLATE) - sync_official_device_templates() - assert DeviceTemplate.objects.filter(official_key="test_template_key").exists() - - -@pytest.mark.django_db -def test_sync_templates_is_marked_official(template_dir): - _write_json(template_dir, "test_template.json", MINIMAL_TEMPLATE) - sync_official_device_templates() - template = DeviceTemplate.objects.get(official_key="test_template_key") - assert template.official is True - - -@pytest.mark.django_db -def test_sync_templates_uses_name_field_not_filename(template_dir): - """The 'name' key in JSON is used as the DB name, not the filename stem.""" - data = {**MINIMAL_TEMPLATE, "name": "My Custom Name"} - _write_json(template_dir, "file_name_irrelevant.json", data) - sync_official_device_templates() - assert DeviceTemplate.objects.filter(name="My Custom Name").exists() - - -@pytest.mark.django_db -def test_sync_templates_stores_vendor_model_product(template_dir): - data = {**MINIMAL_TEMPLATE, "vendor": "Dell", "model": "PowerEdge", "product": "iDRAC"} - _write_json(template_dir, "test_template.json", data) - sync_official_device_templates() - template = DeviceTemplate.objects.get(official_key="test_template_key") - assert template.vendor == "Dell" - assert template.model == "PowerEdge" - assert template.product == "iDRAC" - - -# --------------------------------------------------------------------------- -# sync_official_device_templates — update existing record -# --------------------------------------------------------------------------- - -@pytest.mark.django_db -def test_sync_templates_updates_existing_record(template_dir): - _write_json(template_dir, "test_template.json", MINIMAL_TEMPLATE) - sync_official_device_templates() - - updated = {**MINIMAL_TEMPLATE, "description": "Updated", "vendor": "Cisco"} - _write_json(template_dir, "test_template.json", updated) - sync_official_device_templates() - - template = DeviceTemplate.objects.get(official_key="test_template_key") - assert template.description == "Updated" - assert template.vendor == "Cisco" - assert DeviceTemplate.objects.filter(official_key="test_template_key").count() == 1 - - -@pytest.mark.django_db -def test_sync_templates_idempotent(template_dir): - _write_json(template_dir, "test_template.json", MINIMAL_TEMPLATE) - sync_official_device_templates() - sync_official_device_templates() - assert DeviceTemplate.objects.filter(official_key="test_template_key").count() == 1 - - -# --------------------------------------------------------------------------- -# sync_official_device_templates — profile linking (three lookup paths) -# --------------------------------------------------------------------------- - -@pytest.mark.django_db -def test_sync_templates_links_profiles_via_official_key(tmp_path, settings): - """Primary path: profile resolved by its official_key value.""" - (tmp_path / "SNMP" / "data" / "official_profiles").mkdir(parents=True) - tdir = tmp_path / "SNMP" / "data" / "official_device_templates" - tdir.mkdir(parents=True) - settings.BASE_DIR = str(tmp_path) - - profile = Profile.objects.create( - official_key="linked_profile_key", - name="linked_profile.json", - vendor="Generic", - profile_data={"is_official_placeholder": True}, - ) - data = {**MINIMAL_TEMPLATE, "profiles": ["linked_profile_key"]} - _write_json(str(tdir), "test_template.json", data) - sync_official_device_templates() - - template = DeviceTemplate.objects.get(official_key="test_template_key") - assert profile in template.profiles.all() - - -@pytest.mark.django_db -def test_sync_templates_links_profiles_via_json_name_fallback(tmp_path, settings): - """ - Fallback path: profile referenced without .json extension is found by - appending .json to the lookup name (un-migrated official profile). - """ - (tmp_path / "SNMP" / "data" / "official_profiles").mkdir(parents=True) - tdir = tmp_path / "SNMP" / "data" / "official_device_templates" - tdir.mkdir(parents=True) - settings.BASE_DIR = str(tmp_path) - - profile = Profile.objects.create( - official_key=None, - name="linked_profile.json", - vendor="Generic", - profile_data={"is_official_placeholder": True}, - ) - data = {**MINIMAL_TEMPLATE, "profiles": ["linked_profile"]} - _write_json(str(tdir), "test_template.json", data) - sync_official_device_templates() - - template = DeviceTemplate.objects.get(official_key="test_template_key") - assert profile in template.profiles.all() - - -@pytest.mark.django_db -def test_sync_templates_links_profiles_via_bare_name_fallback(tmp_path, settings): - """ - Fallback path: user-created custom profile matched by exact bare name. - """ - (tmp_path / "SNMP" / "data" / "official_profiles").mkdir(parents=True) - tdir = tmp_path / "SNMP" / "data" / "official_device_templates" - tdir.mkdir(parents=True) - settings.BASE_DIR = str(tmp_path) - - profile = Profile.objects.create( - official_key=None, - name="custom_profile", - vendor="Generic", - profile_data={"get": {}, "walk": {}, "table": {}}, - ) - data = {**MINIMAL_TEMPLATE, "profiles": ["custom_profile"]} - _write_json(str(tdir), "test_template.json", data) - sync_official_device_templates() - - template = DeviceTemplate.objects.get(official_key="test_template_key") - assert profile in template.profiles.all() - - -@pytest.mark.django_db -def test_sync_templates_links_multiple_profiles(tmp_path, settings): - (tmp_path / "SNMP" / "data" / "official_profiles").mkdir(parents=True) - tdir = tmp_path / "SNMP" / "data" / "official_device_templates" - tdir.mkdir(parents=True) - settings.BASE_DIR = str(tmp_path) - - profiles = [] - for i in range(3): - p = Profile.objects.create( - official_key=f"profile_key_{i}", - name=f"profile_{i}.json", - vendor="Generic", - profile_data={"is_official_placeholder": True}, - ) - profiles.append(p) - - data = {**MINIMAL_TEMPLATE, "profiles": [f"profile_key_{i}" for i in range(3)]} - _write_json(str(tdir), "test_template.json", data) - sync_official_device_templates() - - template = DeviceTemplate.objects.get(official_key="test_template_key") - assert template.profiles.count() == 3 - - -# --------------------------------------------------------------------------- -# sync_official_device_templates — profile linking edge cases -# --------------------------------------------------------------------------- - -@pytest.mark.django_db -def test_sync_templates_missing_profile_skipped_gracefully(template_dir): - """ - A profile name listed in the JSON that doesn't exist in the DB should be - silently skipped. The template itself must still be created. - """ - data = {**MINIMAL_TEMPLATE, "profiles": ["nonexistent_profile"]} - _write_json(template_dir, "test_template.json", data) - sync_official_device_templates() - - template = DeviceTemplate.objects.get(official_key="test_template_key") - assert template.profiles.count() == 0 - - -@pytest.mark.django_db -def test_sync_templates_empty_profiles_list_does_not_clear_existing(tmp_path, settings): - """ - profiles: [] in JSON is treated as a no-op — existing M2M rows must not - be cleared. This is the current documented behaviour of the if-guard. - """ - (tmp_path / "SNMP" / "data" / "official_profiles").mkdir(parents=True) - tdir = tmp_path / "SNMP" / "data" / "official_device_templates" - tdir.mkdir(parents=True) - settings.BASE_DIR = str(tmp_path) - - profile = Profile.objects.create( - official_key="kept_profile", - name="kept_profile.json", - vendor="Generic", - profile_data={"is_official_placeholder": True}, - ) - template = DeviceTemplate.objects.create( - official_key="test_template_key", - name="Test Template", - vendor="Generic", - official=True, - ) - template.profiles.add(profile) - assert template.profiles.count() == 1 - - _write_json(str(tdir), "test_template.json", {**MINIMAL_TEMPLATE, "profiles": []}) - sync_official_device_templates() - - template.refresh_from_db() - assert template.profiles.count() == 1 - - -# --------------------------------------------------------------------------- -# sync_official_device_templates — fix #6 and skip/guard cases -# --------------------------------------------------------------------------- - -@pytest.mark.django_db -def test_sync_templates_missing_vendor_defaults_to_any(template_dir): - """Regression test for fix #6 on the template sync path.""" - no_vendor = {k: v for k, v in MINIMAL_TEMPLATE.items() if k != "vendor"} - _write_json(template_dir, "no_vendor.json", no_vendor) - sync_official_device_templates() - - template = DeviceTemplate.objects.get(official_key="test_template_key") - assert template.vendor == "Any" - - -@pytest.mark.django_db -def test_sync_templates_skips_file_without_official_key(template_dir): - no_key = {k: v for k, v in MINIMAL_TEMPLATE.items() if k != "official_key"} - _write_json(template_dir, "no_key.json", no_key) - sync_official_device_templates() - assert DeviceTemplate.objects.count() == 0 - - -@pytest.mark.django_db -def test_sync_templates_handles_malformed_json_gracefully(template_dir): - with open(os.path.join(template_dir, "bad.json"), "w") as f: - f.write("{ not valid json }") - sync_official_device_templates() - assert DeviceTemplate.objects.count() == 0 - - -@pytest.mark.django_db -def test_sync_templates_handles_empty_directory(template_dir): - sync_official_device_templates() - assert DeviceTemplate.objects.count() == 0 - - -# --------------------------------------------------------------------------- -# Management command — call_command end-to-end -# --------------------------------------------------------------------------- - -@pytest.mark.django_db -def test_command_syncs_profiles_and_templates(both_dirs): - profile_dir, template_dir = both_dirs - _write_json(profile_dir, "p.json", {**MINIMAL_PROFILE, "official_key": "cmd_profile_key"}) - _write_json(template_dir, "t.json", {**MINIMAL_TEMPLATE, "official_key": "cmd_template_key"}) - - call_command("sync_snmp_official_data", stdout=StringIO()) - - assert Profile.objects.filter(official_key="cmd_profile_key").exists() - assert DeviceTemplate.objects.filter(official_key="cmd_template_key").exists() - - -@pytest.mark.django_db -def test_command_output_mentions_profiles_and_templates(both_dirs): - profile_dir, _ = both_dirs - _write_json(profile_dir, "p.json", {**MINIMAL_PROFILE, "official_key": "out_key"}) - - out = StringIO() - call_command("sync_snmp_official_data", stdout=out) - output = out.getvalue().lower() - - assert "profile" in output - assert "template" in output - - -@pytest.mark.django_db -def test_command_does_not_raise_on_malformed_json(both_dirs): - """A corrupt JSON file must not abort startup.""" - profile_dir, _ = both_dirs - with open(os.path.join(profile_dir, "bad.json"), "w") as f: - f.write("{ not valid json }") - - out = StringIO() - call_command("sync_snmp_official_data", stdout=out) - - -# --------------------------------------------------------------------------- -# Management command — --cleanup: stale-by-official_key records -# --------------------------------------------------------------------------- - -@pytest.mark.django_db -def test_command_without_cleanup_leaves_stale_records(both_dirs): - """Without --cleanup, stale official records must persist in the DB.""" - _, _ = both_dirs - Profile.objects.create( - official_key="stale_key", - name="stale.json", - vendor="Any", - profile_data={"is_official_placeholder": True}, - ) - - call_command("sync_snmp_official_data", stdout=StringIO()) - assert Profile.objects.filter(official_key="stale_key").exists() - - -@pytest.mark.django_db -def test_command_cleanup_deletes_unused_stale_profile(both_dirs): - """Stale official profile with no DeviceTemplate referencing it is deleted.""" - _, _ = both_dirs - Profile.objects.create( - official_key="stale_key", - name="stale.json", - vendor="Any", - profile_data={"is_official_placeholder": True}, - ) - - call_command("sync_snmp_official_data", cleanup=True, stdout=StringIO()) - assert not Profile.objects.filter(official_key="stale_key").exists() - - -@pytest.mark.django_db -def test_command_cleanup_orphans_in_use_stale_profile(both_dirs): - """ - Stale official profile referenced by a DeviceTemplate must be marked - is_orphaned=True instead of deleted. - """ - _, _ = both_dirs - stale_profile = Profile.objects.create( - official_key="stale_in_use_key", - name="stale_in_use.json", - vendor="Any", - profile_data={"is_official_placeholder": True}, - ) - using_template = DeviceTemplate.objects.create( - name="Uses Stale Profile", - vendor="Generic", - official=False, - ) - using_template.profiles.add(stale_profile) - - call_command("sync_snmp_official_data", cleanup=True, stdout=StringIO()) - - stale_profile.refresh_from_db() - assert Profile.objects.filter(official_key="stale_in_use_key").exists() - assert stale_profile.profile_data.get("is_orphaned") is True - - -@pytest.mark.django_db -def test_command_cleanup_deletes_unused_stale_template(both_dirs): - """Stale official template with no devices assigned is deleted.""" - _, _ = both_dirs - DeviceTemplate.objects.create( - official_key="stale_template_key", - name="Stale Template", - vendor="Any", - official=True, - ) - - call_command("sync_snmp_official_data", cleanup=True, stdout=StringIO()) - assert not DeviceTemplate.objects.filter(official_key="stale_template_key").exists() - - -@pytest.mark.django_db -def test_command_cleanup_does_not_delete_in_use_stale_template(both_dirs): - """ - Stale official template with devices still assigned must be kept. - The command should log a warning but not delete it. - """ - from SNMP.models import Credential, Network, Device - from PipelineManager.models import Connection - - _, _ = both_dirs - stale_template = DeviceTemplate.objects.create( - official_key="stale_in_use_template", - name="Stale In Use", - vendor="Any", - official=True, - ) - - conn = Connection.objects.create( - name="Test Conn", - connection_type="CENTRALIZED", - host="https://localhost:9200", - username="elastic", - password="changeme", - ) - cred = Credential.objects.create(name="cred", version="2c", community="public") - net = Network.objects.create( - name="net", - network_range="10.0.0.0/24", - connection=conn, - discovery_credential=cred, - interval=30, - ) - Device.objects.create( - name="test_device", - ip_address="10.0.0.1", - port=161, - retries=2, - timeout=1000, - credential=cred, - network=net, - device_template=stale_template, - ) - - call_command("sync_snmp_official_data", cleanup=True, stdout=StringIO()) - assert DeviceTemplate.objects.filter(official_key="stale_in_use_template").exists() - - -# --------------------------------------------------------------------------- -# Management command — --cleanup: legacy (stale-by-flag) records -# --------------------------------------------------------------------------- - -@pytest.mark.django_db -def test_command_cleanup_deletes_legacy_stale_profile(both_dirs): - """ - Old-style official profiles (no official_key, has is_official_placeholder) - that were not backfilled during sync are treated as stale and deleted. - """ - _, _ = both_dirs - legacy = Profile.objects.create( - official_key=None, - name="legacy_no_key.json", - vendor="Any", - profile_data={"is_official_placeholder": True}, - ) - - call_command("sync_snmp_official_data", cleanup=True, stdout=StringIO()) - assert not Profile.objects.filter(pk=legacy.pk).exists() - - -@pytest.mark.django_db -def test_command_cleanup_deletes_legacy_stale_template(both_dirs): - """ - Official templates with no official_key after sync ran are stale and - must be deleted when not in use. - """ - _, _ = both_dirs - legacy = DeviceTemplate.objects.create( - official_key=None, - name="Legacy Template", - vendor="Any", - official=True, - ) - - call_command("sync_snmp_official_data", cleanup=True, stdout=StringIO()) - assert not DeviceTemplate.objects.filter(pk=legacy.pk).exists() - - -# --------------------------------------------------------------------------- -# Management command — --cleanup: output counts -# --------------------------------------------------------------------------- - -@pytest.mark.django_db -def test_command_cleanup_output_reports_deleted_counts(both_dirs): - _, _ = both_dirs - Profile.objects.create( - official_key="deleted_profile", - name="deleted.json", - vendor="Any", - profile_data={"is_official_placeholder": True}, - ) - - out = StringIO() - call_command("sync_snmp_official_data", cleanup=True, stdout=out) - output = out.getvalue().lower() - - assert "deleted" in output or "profile" in output diff --git a/src/logstashui/SNMP/tests/test_inline_grounding.py b/src/logstashui/SNMP/tests/test_inline_grounding.py deleted file mode 100644 index 6b121d8..0000000 --- a/src/logstashui/SNMP/tests/test_inline_grounding.py +++ /dev/null @@ -1,66 +0,0 @@ -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. -""" -Unit tests for SNMP/inline_grounding.py. - -No network / no live LLM: a temp data dir is used so the selection logic is -deterministic. -""" -import json -import os -import tempfile -from unittest import mock - -from django.test import SimpleTestCase - -from SNMP import inline_grounding as ig - - -def _seed(tmp): - for sub in ("official_profiles", "schema_reference", "mib_reference"): - os.makedirs(os.path.join(tmp, sub)) - with open(os.path.join(tmp, "authoring_instructions.md"), "w") as f: - f.write("AUTHORING RULES") - with open(os.path.join(tmp, "schema_reference", "s.md"), "w") as f: - f.write("NAMING DICT") - with open(os.path.join(tmp, "mib_reference", "std_x.json"), "w") as f: - json.dump({"name": "std_x"}, f) - for name, vendor in [("generic_interfaces", "Any"), ("cisco_x", "Cisco"), ("arista_x", "Arista")]: - with open(os.path.join(tmp, "official_profiles", f"{name}.json"), "w") as f: - json.dump({"name": name, "vendor": vendor, "get": {"o": "1"}}, f) - - -class InlineGroundingTests(SimpleTestCase): - def test_relevance_rules(self): - self.assertTrue(ig._relevant({"vendor": "Any"}, "Arista")) - self.assertTrue(ig._relevant({"vendor": ""}, "whatever")) - self.assertTrue(ig._relevant({"vendor": "Arista"}, "Arista Networks EOS")) - self.assertFalse(ig._relevant({"vendor": "Cisco"}, "Arista")) - self.assertFalse(ig._relevant({"vendor": "Cisco"}, "")) - - def test_grounding_includes_generic_and_vendor_match_only(self): - with tempfile.TemporaryDirectory() as tmp: - _seed(tmp) - with mock.patch.object(ig, "_DATA", tmp): - g = ig.build_grounding("Arista Networks EOS 4.36") - self.assertIn("NAMING DICT", g) - self.assertIn("std_x", g) - self.assertIn("generic_interfaces", g) - self.assertIn("arista_x", g) - self.assertNotIn("cisco_x", g) - - def test_grounding_has_all_sections(self): - with tempfile.TemporaryDirectory() as tmp: - _seed(tmp) - with mock.patch.object(ig, "_DATA", tmp): - g = ig.build_grounding("Any") - self.assertIn("FIELD NAMING SCHEMA", g) - self.assertIn("STANDARD-MIB REFERENCES", g) - self.assertIn("REFERENCE PROFILES", g) - - def test_load_instructions_reads_local_file(self): - with tempfile.TemporaryDirectory() as tmp: - _seed(tmp) - with mock.patch.object(ig, "_DATA", tmp): - self.assertEqual(ig.load_instructions(), "AUTHORING RULES") diff --git a/src/logstashui/SNMP/tests/test_models.py b/src/logstashui/SNMP/tests/test_models.py deleted file mode 100644 index 7896c35..0000000 --- a/src/logstashui/SNMP/tests/test_models.py +++ /dev/null @@ -1,498 +0,0 @@ -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. - -import pytest -from django.core.exceptions import ValidationError -from django.utils import timezone - -from SNMP.models import ( - Credential, Device, DeviceTemplate, Network, Profile, SNMPDeploymentState -) -from PipelineManager.models import Connection - - -# --------------------------------------------------------------------------- -# Shared fixtures -# --------------------------------------------------------------------------- - -@pytest.fixture -def test_connection(db): - return Connection.objects.create( - name='Test Connection', - connection_type='CENTRALIZED', - host='https://localhost:9200', - username='elastic', - password='changeme', - ) - - -@pytest.fixture -def test_credential_v2c(db): - return Credential.objects.create( - name='Cred v2c', - version='2c', - community='public', - ) - - -@pytest.fixture -def test_network(db, test_connection, test_credential_v2c): - return Network.objects.create( - name='Net', - network_range='10.0.0.0/8', - connection=test_connection, - ) - - -@pytest.fixture -def default_template(db): - return DeviceTemplate.objects.create( - name='default', - vendor='Any', - official=True, - ) - - -# =========================================================================== -# SNMPDeploymentState -# =========================================================================== - -@pytest.mark.django_db -class TestSNMPDeploymentState: - - def test_mark_config_changed_creates_state(self): - """mark_config_changed creates the singleton row when it doesn't exist.""" - SNMPDeploymentState.objects.all().delete() - SNMPDeploymentState.mark_config_changed() - state = SNMPDeploymentState.objects.get(id=1) - assert state.last_config_change is not None - - def test_mark_config_changed_updates_timestamp(self): - """Successive calls to mark_config_changed advance the timestamp.""" - SNMPDeploymentState.objects.all().delete() - SNMPDeploymentState.mark_config_changed() - first = SNMPDeploymentState.objects.get(id=1).last_config_change - SNMPDeploymentState.mark_config_changed() - second = SNMPDeploymentState.objects.get(id=1).last_config_change - assert second >= first - - def test_has_undeployed_changes_no_state(self): - """has_undeployed_changes returns True when no row exists (never deployed).""" - SNMPDeploymentState.objects.all().delete() - assert SNMPDeploymentState.has_undeployed_changes() is True - - def test_has_undeployed_changes_no_deployment(self): - """has_undeployed_changes is True when config changed but never deployed.""" - SNMPDeploymentState.objects.all().delete() - SNMPDeploymentState.mark_config_changed() - assert SNMPDeploymentState.has_undeployed_changes() is True - - def test_has_undeployed_changes_after_sync(self): - """has_undeployed_changes is False when last_deployment >= last_config_change.""" - SNMPDeploymentState.objects.all().delete() - now = timezone.now() - SNMPDeploymentState.objects.create( - id=1, - last_config_change=now, - last_deployment=now, - ) - assert SNMPDeploymentState.has_undeployed_changes() is False - - def test_has_undeployed_changes_after_new_change(self): - """has_undeployed_changes is True again when config changes after deployment.""" - from datetime import timedelta - SNMPDeploymentState.objects.all().delete() - # Seed the state with timestamps 60 seconds in the past so mark_config_changed - # will produce a strictly later timestamp. - past = timezone.now() - timedelta(seconds=60) - SNMPDeploymentState.objects.create( - id=1, - last_config_change=past, - last_deployment=past, - ) - SNMPDeploymentState.mark_config_changed() - assert SNMPDeploymentState.has_undeployed_changes() is True - - def test_str_never_deployed(self): - """__str__ returns 'Never deployed' when last_deployment is None.""" - SNMPDeploymentState.objects.all().delete() - state = SNMPDeploymentState.objects.create(id=1) - assert str(state) == 'Never deployed' - - def test_str_with_deployment(self): - """__str__ includes the timestamp when last_deployment is set.""" - SNMPDeploymentState.objects.all().delete() - now = timezone.now() - state = SNMPDeploymentState.objects.create(id=1, last_deployment=now) - assert 'Last deployed' in str(state) - - -# =========================================================================== -# DeviceTemplate.matches_device -# =========================================================================== - -@pytest.mark.django_db -class TestDeviceTemplateMatchesDevice: - - def test_matches_with_all_rules(self): - """matches_device returns True when all rules appear in device_info.""" - tmpl = DeviceTemplate.objects.create( - name='cisco_switch', - vendor='Cisco', - matching_rules=['cisco', 'catalyst'], - ) - assert tmpl.matches_device('Cisco Catalyst 9300 switch') is True - - def test_no_match_when_rule_absent(self): - """matches_device returns False when a rule is not found.""" - tmpl = DeviceTemplate.objects.create( - name='cisco_switch2', - vendor='Cisco', - matching_rules=['cisco', 'catalyst'], - ) - assert tmpl.matches_device('Juniper EX2300') is False - - def test_case_insensitive(self): - """matches_device is case-insensitive.""" - tmpl = DeviceTemplate.objects.create( - name='cisco_switch3', - vendor='Cisco', - matching_rules=['CISCO'], - ) - assert tmpl.matches_device('cisco ios') is True - - def test_empty_matching_rules(self): - """matches_device returns False when matching_rules is empty.""" - tmpl = DeviceTemplate.objects.create( - name='generic_tmpl', - vendor='Generic', - matching_rules=[], - ) - assert tmpl.matches_device('anything') is False - - def test_empty_device_info(self): - """matches_device returns False when device_info is empty/None.""" - tmpl = DeviceTemplate.objects.create( - name='cisco_switch4', - vendor='Cisco', - matching_rules=['cisco'], - ) - assert tmpl.matches_device('') is False - assert tmpl.matches_device(None) is False - - def test_partial_substring_match(self): - """A single matching rule appearing in a longer string is sufficient.""" - tmpl = DeviceTemplate.objects.create( - name='dell_idrac', - vendor='Dell', - matching_rules=['idrac'], - ) - assert tmpl.matches_device('Dell iDRAC 9 server') is True - - -# =========================================================================== -# DeviceTemplate.clean – matching_rules validation -# =========================================================================== - -@pytest.mark.django_db -class TestDeviceTemplateClean: - - def test_matching_rules_must_be_list(self): - """matching_rules must be a list; a dict raises ValidationError.""" - with pytest.raises(ValidationError): - DeviceTemplate.objects.create( - name='bad_rules_dict', - vendor='Any', - matching_rules={'key': 'value'}, - ) - - def test_matching_rules_items_must_be_strings(self): - """Each item in matching_rules must be a string.""" - with pytest.raises(ValidationError): - DeviceTemplate.objects.create( - name='bad_rules_int', - vendor='Any', - matching_rules=[1, 2, 3], - ) - - def test_empty_matching_rules_valid(self): - """An empty list is a valid matching_rules value.""" - tmpl = DeviceTemplate.objects.create( - name='empty_rules_ok', - vendor='Any', - matching_rules=[], - ) - assert tmpl.id is not None - - -# =========================================================================== -# Credential – decryption helpers -# =========================================================================== - -@pytest.mark.django_db -class TestCredentialDecryption: - - def test_get_community_returns_plaintext(self): - """get_community() decrypts and returns the community string.""" - cred = Credential.objects.create( - name='comm_test', - version='2c', - community='secret_community', - ) - cred.refresh_from_db() - assert cred.get_community() == 'secret_community' - - def test_get_community_none_when_blank(self): - """get_community() returns None when community is blank.""" - cred = Credential.objects.create( - name='comm_blank', - version='2c', - community='placeholder', # must pass model clean - ) - # Manually blank out after creation to avoid validation - Credential.objects.filter(pk=cred.pk).update(community='') - cred.refresh_from_db() - assert cred.get_community() is None - - def test_get_auth_pass_returns_plaintext(self): - """get_auth_pass() decrypts and returns the auth passphrase.""" - cred = Credential.objects.create( - name='auth_test', - version='3', - security_name='user1', - security_level='authPriv', - auth_protocol='sha', - auth_pass='authsecret', - priv_protocol='aes', - priv_pass='privsecret', - ) - cred.refresh_from_db() - assert cred.get_auth_pass() == 'authsecret' - - def test_get_priv_pass_returns_plaintext(self): - """get_priv_pass() decrypts and returns the priv passphrase.""" - cred = Credential.objects.create( - name='priv_test', - version='3', - security_name='user2', - security_level='authPriv', - auth_protocol='sha', - auth_pass='authsecret2', - priv_protocol='aes', - priv_pass='privsecret2', - ) - cred.refresh_from_db() - assert cred.get_priv_pass() == 'privsecret2' - - def test_get_auth_pass_none_when_blank(self): - """get_auth_pass() returns None when auth_pass is blank.""" - cred = Credential.objects.create( - name='no_auth_pass', - version='3', - security_name='user3', - security_level='noAuthNoPriv', - ) - cred.refresh_from_db() - assert cred.get_auth_pass() is None - - def test_double_save_does_not_double_encrypt(self): - """Saving a credential twice does not encrypt an already-encrypted value.""" - cred = Credential.objects.create( - name='double_save_test', - version='2c', - community='test_community', - ) - cred.refresh_from_db() - first_community = cred.community # encrypted token - cred.description = 'Updated' - cred.save() - cred.refresh_from_db() - assert cred.community == first_community - assert cred.get_community() == 'test_community' - - -# =========================================================================== -# Credential.clean – SNMP version validation -# =========================================================================== - -@pytest.mark.django_db -class TestCredentialClean: - - def test_v2c_requires_community(self): - """v2c credential requires a community string.""" - with pytest.raises(ValidationError): - cred = Credential(name='no_comm', version='2c', community='') - cred.full_clean() - - def test_v3_noauthnopriv_rejects_auth_fields(self): - """noAuthNoPriv should not have auth/priv fields set.""" - with pytest.raises(ValidationError): - Credential.objects.create( - name='bad_noauth', - version='3', - security_name='user', - security_level='noAuthNoPriv', - auth_protocol='sha', - auth_pass='pass', - ) - - def test_v3_authnopriv_requires_auth_protocol(self): - """authNoPriv requires auth_protocol.""" - with pytest.raises(ValidationError): - Credential.objects.create( - name='bad_authnopriv', - version='3', - security_name='user', - security_level='authNoPriv', - auth_protocol='', - auth_pass='pass', - ) - - def test_v3_authnopriv_rejects_priv_fields(self): - """authNoPriv must not have priv fields set.""" - with pytest.raises(ValidationError): - Credential.objects.create( - name='bad_priv', - version='3', - security_name='user', - security_level='authNoPriv', - auth_protocol='sha', - auth_pass='pass', - priv_protocol='aes', - ) - - def test_v3_authpriv_requires_all_fields(self): - """authPriv requires both auth and priv protocol/pass.""" - with pytest.raises(ValidationError): - Credential.objects.create( - name='bad_authpriv', - version='3', - security_name='user', - security_level='authPriv', - auth_protocol='sha', - auth_pass='pass', - priv_protocol='aes', - priv_pass='', # missing - ) - - -# =========================================================================== -# Network.clean – CIDR validation -# =========================================================================== - -@pytest.mark.django_db -class TestNetworkClean: - - def test_valid_cidr_saves(self, test_connection): - """A valid CIDR network range saves without error.""" - net = Network.objects.create( - name='valid_net', - network_range='192.168.0.0/16', - connection=test_connection, - ) - assert net.id is not None - - def test_invalid_cidr_raises_validation_error(self, test_connection): - """An invalid CIDR raises ValidationError on save.""" - with pytest.raises(ValidationError): - Network.objects.create( - name='invalid_net', - network_range='not-a-cidr', - connection=test_connection, - ) - - def test_host_cidr_accepted_non_strict(self, test_connection): - """Non-strict CIDR (host bits set) is accepted by the model.""" - net = Network.objects.create( - name='host_cidr', - network_range='192.168.1.1/24', - connection=test_connection, - ) - assert net.id is not None - - -# =========================================================================== -# Device.clean – validation -# =========================================================================== - -@pytest.mark.django_db -class TestDeviceClean: - - def test_device_requires_ip_or_hostname(self, test_credential_v2c, test_network): - """Device.clean raises ValidationError if neither ip_address nor hostname is set.""" - with pytest.raises(ValidationError): - Device.objects.create( - name='no_addr', - ip_address=None, - hostname=None, - credential=test_credential_v2c, - network=test_network, - ) - - def test_device_invalid_ip_raises(self, test_credential_v2c, test_network): - """Device.clean raises ValidationError for an invalid IP address.""" - with pytest.raises(ValidationError): - Device.objects.create( - name='bad_ip', - ip_address='999.999.999.999', - credential=test_credential_v2c, - network=test_network, - ) - - def test_device_valid_hostname_only(self, test_credential_v2c, test_network): - """A device with only a hostname (no IP) is valid.""" - device = Device.objects.create( - name='hostname_only_dev', - hostname='mydevice.example.com', - ip_address=None, - credential=test_credential_v2c, - network=test_network, - ) - assert device.id is not None - - def test_device_str_uses_ip(self, test_credential_v2c, test_network): - """Device.__str__ uses the IP address when present.""" - device = Device.objects.create( - name='str_test', - ip_address='10.0.0.1', - credential=test_credential_v2c, - network=test_network, - ) - assert '10.0.0.1' in str(device) - - def test_device_str_fallback_no_address(self, test_credential_v2c, test_network): - """Device.__str__ falls back to 'no address' when both ip/hostname are None after object construction.""" - # Bypass model validation by using update() to set both to None - device = Device.objects.create( - name='str_no_addr', - ip_address='1.2.3.4', - credential=test_credential_v2c, - network=test_network, - ) - Device.objects.filter(pk=device.pk).update(ip_address=None, hostname=None) - device.refresh_from_db() - assert 'no address' in str(device) - - -# =========================================================================== -# Profile.clean – validation -# =========================================================================== - -@pytest.mark.django_db -class TestProfileClean: - - def test_profile_data_must_be_dict(self): - """Profile.clean raises ValidationError when profile_data is not a dict.""" - with pytest.raises(ValidationError): - p = Profile(name='bad_profile', vendor='Generic', profile_data='not a dict') - p.full_clean() - - def test_valid_profile_saves(self): - """A profile with valid dict profile_data saves successfully.""" - p = Profile.objects.create( - name='ok_profile', - vendor='Generic', - profile_data={'get': {'sysDescr': '1.3.6.1.2.1.1.1.0'}}, - ) - assert p.id is not None diff --git a/src/logstashui/SNMP/tests/test_network_map.py b/src/logstashui/SNMP/tests/test_network_map.py deleted file mode 100644 index cba6d1c..0000000 --- a/src/logstashui/SNMP/tests/test_network_map.py +++ /dev/null @@ -1,618 +0,0 @@ -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. - -""" -Tests for SNMP.network_map — adjacency-to-graph conversion and the -get_networks_list / get_network_map_data view helpers. -All Elasticsearch I/O is mocked. -""" - -import json -import pytest -from unittest.mock import patch, MagicMock -from django.test import RequestFactory - -from SNMP.network_map import ( - convert_adjacency_to_graph, - get_networks_list, - get_network_map_data, - get_cdp_adjacencies, - get_edge_interface_detail, -) -from SNMP.models import Network, Device, Credential -from PipelineManager.models import Connection - - -# =========================================================================== -# Fixtures -# =========================================================================== - -@pytest.fixture -def test_connection(db): - return Connection.objects.create( - name='NM Test Connection', - connection_type='CENTRALIZED', - host='https://localhost:9200', - username='elastic', - password='changeme' - ) - - -@pytest.fixture -def test_credential(db): - return Credential.objects.create( - name='nm_test_cred', - version='2c', - community='public' - ) - - -@pytest.fixture -def test_network(db, test_connection, test_credential): - return Network.objects.create( - name='NM Test Network', - network_range='10.0.0.0/24', - connection=test_connection, - discovery_credential=test_credential, - interval=30 - ) - - -@pytest.fixture -def test_device(db, test_network, test_credential): - return Device.objects.create( - name='switch-a', - ip_address='10.0.0.1', - port=161, - retries=1, - timeout=500, - credential=test_credential, - network=test_network, - ) - - -@pytest.fixture -def rf(): - return RequestFactory() - - -# =========================================================================== -# convert_adjacency_to_graph -# =========================================================================== - -class TestConvertAdjacencyToGraph: - """ - Tests for the pure graph-conversion function. - The only DB touch is the device-ID lookup at the end, which is - guarded by try/except; we let it fail silently in the test DB. - """ - - def test_empty_adjacency_table_returns_empty_graph(self, db): - result = convert_adjacency_to_graph({}) - assert result == {'nodes': [], 'edges': []} - - def test_single_device_no_neighbors_creates_node(self, db): - adjacency = { - 'Production (10.0.0.0/24)': { - 'switch-a': {} - } - } - result = convert_adjacency_to_graph(adjacency) - assert len(result['nodes']) == 1 - assert result['nodes'][0]['id'] == 'switch-a' - assert result['edges'] == [] - - def test_device_with_one_neighbor_creates_edge(self, db): - adjacency = { - 'Production': { - 'switch-a': { - 'GigabitEthernet0/1': { - 'device_id': 'switch-b', - 'port': 'GigabitEthernet0/2', - 'platform': 'Cisco IOS', - 'capabilities': 'Switch', - 'address': '10.0.0.2', - 'version': '15.2' - } - } - } - } - result = convert_adjacency_to_graph(adjacency) - assert len(result['nodes']) == 2 - assert len(result['edges']) == 1 - edge = result['edges'][0] - assert edge['source'] == 'switch-a' - assert edge['target'] == 'switch-b' - assert edge['source_interface'] == 'GigabitEthernet0/1' - assert edge['target_interface'] == 'GigabitEthernet0/2' - - def test_bidirectional_connection_creates_single_edge(self, db): - adjacency = { - 'Production': { - 'switch-a': { - 'Gi0/1': { - 'device_id': 'switch-b', 'port': 'Gi0/2', - 'platform': '', 'capabilities': '', 'address': '', 'version': '' - } - }, - 'switch-b': { - 'Gi0/2': { - 'device_id': 'switch-a', 'port': 'Gi0/1', - 'platform': '', 'capabilities': '', 'address': '', 'version': '' - } - } - } - } - result = convert_adjacency_to_graph(adjacency) - assert len(result['edges']) == 1 - - def test_managed_device_has_managed_true(self, db): - adjacency = { - 'Production': { - 'switch-a': { - 'Gi0/1': { - 'device_id': 'external-router', 'port': 'Eth0', - 'platform': '', 'capabilities': '', 'address': '', 'version': '' - } - } - } - } - result = convert_adjacency_to_graph(adjacency) - managed_node = next(n for n in result['nodes'] if n['id'] == 'switch-a') - assert managed_node['managed'] is True - - def test_discovered_only_device_has_managed_false(self, db): - adjacency = { - 'Production': { - 'switch-a': { - 'Gi0/1': { - 'device_id': 'external-router', 'port': 'Eth0', - 'platform': '', 'capabilities': '', 'address': '', 'version': '' - } - } - } - } - result = convert_adjacency_to_graph(adjacency) - discovered_node = next(n for n in result['nodes'] if n['id'] == 'external-router') - assert discovered_node['managed'] is False - - def test_device_that_appears_in_both_sides_is_managed(self, db): - adjacency = { - 'Production': { - 'switch-a': { - 'Gi0/1': { - 'device_id': 'switch-b', 'port': 'Gi0/2', - 'platform': '', 'capabilities': '', 'address': '', 'version': '' - } - }, - 'switch-b': {} - } - } - result = convert_adjacency_to_graph(adjacency) - b_node = next(n for n in result['nodes'] if n['id'] == 'switch-b') - assert b_node['managed'] is True - - def test_interface_count_increments_per_neighbor(self, db): - adjacency = { - 'Production': { - 'switch-a': { - 'Gi0/1': { - 'device_id': 'switch-b', 'port': 'Gi0/2', - 'platform': '', 'capabilities': '', 'address': '', 'version': '' - }, - 'Gi0/2': { - 'device_id': 'switch-c', 'port': 'Gi0/1', - 'platform': '', 'capabilities': '', 'address': '', 'version': '' - } - } - } - } - result = convert_adjacency_to_graph(adjacency) - a_node = next(n for n in result['nodes'] if n['id'] == 'switch-a') - assert a_node['interface_count'] == 2 - - def test_neighbor_without_device_id_skipped(self, db): - adjacency = { - 'Production': { - 'switch-a': { - 'Gi0/1': { - 'device_id': '', # empty — no neighbor name - 'port': 'Gi0/2', - 'platform': '', 'capabilities': '', 'address': '', 'version': '' - } - } - } - } - result = convert_adjacency_to_graph(adjacency) - # No edge should be created for an empty device_id - assert result['edges'] == [] - - def test_multiple_networks_all_included(self, db): - adjacency = { - 'Network A': {'device-a': {}}, - 'Network B': {'device-b': {}}, - } - result = convert_adjacency_to_graph(adjacency) - node_ids = {n['id'] for n in result['nodes']} - assert 'device-a' in node_ids - assert 'device-b' in node_ids - - def test_db_device_id_enrichment(self, test_device, db): - """Managed nodes get a device_id from the DB if their id matches.""" - adjacency = { - 'NM Test Network (10.0.0.0/24)': { - 'switch-a': {} - } - } - result = convert_adjacency_to_graph(adjacency) - # Node 'switch-a' matches the device name in the DB - a_node = next(n for n in result['nodes'] if n['id'] == 'switch-a') - assert a_node.get('device_id') == test_device.id - - def test_edge_contains_platform_and_capabilities(self, db): - adjacency = { - 'Production': { - 'switch-a': { - 'Gi0/1': { - 'device_id': 'switch-b', - 'port': 'Gi0/2', - 'platform': 'Cisco 3750', - 'capabilities': 'Switch Router', - 'address': '10.0.0.2', - 'version': '15.2' - } - } - } - } - result = convert_adjacency_to_graph(adjacency) - edge = result['edges'][0] - assert edge['platform'] == 'Cisco 3750' - assert edge['capabilities'] == 'Switch Router' - - -# =========================================================================== -# get_networks_list -# =========================================================================== - -@pytest.mark.django_db -class TestGetNetworksList: - - def test_no_networks_returns_empty_list(self, rf): - request = rf.get('/SNMP/GetNetworksList/') - response = get_networks_list(request) - data = json.loads(response.content) - assert data['success'] is True - assert data['networks'] == [] - - def test_returns_network_with_correct_fields(self, rf, test_network): - request = rf.get('/SNMP/GetNetworksList/') - response = get_networks_list(request) - data = json.loads(response.content) - assert data['success'] is True - assert len(data['networks']) == 1 - network = data['networks'][0] - assert network['id'] == test_network.id - assert network['name'] == test_network.name - assert network['network_range'] == test_network.network_range - assert 'device_count' in network - - def test_device_count_is_correct(self, rf, test_network, test_device): - request = rf.get('/SNMP/GetNetworksList/') - response = get_networks_list(request) - data = json.loads(response.content) - assert data['networks'][0]['device_count'] == 1 - - def test_networks_returned_alphabetically(self, rf, db, test_credential, test_connection): - Network.objects.create( - name='Zebra Network', network_range='10.2.0.0/24', - connection=test_connection, discovery_credential=test_credential, interval=30 - ) - Network.objects.create( - name='Alpha Network', network_range='10.3.0.0/24', - connection=test_connection, discovery_credential=test_credential, interval=30 - ) - request = rf.get('/SNMP/GetNetworksList/') - response = get_networks_list(request) - data = json.loads(response.content) - names = [n['name'] for n in data['networks']] - assert names == sorted(names) - - -# =========================================================================== -# get_network_map_data -# =========================================================================== - -@pytest.mark.django_db -class TestGetNetworkMapData: - - @patch('SNMP.network_map.get_cdp_adjacencies') - def test_no_networks_returns_empty_graph(self, mock_cdp, rf): - mock_cdp.return_value = { - 'success': False, - 'error': 'No connections', - 'adjacency_table': {} - } - request = rf.get('/SNMP/GetNetworkMap/') - response = get_network_map_data(request) - data = json.loads(response.content) - assert data['graph']['nodes'] == [] - assert data['graph']['edges'] == [] - - @patch('SNMP.network_map.get_cdp_adjacencies') - def test_with_adjacency_data_returns_graph(self, mock_cdp, rf): - mock_cdp.return_value = { - 'success': True, - 'adjacency_table': { - 'Production': { - 'switch-a': { - 'Gi0/1': { - 'device_id': 'switch-b', 'port': 'Gi0/2', - 'platform': '', 'capabilities': '', 'address': '', 'version': '' - } - } - } - }, - 'errors': None - } - request = rf.get('/SNMP/GetNetworkMap/') - response = get_network_map_data(request) - data = json.loads(response.content) - assert data['success'] is True - assert len(data['graph']['nodes']) == 2 - assert len(data['graph']['edges']) == 1 - - @patch('SNMP.network_map.get_cdp_adjacencies') - def test_network_filter_passed_to_cdp(self, mock_cdp, rf): - mock_cdp.return_value = { - 'success': False, - 'adjacency_table': {}, - 'error': 'none' - } - request = rf.get('/SNMP/GetNetworkMap/?networks=1&networks=2') - get_network_map_data(request) - mock_cdp.assert_called_once_with(network_ids=[1, 2]) - - @patch('SNMP.network_map.get_cdp_adjacencies') - def test_no_network_filter_passes_none(self, mock_cdp, rf): - mock_cdp.return_value = { - 'success': False, - 'adjacency_table': {}, - 'error': 'none' - } - request = rf.get('/SNMP/GetNetworkMap/') - get_network_map_data(request) - mock_cdp.assert_called_once_with(network_ids=None) - - @patch('SNMP.network_map.get_cdp_adjacencies') - def test_exception_returns_500_response(self, mock_cdp, rf): - mock_cdp.side_effect = Exception('Unexpected failure') - request = rf.get('/SNMP/GetNetworkMap/') - response = get_network_map_data(request) - assert response.status_code == 500 - data = json.loads(response.content) - assert data['success'] is False - - -# =========================================================================== -# get_cdp_adjacencies -# =========================================================================== - -@pytest.mark.django_db -class TestGetCdpAdjacencies: - """ - get_cdp_adjacencies queries ES for CDP/LLDP neighbor data. - All ES I/O is mocked; only the Django ORM layer is real. - """ - - def _empty_cdp_response(self): - """ES response with no CDP buckets.""" - return {'aggregations': {'cdp_adjacencies': {'buckets': []}}} - - def _cdp_response(self, host_sysname, table_index, neighbor_device_id, neighbor_port, - polled_address='10.0.0.1', network_name=''): - """Minimal ES response with one CDP bucket.""" - return { - 'aggregations': { - 'cdp_adjacencies': { - 'buckets': [ - { - 'key': {'host_name': host_sysname, 'cdp_row_index': table_index}, - 'latest': { - 'hits': { - 'hits': [ - { - '_source': { - 'host': { - 'sysname': host_sysname, - 'polled_address': polled_address, - 'hostname': '', - }, - 'network': { - 'name': network_name, - 'neighbor': { - 'index': table_index, - 'device_id': neighbor_device_id, - 'port': neighbor_port, - 'platform': 'Cisco IOS', - 'capabilities': 'Switch', - 'address': '10.0.0.2', - 'version': '15.2', - } - }, - 'event': {'category': 'network.neighbor'}, - } - } - ] - } - } - } - ] - } - } - } - - def test_no_networks_returns_failure(self): - result = get_cdp_adjacencies() - assert result['success'] is False - assert 'adjacency_table' in result - - def test_network_without_connection_not_queried(self, test_credential, db): - Network.objects.create( - name='No Conn', - network_range='10.99.0.0/24', - discovery_credential=test_credential, - interval=30, - connection=None, - ) - result = get_cdp_adjacencies() - assert result['success'] is False - - @patch('SNMP.network_map.get_elastic_connection') - def test_empty_cdp_response_returns_empty_adjacency(self, mock_get_es, test_network): - mock_es = MagicMock() - mock_es.search.return_value = self._empty_cdp_response() - mock_get_es.return_value = mock_es - - result = get_cdp_adjacencies() - assert result['success'] is True - assert result['adjacency_table'] == {} - assert result['errors'] is None - - @patch('SNMP.network_map.get_elastic_connection') - def test_cdp_data_populates_adjacency_table(self, mock_get_es, test_network, test_device): - mock_es = MagicMock() - mock_es.search.side_effect = [ - self._cdp_response( - host_sysname='switch-a', - table_index='1.1', - neighbor_device_id='switch-b', - neighbor_port='Gi0/2', - polled_address='10.0.0.1', - network_name='NM Test Network (10.0.0.0/24)', - ), - {'hits': {'hits': []}}, # interface name lookup returns nothing - ] - mock_get_es.return_value = mock_es - - result = get_cdp_adjacencies() - assert result['success'] is True - # adjacency table is non-empty - assert result['adjacency_table'] - - @patch('SNMP.network_map.get_elastic_connection') - def test_es_error_recorded_in_errors_list(self, mock_get_es, test_network): - mock_get_es.side_effect = Exception('ES down') - result = get_cdp_adjacencies() - assert result['success'] is True - assert result['errors'] is not None - assert len(result['errors']) == 1 - - @patch('SNMP.network_map.get_elastic_connection') - def test_network_id_filter_restricts_scope(self, mock_get_es, test_network): - mock_es = MagicMock() - mock_es.search.return_value = self._empty_cdp_response() - mock_get_es.return_value = mock_es - - result = get_cdp_adjacencies(network_ids=[test_network.id]) - assert result['success'] is True - - @patch('SNMP.network_map.get_elastic_connection') - def test_outer_exception_returns_failure(self, mock_get_es, test_network): - # Trigger the outer try/except by making Network.objects.filter raise - with patch('SNMP.network_map.Network.objects') as mock_objs: - mock_objs.filter.side_effect = Exception('DB error') - result = get_cdp_adjacencies() - assert result['success'] is False - assert 'error' in result - - -# =========================================================================== -# get_edge_interface_detail -# =========================================================================== - -@pytest.mark.django_db -class TestGetEdgeInterfaceDetail: - - @pytest.fixture - def rf(self): - from django.test import RequestFactory - return RequestFactory() - - def test_missing_source_returns_400(self, rf): - request = rf.get('/SNMP/GetEdgeInterfaceDetail/') - response = get_edge_interface_detail(request) - assert response.status_code == 400 - data = json.loads(response.content) - assert data['success'] is False - - def test_missing_source_iface_returns_400(self, rf): - request = rf.get('/SNMP/GetEdgeInterfaceDetail/?source=switch-a') - response = get_edge_interface_detail(request) - assert response.status_code == 400 - - def test_no_es_connections_returns_400(self, rf): - request = rf.get( - '/SNMP/GetEdgeInterfaceDetail/?source=switch-a&source_iface=Gi0/1' - ) - response = get_edge_interface_detail(request) - assert response.status_code == 400 - data = json.loads(response.content) - assert data['success'] is False - - @patch('SNMP.network_map.get_elastic_connection') - def test_returns_interface_data_for_source(self, mock_get_es, rf, test_network): - mock_es = MagicMock() - mock_es.search.return_value = { - 'hits': { - 'hits': [ - {'_source': {'interface': {'name': 'Gi0/1', 'speed': 1000}}} - ] - } - } - mock_get_es.return_value = mock_es - - request = rf.get( - '/SNMP/GetEdgeInterfaceDetail/' - '?source=switch-a&source_iface=Gi0/1' - '&target=switch-b&target_iface=Gi0/2' - ) - response = get_edge_interface_detail(request) - assert response.status_code == 200 - data = json.loads(response.content) - assert data['success'] is True - assert data['source']['sysname'] == 'switch-a' - assert data['source']['iface_name'] == 'Gi0/1' - assert data['target']['sysname'] == 'switch-b' - - @patch('SNMP.network_map.get_elastic_connection') - def test_no_hits_returns_none_interface(self, mock_get_es, rf, test_network): - mock_es = MagicMock() - mock_es.search.return_value = {'hits': {'hits': []}} - mock_get_es.return_value = mock_es - - request = rf.get( - '/SNMP/GetEdgeInterfaceDetail/' - '?source=unknown-device&source_iface=Gi0/1' - ) - response = get_edge_interface_detail(request) - assert response.status_code == 200 - data = json.loads(response.content) - assert data['success'] is True - assert data['source']['interface'] is None - - @patch('SNMP.network_map.get_elastic_connection') - def test_es_exception_on_lookup_still_returns_200(self, mock_get_es, rf, test_network): - mock_es = MagicMock() - mock_es.search.side_effect = Exception('ES lookup failed') - mock_get_es.return_value = mock_es - - request = rf.get( - '/SNMP/GetEdgeInterfaceDetail/' - '?source=switch-a&source_iface=Gi0/1' - ) - response = get_edge_interface_detail(request) - assert response.status_code == 200 - data = json.loads(response.content) - assert data['success'] is True - assert data['source']['interface'] is None diff --git a/src/logstashui/SNMP/tests/test_overview.py b/src/logstashui/SNMP/tests/test_overview.py deleted file mode 100644 index b4384e5..0000000 --- a/src/logstashui/SNMP/tests/test_overview.py +++ /dev/null @@ -1,482 +0,0 @@ -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. - -""" -Tests for SNMP.overview — Elasticsearch query functions for the Overview page. -All Elasticsearch I/O is mocked; only the Django DB layer is real. -""" - -import pytest -from unittest.mock import patch, MagicMock - -from SNMP.overview import ( - get_discovered_devices_count, - get_high_resource_usage, - get_template_data_categories, -) -from SNMP.models import Network, Device, Credential -from PipelineManager.models import Connection - - -# =========================================================================== -# Fixtures -# =========================================================================== - -@pytest.fixture -def test_connection(db): - return Connection.objects.create( - name='Overview Test Connection', - connection_type='CENTRALIZED', - host='https://localhost:9200', - username='elastic', - password='changeme' - ) - - -@pytest.fixture -def test_credential(db): - return Credential.objects.create( - name='overview_test_cred', - version='2c', - community='public' - ) - - -@pytest.fixture -def test_network(db, test_connection, test_credential): - return Network.objects.create( - name='Overview Test Network', - network_range='10.0.0.0/24', - connection=test_connection, - discovery_credential=test_credential, - interval=30 - ) - - -@pytest.fixture -def test_device(db, test_network, test_credential): - return Device.objects.create( - name='overview_test_device', - ip_address='10.0.0.1', - port=161, - retries=1, - timeout=500, - credential=test_credential, - network=test_network, - ) - - -def _make_es_client(cardinality_value=5): - """Return a mock ES client with a canned discovered-devices response.""" - mock_es = MagicMock() - mock_es.search.return_value = { - 'aggregations': { - 'unique_hosts': { - 'value': cardinality_value - } - } - } - return mock_es - - -# =========================================================================== -# get_discovered_devices_count -# =========================================================================== - -@pytest.mark.django_db -class TestGetDiscoveredDevicesCount: - - def test_no_networks_returns_success_false(self): - result = get_discovered_devices_count() - assert result['success'] is False - assert result['count'] == 0 - assert 'No Elasticsearch connections' in result['error'] - - def test_network_without_connection_not_queried(self, db, test_credential): - Network.objects.create( - name='Unconnected Network', - network_range='192.168.0.0/24', - discovery_credential=test_credential, - interval=30, - connection=None - ) - result = get_discovered_devices_count() - assert result['success'] is False - - @patch('SNMP.overview.get_elastic_connection') - def test_returns_count_from_es_aggregation(self, mock_get_es, test_network): - mock_get_es.return_value = _make_es_client(cardinality_value=7) - - result = get_discovered_devices_count() - assert result['success'] is True - assert result['count'] == 7 - - @patch('SNMP.overview.get_elastic_connection') - def test_merges_counts_across_multiple_connections(self, mock_get_es, db, test_credential): - conn1 = Connection.objects.create( - name='OV Conn 1', connection_type='CENTRALIZED', - host='https://es1:9200', username='e', password='p' - ) - conn2 = Connection.objects.create( - name='OV Conn 2', connection_type='CENTRALIZED', - host='https://es2:9200', username='e', password='p' - ) - Network.objects.create( - name='Net 1', network_range='10.1.0.0/24', - connection=conn1, discovery_credential=test_credential, interval=30 - ) - Network.objects.create( - name='Net 2', network_range='10.2.0.0/24', - connection=conn2, discovery_credential=test_credential, interval=30 - ) - - mock_es1 = _make_es_client(cardinality_value=3) - mock_es2 = _make_es_client(cardinality_value=4) - mock_get_es.side_effect = [mock_es1, mock_es2] - - result = get_discovered_devices_count() - assert result['success'] is True - assert result['count'] == 7 - - @patch('SNMP.overview.get_elastic_connection') - def test_es_error_tracked_in_errors_list(self, mock_get_es, test_network): - mock_get_es.side_effect = Exception('Connection refused') - - result = get_discovered_devices_count() - assert result['success'] is True # overall success even with per-connection error - assert result['count'] == 0 - assert result['errors'] is not None - assert len(result['errors']) == 1 - - @patch('SNMP.overview.get_elastic_connection') - def test_no_errors_returns_none_for_errors_key(self, mock_get_es, test_network): - mock_get_es.return_value = _make_es_client(cardinality_value=2) - - result = get_discovered_devices_count() - assert result['errors'] is None - - @patch('SNMP.overview.get_elastic_connection') - def test_response_without_aggregations_treated_as_zero(self, mock_get_es, test_network): - mock_es = MagicMock() - mock_es.search.return_value = {} # no 'aggregations' key - mock_get_es.return_value = mock_es - - result = get_discovered_devices_count() - assert result['success'] is True - assert result['count'] == 0 - - -# =========================================================================== -# get_high_resource_usage -# =========================================================================== - -@pytest.mark.django_db -class TestGetHighResourceUsage: - - def test_no_devices_returns_empty_lists(self): - result = get_high_resource_usage() - assert result['success'] is True - assert result['high_cpu'] == [] - assert result['high_memory'] == [] - - def test_device_without_network_connection_skipped(self, db, test_credential, test_network): - # Create a network with no ES connection - net_no_conn = Network.objects.create( - name='No Conn Net', - network_range='172.16.0.0/24', - discovery_credential=test_credential, - interval=30, - connection=None - ) - Device.objects.create( - name='disconnected_device', - ip_address='172.16.0.1', - port=161, - retries=1, - timeout=500, - credential=test_credential, - network=net_no_conn, - ) - - result = get_high_resource_usage() - assert result['success'] is True - assert result['high_cpu'] == [] - assert result['high_memory'] == [] - - @patch('SNMP.overview.get_elastic_connection') - def test_device_with_high_cpu_appears_in_high_cpu_list(self, mock_get_es, test_device): - mock_es = MagicMock() - mock_es.search.return_value = { - 'aggregations': { - 'devices': { - 'buckets': [ - { - 'key': '10.0.0.1', - 'latest_cpu': { - 'hits': { - 'hits': [ - {'_source': {'system': {'cpu': {'total': {'norm': {'pct': 0.95}}}}}} - ] - } - }, - 'latest_memory': {'hits': {'hits': []}} - } - ] - } - } - } - mock_get_es.return_value = mock_es - - result = get_high_resource_usage() - assert result['success'] is True - assert len(result['high_cpu']) == 1 - assert result['high_cpu'][0]['cpu_pct'] == 95.0 - assert result['high_cpu'][0]['ip_address'] == '10.0.0.1' - - @patch('SNMP.overview.get_elastic_connection') - def test_device_with_high_memory_appears_in_high_memory_list(self, mock_get_es, test_device): - mock_es = MagicMock() - mock_es.search.return_value = { - 'aggregations': { - 'devices': { - 'buckets': [ - { - 'key': '10.0.0.1', - 'latest_cpu': {'hits': {'hits': []}}, - 'latest_memory': { - 'hits': { - 'hits': [ - {'_source': {'system': {'memory': {'actual': {'used': {'pct': 0.87}}}}}} - ] - } - } - } - ] - } - } - } - mock_get_es.return_value = mock_es - - result = get_high_resource_usage() - assert result['success'] is True - assert len(result['high_memory']) == 1 - assert result['high_memory'][0]['memory_pct'] == 87.0 - - @patch('SNMP.overview.get_elastic_connection') - def test_device_below_threshold_not_included(self, mock_get_es, test_device): - mock_es = MagicMock() - mock_es.search.return_value = { - 'aggregations': { - 'devices': { - 'buckets': [ - { - 'key': '10.0.0.1', - 'latest_cpu': { - 'hits': { - 'hits': [ - {'_source': {'system': {'cpu': {'total': {'norm': {'pct': 0.5}}}}}} - ] - } - }, - 'latest_memory': {'hits': {'hits': []}} - } - ] - } - } - } - mock_get_es.return_value = mock_es - - result = get_high_resource_usage() - assert result['high_cpu'] == [] - - @patch('SNMP.overview.get_elastic_connection') - def test_high_cpu_sorted_highest_first(self, mock_get_es, db, test_network, test_credential): - Device.objects.create( - name='device_b', ip_address='10.0.0.2', port=161, - retries=1, timeout=500, credential=test_credential, network=test_network - ) - - mock_es = MagicMock() - mock_es.search.return_value = { - 'aggregations': { - 'devices': { - 'buckets': [ - { - 'key': '10.0.0.1', - 'latest_cpu': { - 'hits': { - 'hits': [{'_source': {'system': {'cpu': {'total': {'norm': {'pct': 0.85}}}}}}] - } - }, - 'latest_memory': {'hits': {'hits': []}} - }, - { - 'key': '10.0.0.2', - 'latest_cpu': { - 'hits': { - 'hits': [{'_source': {'system': {'cpu': {'total': {'norm': {'pct': 0.95}}}}}}] - } - }, - 'latest_memory': {'hits': {'hits': []}} - } - ] - } - } - } - mock_get_es.return_value = mock_es - - result = get_high_resource_usage() - assert result['high_cpu'][0]['cpu_pct'] == 95.0 - assert result['high_cpu'][1]['cpu_pct'] == 85.0 - - @patch('SNMP.overview.get_elastic_connection') - def test_es_error_tracked_in_errors_list(self, mock_get_es, test_device): - mock_get_es.side_effect = Exception('Connection refused') - - result = get_high_resource_usage() - assert result['success'] is True - assert result['errors'] is not None - - @patch('SNMP.overview.get_elastic_connection') - def test_no_errors_returns_none_for_errors_key(self, mock_get_es, test_device): - mock_es = MagicMock() - mock_es.search.return_value = {'aggregations': {'devices': {'buckets': []}}} - mock_get_es.return_value = mock_es - - result = get_high_resource_usage() - assert result['errors'] is None - - -# =========================================================================== -# get_template_data_categories -# =========================================================================== - -@pytest.mark.django_db -class TestGetTemplateDataCategories: - - def test_no_networks_returns_empty_templates(self): - result = get_template_data_categories() - assert result['success'] is True - assert result['templates'] == [] - - @patch('SNMP.overview.get_elastic_connection') - def test_returns_templates_with_categories(self, mock_get_es, test_network): - mock_es = MagicMock() - mock_es.search.return_value = { - 'aggregations': { - 'templates': { - 'buckets': [ - { - 'key': 'dell_idrac', - 'categories': { - 'buckets': [ - {'key': 'system'}, - {'key': 'interface'}, - ] - } - } - ] - } - } - } - mock_get_es.return_value = mock_es - - result = get_template_data_categories() - assert result['success'] is True - assert len(result['templates']) == 1 - assert result['templates'][0]['template_name'] == 'dell_idrac' - assert 'system' in result['templates'][0]['categories'] - assert 'interface' in result['templates'][0]['categories'] - - @patch('SNMP.overview.get_elastic_connection') - def test_categories_sorted_alphabetically(self, mock_get_es, test_network): - mock_es = MagicMock() - mock_es.search.return_value = { - 'aggregations': { - 'templates': { - 'buckets': [ - { - 'key': 'generic', - 'categories': { - 'buckets': [ - {'key': 'system'}, - {'key': 'interface'}, - {'key': 'entity_sensor'}, - ] - } - } - ] - } - } - } - mock_get_es.return_value = mock_es - - result = get_template_data_categories() - cats = result['templates'][0]['categories'] - assert cats == sorted(cats) - - @patch('SNMP.overview.get_elastic_connection') - def test_merges_categories_across_connections(self, mock_get_es, db, test_credential): - conn1 = Connection.objects.create( - name='Cat Conn 1', connection_type='CENTRALIZED', - host='https://es1:9200', username='e', password='p' - ) - conn2 = Connection.objects.create( - name='Cat Conn 2', connection_type='CENTRALIZED', - host='https://es2:9200', username='e', password='p' - ) - Network.objects.create( - name='CatNet1', network_range='10.10.0.0/24', - connection=conn1, discovery_credential=test_credential, interval=30 - ) - Network.objects.create( - name='CatNet2', network_range='10.11.0.0/24', - connection=conn2, discovery_credential=test_credential, interval=30 - ) - - def side_effect(conn_id): - mock_es = MagicMock() - if conn_id == conn1.id: - mock_es.search.return_value = { - 'aggregations': { - 'templates': { - 'buckets': [{'key': 'cisco', 'categories': {'buckets': [{'key': 'system'}]}}] - } - } - } - else: - mock_es.search.return_value = { - 'aggregations': { - 'templates': { - 'buckets': [{'key': 'cisco', 'categories': {'buckets': [{'key': 'interface'}]}}] - } - } - } - return mock_es - - mock_get_es.side_effect = side_effect - - result = get_template_data_categories() - assert result['success'] is True - cisco_template = next(t for t in result['templates'] if t['template_name'] == 'cisco') - assert 'system' in cisco_template['categories'] - assert 'interface' in cisco_template['categories'] - - @patch('SNMP.overview.get_elastic_connection') - def test_es_error_tracked_continues(self, mock_get_es, test_network): - mock_get_es.side_effect = Exception('ES down') - - result = get_template_data_categories() - assert result['success'] is True - assert result['errors'] is not None - - @patch('SNMP.overview.get_elastic_connection') - def test_response_without_aggregations_returns_empty(self, mock_get_es, test_network): - mock_es = MagicMock() - mock_es.search.return_value = {} - mock_get_es.return_value = mock_es - - result = get_template_data_categories() - assert result['success'] is True - assert result['templates'] == [] diff --git a/src/logstashui/SNMP/tests/test_snmp_crud.py b/src/logstashui/SNMP/tests/test_snmp_crud.py deleted file mode 100644 index a1e4e09..0000000 --- a/src/logstashui/SNMP/tests/test_snmp_crud.py +++ /dev/null @@ -1,2485 +0,0 @@ -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. - -import pytest -from django.contrib.auth.models import User -from django.test import Client -from unittest.mock import patch, MagicMock, Mock -import json - -from SNMP.models import Network, Device, Credential, Profile, DeviceTemplate -from PipelineManager.models import Connection -from Management.models import UserProfile - - -@pytest.fixture -def admin_user(db): - """Create a uer with admin profile""" - user = User.objects.create_user( - username='admin_user', - password='testpass123', - email='admin@example.com' - ) - profile, created = UserProfile.objects.get_or_create(user=user, defaults={'role': 'admin'}) - if not created: - profile.role = 'admin' - profile.save() - return user - - -@pytest.fixture -def readonly_user(db): - """Create a user with readonly profile""" - user = User.objects.create_user( - username='readonly_user', - password='testpass123', - email='readonly@example.com' - ) - profile = UserProfile.objects.get(user=user) - profile.role = 'readonly' - profile.save() - user.refresh_from_db() - return user - - -@pytest.fixture -def authenticated_client(admin_user): - """Create an authenticated client with admin user""" - client = Client() - client.force_login(admin_user) - return client - - -@pytest.fixture -def readonly_client(readonly_user): - """Create an authenticated client with readonly user""" - client = Client() - client.force_login(readonly_user) - return client - - -@pytest.fixture -def test_connection(db): - """Create a test Elasticsearch connection""" - return Connection.objects.create( - name='Test Connection', - connection_type='CENTRALIZED', - host='https://localhost:9200', - username='elastic', - password='changeme' - ) - - -@pytest.fixture -def test_credential_v2c(db): - """Create a test SNMP v2c credential""" - return Credential.objects.create( - name='Test Credential v2c', - version='2c', - community='public', - description='Test SNMP v2c credential' - ) - - -@pytest.fixture -def test_credential_v3(db): - """Create a test SNMP v3 credential""" - return Credential.objects.create( - name='Test Credential v3', - version='3', - security_name='snmpuser', - security_level='authPriv', - auth_protocol='sha', - auth_pass='authpassword', - priv_protocol='aes', - priv_pass='privpassword', - description='Test SNMP v3 credential' - ) - - -@pytest.fixture -def test_network(db, test_connection, test_credential_v2c): - """Create a test SNMP network""" - return Network.objects.create( - name='Test Network', - network_range='192.168.1.0/24', - connection=test_connection, - discovery_credential=test_credential_v2c, - discovery_enabled=True, - traps_enabled=False, - interval=30 - ) - - -@pytest.fixture -def test_device(db, test_network, test_credential_v2c): - """Create a test SNMP device""" - return Device.objects.create( - name='Test Device', - ip_address='192.168.1.100', - port=161, - retries=2, - timeout=1000, - credential=test_credential_v2c, - network=test_network - ) - - -@pytest.fixture -def test_profile(db): - """Create a test user profile""" - return Profile.objects.create( - name='custom_profile', - description='Custom test profile', - vendor='Generic', - profile_data={ - 'get': { - 'test.metric': '1.3.6.1.2.1.1.1.0' - }, - 'walk': {}, - 'table': {} - } - ) - - -# ============================================================================ -# Credential CRUD Tests -# ============================================================================ - -@pytest.mark.django_db -class TestCredentialCRUD: - """Test Credential Create, Read, Update, Delete operations""" - - def test_get_credentials(self, authenticated_client, test_credential_v2c): - """Test getting all credentials""" - response = authenticated_client.get('/SNMP/GetCredentials/') - assert response.status_code == 200 - data = json.loads(response.content) - assert isinstance(data, list) - assert len(data) >= 1 - assert any(c['name'] == 'Test Credential v2c' for c in data) - - def test_get_credential_by_id(self, authenticated_client, test_credential_v2c): - """Test getting a single credential by ID""" - response = authenticated_client.get(f'/SNMP/GetCredential/{test_credential_v2c.id}/') - assert response.status_code == 200 - data = json.loads(response.content) - assert data['name'] == 'Test Credential v2c' - assert data['version'] == '2c' - # Community should be masked - assert data['community'] == '***' - - def test_get_credential_not_found(self, authenticated_client): - """Test getting a non-existent credential""" - response = authenticated_client.get('/SNMP/GetCredential/99999/') - assert response.status_code == 404 - - def test_add_credential_v2c_requires_admin(self, readonly_client): - """Test that adding a credential requires admin role""" - response = readonly_client.post('/SNMP/AddCredential/', { - 'name': 'New Credential', - 'version': '2c', - 'community': 'public' - }) - assert response.status_code == 403 - assert b'Admin role required' in response.content - - def test_add_credential_v2c_success(self, authenticated_client): - """Test successfully adding a v2c credential""" - response = authenticated_client.post('/SNMP/AddCredential/', { - 'name': 'New v2c Credential', - 'version': '2c', - 'community': 'private', - 'description': 'Test description' - }) - assert response.status_code == 200 - data = json.loads(response.content) - assert 'id' in data - assert 'Credential created successfully!' in data['message'] - - # Verify credential was created - credential = Credential.objects.get(name='New v2c Credential') - assert credential.version == '2c' - assert credential.get_community() == 'private' - - def test_add_credential_v3_success(self, authenticated_client): - """Test successfully adding a v3 credential""" - response = authenticated_client.post('/SNMP/AddCredential/', { - 'name': 'New v3 Credential', - 'version': '3', - 'security_name': 'testuser', - 'security_level': 'authPriv', - 'auth_protocol': 'sha', - 'auth_pass': 'authpass123', - 'priv_protocol': 'aes', - 'priv_pass': 'privpass123' - }) - assert response.status_code == 200 - data = json.loads(response.content) - assert 'id' in data - - # Verify credential was created - credential = Credential.objects.get(name='New v3 Credential') - assert credential.version == '3' - assert credential.security_name == 'testuser' - assert credential.get_auth_pass() == 'authpass123' - assert credential.get_priv_pass() == 'privpass123' - - def test_add_credential_validation_error(self, authenticated_client): - """Test adding a credential with validation errors""" - response = authenticated_client.post('/SNMP/AddCredential/', { - 'name': 'Invalid Credential', - 'version': '2c', - 'community': '' # Empty community should fail - }) - assert response.status_code == 400 - assert b'Community string is required' in response.content - - def test_update_credential_requires_admin(self, readonly_client, test_credential_v2c): - """Test that updating a credential requires admin role""" - response = readonly_client.post(f'/SNMP/UpdateCredential/{test_credential_v2c.id}/', { - 'name': 'Updated Name', - 'version': '2c', - 'community': 'newcommunity' - }) - assert response.status_code == 403 - - def test_update_credential_success(self, authenticated_client, test_credential_v2c): - """Test successfully updating a credential""" - response = authenticated_client.post(f'/SNMP/UpdateCredential/{test_credential_v2c.id}/', { - 'name': 'Updated Credential', - 'version': '2c', - 'community': 'newcommunity', - 'description': 'Updated description' - }) - assert response.status_code == 200 - - # Verify credential was updated - test_credential_v2c.refresh_from_db() - assert test_credential_v2c.name == 'Updated Credential' - assert test_credential_v2c.get_community() == 'newcommunity' - - def test_update_credential_not_found(self, authenticated_client): - """Test updating a non-existent credential""" - response = authenticated_client.post('/SNMP/UpdateCredential/99999/', { - 'name': 'Test', - 'version': '2c', - 'community': 'public' - }) - assert response.status_code == 404 - - def test_delete_credential_requires_admin(self, readonly_client, test_credential_v2c): - """Test that deleting a credential requires admin role""" - response = readonly_client.post(f'/SNMP/DeleteCredential/{test_credential_v2c.id}/') - assert response.status_code == 403 - - def test_delete_credential_success(self, authenticated_client, test_credential_v2c): - """Test successfully deleting a credential""" - credential_id = test_credential_v2c.id - response = authenticated_client.post(f'/SNMP/DeleteCredential/{credential_id}/') - assert response.status_code == 200 - assert b'Credential deleted successfully!' in response.content - - # Verify credential was deleted - assert not Credential.objects.filter(id=credential_id).exists() - - def test_delete_credential_not_found(self, authenticated_client): - """Test deleting a non-existent credential""" - response = authenticated_client.post('/SNMP/DeleteCredential/99999/') - assert response.status_code == 404 - - -# ============================================================================ -# Network CRUD Tests -# ============================================================================ - -@pytest.mark.django_db -class TestNetworkCRUD: - """Test Network Create, Read, Update, Delete operations""" - - def test_get_networks(self, authenticated_client, test_network): - """Test getting all networks""" - response = authenticated_client.get('/SNMP/GetNetworks/') - assert response.status_code == 200 - data = json.loads(response.content) - assert isinstance(data, list) - assert len(data) >= 1 - assert any(n['name'] == 'Test Network' for n in data) - - def test_get_network_by_id(self, authenticated_client, test_network): - """Test getting a single network by ID""" - response = authenticated_client.get(f'/SNMP/GetNetwork/{test_network.id}/') - assert response.status_code == 200 - data = json.loads(response.content) - assert data['name'] == 'Test Network' - assert data['network_range'] == '192.168.1.0/24' - - def test_add_network_requires_admin(self, readonly_client, test_connection, test_credential_v2c): - """Test that adding a network requires admin role""" - response = readonly_client.post('/SNMP/AddNetwork/', { - 'name': 'New Network', - 'network_range': '10.0.0.0/24', - 'connection': test_connection.id, - 'discovery_credential': test_credential_v2c.id - }) - assert response.status_code == 403 - - def test_add_network_success(self, authenticated_client, test_connection, test_credential_v2c): - """Test successfully adding a network""" - response = authenticated_client.post('/SNMP/AddNetwork/', { - 'name': 'New Network', - 'network_range': '10.0.0.0/24', - 'connection': test_connection.id, - 'discovery_credential': test_credential_v2c.id, - 'discovery_enabled': 'true', - 'traps_enabled': 'false', - 'interval': '60' - }) - assert response.status_code == 200 - data = json.loads(response.content) - assert 'id' in data - assert 'Network created successfully!' in data['message'] - - # Verify network was created - network = Network.objects.get(name='New Network') - assert network.network_range == '10.0.0.0/24' - assert network.interval == 60 - - def test_add_network_invalid_cidr(self, authenticated_client, test_connection): - """Test adding a network with invalid CIDR notation""" - response = authenticated_client.post('/SNMP/AddNetwork/', { - 'name': 'Invalid Network', - 'network_range': 'not-a-valid-cidr', - }) - assert response.status_code == 400 - assert b'Invalid CIDR notation' in response.content - - def test_update_network_requires_admin(self, readonly_client, test_network): - """Test that updating a network requires admin role""" - response = readonly_client.post(f'/SNMP/UpdateNetwork/{test_network.id}/', { - 'name': 'Updated Network', - 'network_range': '192.168.1.0/24', - }) - assert response.status_code == 403 - - def test_update_network_success(self, authenticated_client, test_network): - """Test successfully updating a network""" - response = authenticated_client.post(f'/SNMP/UpdateNetwork/{test_network.id}/', { - 'name': 'Updated Network', - 'network_range': '192.168.2.0/24', - 'interval': '120' - }) - assert response.status_code == 200 - - # Verify network was updated - test_network.refresh_from_db() - assert test_network.name == 'Updated Network' - assert test_network.network_range == '192.168.2.0/24' - assert test_network.interval == 120 - - def test_delete_network_requires_admin(self, readonly_client, test_network): - """Test that deleting a network requires admin role""" - response = readonly_client.post(f'/SNMP/DeleteNetwork/{test_network.id}/') - assert response.status_code == 403 - - @patch('SNMP.snmp_crud.get_elastic_connection') - def test_delete_network_success(self, mock_es_conn, authenticated_client, test_network): - """Test successfully deleting a network""" - # Mock Elasticsearch connection - mock_es = MagicMock() - mock_es.logstash.get_pipeline.return_value = {} - mock_es_conn.return_value = mock_es - - network_id = test_network.id - response = authenticated_client.post(f'/SNMP/DeleteNetwork/{network_id}/') - assert response.status_code == 200 - data = json.loads(response.content) - assert data['success'] is True - - # Verify network was deleted - assert not Network.objects.filter(id=network_id).exists() - - def test_get_network_pipeline_name(self, authenticated_client, test_network): - """Test getting the pipeline name for a network""" - response = authenticated_client.get(f'/SNMP/GetNetworkPipelineName/{test_network.id}/') - assert response.status_code == 200 - data = json.loads(response.content) - assert data['success'] is True - assert 'pipeline_name' in data - assert 'snmp-' in data['pipeline_name'] - - -# ============================================================================ -# Device CRUD Tests -# ============================================================================ - -@pytest.mark.django_db -class TestDeviceCRUD: - """Test Device Create, Read, Update, Delete operations""" - - def test_get_devices_paginated(self, authenticated_client, test_device): - """Test getting paginated devices""" - response = authenticated_client.get('/SNMP/GetDevices/?page=1&page_size=25') - assert response.status_code == 200 - data = json.loads(response.content) - assert 'devices' in data - assert 'total' in data - assert 'page' in data - assert len(data['devices']) >= 1 - - def test_get_devices_with_search(self, authenticated_client, test_device): - """Test getting devices with search filter""" - response = authenticated_client.get('/SNMP/GetDevices/?search=Test') - assert response.status_code == 200 - data = json.loads(response.content) - assert len(data['devices']) >= 1 - assert any(d['name'] == 'Test Device' for d in data['devices']) - - def test_get_devices_with_network_filter(self, authenticated_client, test_device, test_network): - """Test getting devices filtered by network""" - response = authenticated_client.get(f'/SNMP/GetDevices/?network={test_network.id}') - assert response.status_code == 200 - data = json.loads(response.content) - assert all(d['network_id'] == test_network.id for d in data['devices']) - - def test_get_device_by_id(self, authenticated_client, test_device): - """Test getting a single device by ID""" - response = authenticated_client.get(f'/SNMP/GetDevice/{test_device.id}/') - assert response.status_code == 200 - data = json.loads(response.content) - assert data['name'] == 'Test Device' - assert data['ip_address'] == '192.168.1.100' - assert 'device_template' in data - - def test_add_device_requires_admin(self, readonly_client, test_network, test_credential_v2c): - """Test that adding a device requires admin role""" - response = readonly_client.post('/SNMP/AddDevice/', { - 'name': 'New Device', - 'ip_address': '192.168.1.101', - 'network': test_network.id, - 'credential': test_credential_v2c.id - }) - assert response.status_code == 403 - - def test_add_device_success(self, authenticated_client, test_network, test_credential_v2c): - """Test successfully adding a device""" - response = authenticated_client.post('/SNMP/AddDevice/', { - 'name': 'New Device', - 'ip_address': '192.168.1.101', - 'port': '161', - 'retries': '3', - 'timeout': '2000', - 'network': test_network.id, - 'credential': test_credential_v2c.id, - 'profiles': ['system'] - }) - assert response.status_code == 200 - data = json.loads(response.content) - assert 'id' in data - assert 'Device created successfully!' in data['message'] - - # Verify device was created - device = Device.objects.get(name='New Device') - assert device.ip_address == '192.168.1.101' - - def test_add_device_auto_adds_system_profile(self, authenticated_client, test_network, test_credential_v2c): - """Test that system profile is automatically added to devices""" - response = authenticated_client.post('/SNMP/AddDevice/', { - 'name': 'Device Without Profiles', - 'ip_address': '192.168.1.102', - 'network': test_network.id, - 'credential': test_credential_v2c.id - }) - assert response.status_code == 200 - - # Verify device was created successfully - device = Device.objects.get(name='Device Without Profiles') - assert device.ip_address == '192.168.1.102' - - def test_add_device_invalid_ip(self, authenticated_client, test_network, test_credential_v2c): - """Test adding a device with invalid IP address""" - response = authenticated_client.post('/SNMP/AddDevice/', { - 'name': 'Invalid Device', - 'ip_address': 'not-an-ip!@#', - 'network': test_network.id, - 'credential': test_credential_v2c.id - }) - assert response.status_code == 400 - - def test_update_device_requires_admin(self, readonly_client, test_device): - """Test that updating a device requires admin role""" - response = readonly_client.post(f'/SNMP/UpdateDevice/{test_device.id}/', { - 'name': 'Updated Device', - 'ip_address': '192.168.1.100' - }) - assert response.status_code == 403 - - def test_update_device_success(self, authenticated_client, test_device): - """Test successfully updating a device""" - response = authenticated_client.post(f'/SNMP/UpdateDevice/{test_device.id}/', { - 'name': 'Updated Device', - 'ip_address': '192.168.1.200', - 'port': '162', - 'profiles': ['system'] - }) - assert response.status_code == 200 - - # Verify device was updated - test_device.refresh_from_db() - assert test_device.name == 'Updated Device' - assert test_device.ip_address == '192.168.1.200' - assert test_device.port == 162 - - def test_delete_device_requires_admin(self, readonly_client, test_device): - """Test that deleting a device requires admin role""" - response = readonly_client.post(f'/SNMP/DeleteDevice/{test_device.id}/') - assert response.status_code == 403 - - def test_delete_device_success(self, authenticated_client, test_device): - """Test successfully deleting a device""" - device_id = test_device.id - response = authenticated_client.post(f'/SNMP/DeleteDevice/{device_id}/') - assert response.status_code == 200 - assert b'Device deleted successfully!' in response.content - - # Verify device was deleted - assert not Device.objects.filter(id=device_id).exists() - - -# ============================================================================ -# Profile CRUD Tests -# ============================================================================ - -@pytest.mark.django_db -class TestProfileCRUD: - """Test Profile Create, Read, Update, Delete operations""" - - def test_get_all_profiles(self, authenticated_client, test_profile): - """Test getting all profiles""" - response = authenticated_client.get('/SNMP/GetAllProfiles/') - assert response.status_code == 200 - data = json.loads(response.content) - assert 'profiles' in data - # The custom profile created by the fixture should always appear - assert any(p['name'] == 'custom_profile' for p in data['profiles']) - - def test_get_official_profile(self, authenticated_client): - """Test getting an official profile (mocks filesystem)""" - fake_data = {'description': 'Generic system profile', 'vendor': 'Generic', 'get': {}} - with patch('SNMP.snmp_crud.os.path.exists', return_value=True), \ - patch('builtins.open', create=True) as mock_open, \ - patch('SNMP.snmp_crud.json.load', return_value=fake_data): - mock_open.return_value.__enter__ = lambda s: s - mock_open.return_value.__exit__ = MagicMock(return_value=False) - response = authenticated_client.get('/SNMP/GetOfficialProfile/system/') - assert response.status_code == 200 - data = json.loads(response.content) - assert data['success'] is True - assert 'profile_data' in data - - def test_get_user_profile(self, authenticated_client, test_profile): - """Test getting a user profile""" - response = authenticated_client.get('/SNMP/GetProfile/custom_profile/') - assert response.status_code == 200 - data = json.loads(response.content) - assert data['success'] is True - assert data['name'] == 'custom_profile' - - def test_add_profile_requires_admin(self, readonly_client): - """Test that adding a profile requires admin role""" - response = readonly_client.post('/SNMP/AddProfile/', - json.dumps({ - 'name': 'new_profile', - 'description': 'Test', - 'profile_data': {'get': {}} - }), - content_type='application/json' - ) - assert response.status_code == 403 - - def test_add_profile_success(self, authenticated_client): - """Test successfully adding a profile""" - response = authenticated_client.post('/SNMP/AddProfile/', - json.dumps({ - 'name': 'new_custom_profile', - 'description': 'New custom profile', - 'type': 'Network', - 'vendor': 'Cisco', - 'profile_data': { - 'get': { - 'custom.metric': '1.3.6.1.4.1.1.1.0' - }, - 'walk': {}, - 'table': {} - } - }), - content_type='application/json' - ) - assert response.status_code == 200 - data = json.loads(response.content) - assert data['success'] is True - - # Verify profile was created - profile = Profile.objects.get(name='new_custom_profile') - assert profile.vendor == 'Cisco' - - def test_add_profile_duplicate_name(self, authenticated_client, test_profile): - """Test adding a profile with duplicate name""" - response = authenticated_client.post('/SNMP/AddProfile/', - json.dumps({ - 'name': 'custom_profile', - 'profile_data': {'get': {}} - }), - content_type='application/json' - ) - assert response.status_code == 400 - data = json.loads(response.content) - assert 'already exists' in data['message'] - - def test_update_profile_requires_admin(self, readonly_client, test_profile): - """Test that updating a profile requires admin role""" - response = readonly_client.post(f'/SNMP/UpdateProfile/{test_profile.name}/', - json.dumps({ - 'name': 'updated_profile', - 'profile_data': {'get': {}} - }), - content_type='application/json' - ) - assert response.status_code == 403 - - def test_update_profile_success(self, authenticated_client, test_profile): - """Test successfully updating a profile""" - response = authenticated_client.post(f'/SNMP/UpdateProfile/{test_profile.name}/', - json.dumps({ - 'description': 'Updated description', - 'vendor': 'Updated Vendor', - 'profile_data': { - 'get': { - 'updated.metric': '1.3.6.1.2.1.1.2.0' - } - } - }), - content_type='application/json' - ) - assert response.status_code == 200 - - # Verify profile was updated - test_profile.refresh_from_db() - assert test_profile.description == 'Updated description' - assert test_profile.vendor == 'Updated Vendor' - - def test_delete_profile_requires_admin(self, readonly_client, test_profile): - """Test that deleting a profile requires admin role""" - response = readonly_client.post(f'/SNMP/DeleteProfile/{test_profile.name}/') - assert response.status_code == 403 - - def test_delete_profile_success(self, authenticated_client, test_profile): - """Test successfully deleting a profile""" - response = authenticated_client.post(f'/SNMP/DeleteProfile/{test_profile.name}/') - assert response.status_code == 200 - data = json.loads(response.content) - assert data['success'] is True - - # Verify profile was deleted - assert not Profile.objects.filter(name='custom_profile').exists() - - def test_delete_system_profile_forbidden(self, authenticated_client): - """Test that system profile cannot be deleted""" - response = authenticated_client.post('/SNMP/DeleteProfile/system/') - assert response.status_code == 403 - data = json.loads(response.content) - assert 'cannot be deleted' in data['message'] - - -# ============================================================================ -# Deploy Configuration Tests -# ============================================================================ - -@pytest.mark.django_db -class TestDeployConfiguration: - """Test configuration deployment operations""" - - def test_get_deploy_diff(self, authenticated_client, test_network, test_device): - """Test getting deploy diff""" - with patch('SNMP.snmp_crud.get_elastic_connection') as mock_es_conn: - mock_es = MagicMock() - mock_es.logstash.get_pipeline.return_value = {} - mock_es_conn.return_value = mock_es - - response = authenticated_client.get('/SNMP/GetDeployDiff/') - assert response.status_code == 200 - data = json.loads(response.content) - assert data['success'] is True - assert 'networks' in data - assert any(c['id'] == test_network.connection.id for c in data['connections']) - - def test_get_deploy_diff_includes_es_connection_for_agent_only_networks( - self, authenticated_client, test_network, test_device, test_connection - ): - """Agent-only setups still need the SNMP index template on their ES connection.""" - from PipelineManager.models import Policy, Connection as AgentConnection - - policy = Policy.objects.create( - name='Simulated SNMP Policy', - settings_path='/etc/logstash/', - logs_path='/var/log/logstash', - binary_path='/usr/share/logstash/bin', - logstash_yml='http.host: "0.0.0.0"', - jvm_options='-Xms1g', - log4j2_properties='logger.logstash.name = logstash', - keystore_password='test_password', - ) - agent = AgentConnection.objects.create( - name='SimulatedSNMP Agent', - connection_type='AGENT', - host='agent.example.com', - agent_id='sim-snmp-001', - is_active=True, - policy=policy, - ) - test_network.deployment_mode = 'AGENT' - test_network.agent_connection = agent - test_network.save(update_fields=['deployment_mode', 'agent_connection']) - - with patch('SNMP.snmp_crud.get_elastic_connection') as mock_es_conn: - mock_es = MagicMock() - mock_es.logstash.get_pipeline.return_value = {} - mock_es_conn.return_value = mock_es - - response = authenticated_client.get('/SNMP/GetDeployDiff/') - - assert response.status_code == 200 - data = json.loads(response.content) - assert data['success'] is True - assert data['connections'] == [ - {'id': test_connection.id, 'name': test_connection.name} - ] - - # GetDeployDiff caches a 60s deploy plan; don't leak an Agent-mode plan - # into later DeployConfiguration tests in this class. - from django.core.cache import cache - cache.delete('snmp_deployment_plan') - - def test_deploy_configuration_requires_admin(self, readonly_client): - """Test that deploying configuration requires admin role""" - response = readonly_client.post('/SNMP/DeployConfiguration/') - assert response.status_code == 403 - - @patch('SNMP.snmp_crud.get_elastic_connection') - def test_deploy_configuration_success(self, mock_es_conn, authenticated_client, test_network, test_device): - """Test successfully deploying configuration""" - mock_es = MagicMock() - mock_es.logstash.get_pipeline.return_value = {} - mock_es.logstash.put_pipeline.return_value = {'acknowledged': True} - mock_es_conn.return_value = mock_es - - response = authenticated_client.post('/SNMP/DeployConfiguration/') - assert response.status_code == 200 - data = json.loads(response.content) - assert data['success'] is True - - @patch('SNMP.snmp_crud.get_elastic_connection') - def test_deploy_configuration_no_networks(self, mock_es_conn, authenticated_client): - """Test deploying with no networks configured""" - response = authenticated_client.post('/SNMP/DeployConfiguration/') - assert response.status_code == 400 - data = json.loads(response.content) - assert 'No networks configured' in data['error'] - - -# ============================================================================ -# Device Status and Visualization Tests -# ============================================================================ - -@pytest.mark.django_db -class TestDeviceStatusAndVisualization: - """Test device status checking and visualization endpoints""" - - @patch('SNMP.snmp_crud.get_elastic_connection') - def test_get_devices_status(self, mock_es_conn, authenticated_client, test_device): - """Test getting device status in batch""" - mock_es = MagicMock() - mock_es.search.return_value = { - 'aggregations': { - 'online_devices': { - 'buckets': [ - {'key': '192.168.1.100', 'doc_count': 10} - ] - } - } - } - mock_es_conn.return_value = mock_es - - response = authenticated_client.get(f'/SNMP/GetDevicesStatus/?device_ids={test_device.id}') - assert response.status_code == 200 - data = json.loads(response.content) - assert data['success'] is True - assert 'statuses' in data - assert data['statuses'][str(test_device.id)]['is_online'] is True - - search_kwargs = mock_es.search.call_args.kwargs - assert search_kwargs['index'] == 'metrics-snmp*' - assert search_kwargs['aggregations']['online_devices']['terms']['field'] == 'host.polled_address' - - @patch('SNMP.snmp_crud.get_elastic_connection') - def test_get_devices_status_hostname_only_device( - self, mock_es_conn, authenticated_client, test_network, test_credential_v2c - ): - """Hostname-only devices are matched on host.polled_address, not IP.""" - device = Device.objects.create( - name='Linux', - hostname='linux_host.lab', - ip_address=None, - port=1161, - credential=test_credential_v2c, - network=test_network, - ) - mock_es = MagicMock() - mock_es.search.return_value = { - 'aggregations': { - 'online_devices': { - 'buckets': [ - {'key': 'linux_host.lab', 'doc_count': 4} - ] - } - } - } - mock_es_conn.return_value = mock_es - - response = authenticated_client.get(f'/SNMP/GetDevicesStatus/?device_ids={device.id}') - assert response.status_code == 200 - data = json.loads(response.content) - assert data['statuses'][str(device.id)]['is_online'] is True - - terms_filter = mock_es.search.call_args.kwargs['query']['bool']['filter'][1] - assert terms_filter == {'terms': {'host.polled_address': ['linux_host.lab']}} - - def test_get_devices_status_invalid_ids(self, authenticated_client): - """Test getting device status with invalid IDs""" - response = authenticated_client.get('/SNMP/GetDevicesStatus/?device_ids=invalid') - assert response.status_code == 400 - - @patch('SNMP.snmp_crud.generate_visualizations') - @patch('SNMP.snmp_crud.get_elastic_connection') - def test_get_device_visualization(self, mock_es_conn, mock_gen_viz, authenticated_client, test_device): - """Test getting device visualization data""" - mock_es = MagicMock() - mock_es.search.return_value = { - 'aggregations': { - 'data_kinds': { - 'buckets': [ - {'key': 'metric', 'doc_count': 100} - ] - } - } - } - mock_es_conn.return_value = mock_es - - # Mock the visualization generation to return simple data - mock_gen_viz.return_value = { - 'charts': [], - 'has_data': True - } - - response = authenticated_client.get(f'/SNMP/GetDeviceVisualization/{test_device.id}/') - assert response.status_code == 200 - data = json.loads(response.content) - assert data['success'] is True - assert 'device' in data - assert 'visualizations' in data - - @patch('SNMP.snmp_crud.get_elastic_connection') - def test_get_discovered_devices(self, mock_es_conn, authenticated_client, test_connection, test_network): - """Test getting discovered devices from Elasticsearch""" - mock_es = MagicMock() - mock_es.search.return_value = { - 'aggregations': { - 'devices_by_host': { - 'buckets': [ - { - 'key': 'device1', - 'latest_doc': { - 'hits': { - 'hits': [ - { - '_source': { - 'host': {'name': 'device1', 'hostname': '192.168.1.50'}, - 'network': {'name': 'Test Network'}, - '@timestamp': '2024-01-01T00:00:00Z' - } - } - ] - } - } - } - ] - } - } - } - mock_es_conn.return_value = mock_es - - response = authenticated_client.get('/SNMP/DiscoveredDevices/') - assert response.status_code == 200 - data = json.loads(response.content) - assert data['success'] is True - assert 'devices' in data - - -# ============================================================================ -# Edge Cases and Error Handling -# ============================================================================ - -@pytest.mark.django_db -class TestEdgeCasesAndErrors: - """Test edge cases and error handling""" - - def test_unauthenticated_access_denied(self, client): - """Test that unauthenticated requests are denied""" - response = client.get('/SNMP/GetCredentials/') - assert response.status_code == 302 - assert '/Management/Login/' in response.url - - def test_credential_encryption(self, authenticated_client): - """Test that credentials are encrypted when saved""" - response = authenticated_client.post('/SNMP/AddCredential/', { - 'name': 'Encryption Test', - 'version': '2c', - 'community': 'secret' - }) - assert response.status_code == 200 - - # Verify community is encrypted in database - credential = Credential.objects.get(name='Encryption Test') - # Encrypted value should start with 'gAAAAA' (Fernet token) - assert credential.community.startswith('gAAAAA') - # But decrypted value should be original - assert credential.get_community() == 'secret' - - def test_network_cidr_validation(self, authenticated_client): - """Test CIDR validation for networks""" - # Valid CIDR within the /20 size limit - response = authenticated_client.post('/SNMP/AddNetwork/', { - 'name': 'Valid CIDR', - 'network_range': '10.0.0.0/24', - }) - assert response.status_code == 200 - - # Networks larger than /20 are rejected (would OOM during discovery) - response = authenticated_client.post('/SNMP/AddNetwork/', { - 'name': 'Too Large CIDR', - 'network_range': '10.0.0.0/8', - }) - assert response.status_code == 400 - data = response.json() - assert not data['success'] - assert 'too large' in data['message'].lower() - - # Invalid CIDR is rejected by model validation - response = authenticated_client.post('/SNMP/AddNetwork/', { - 'name': 'Invalid CIDR', - 'network_range': '999.999.999.999/99', - }) - assert response.status_code == 400 - - def test_device_ip_validation(self, authenticated_client, test_network, test_credential_v2c): - """Test IP address validation for devices""" - # Valid IP address - response = authenticated_client.post('/SNMP/AddDevice/', { - 'name': 'Valid IP Device', - 'ip_address': '192.168.1.1', - 'network': test_network.id, - 'credential': test_credential_v2c.id - }) - assert response.status_code == 200 - - # Hostname in ip_address is no longer valid; ip_address must be a valid IP. - # Hostnames should be set in the 'hostname' field instead. - response = authenticated_client.post('/SNMP/AddDevice/', { - 'name': 'Hostname Device', - 'ip_address': 'router.example.com', - 'network': test_network.id, - 'credential': test_credential_v2c.id - }) - assert response.status_code == 400 - - def test_profile_json_validation(self, authenticated_client): - """Test that profile_data must be valid JSON object""" - # Valid JSON object — vendor is now required - response = authenticated_client.post('/SNMP/AddProfile/', - json.dumps({ - 'name': 'valid_json_profile', - 'vendor': 'Generic', - 'profile_data': {'get': {}, 'walk': {}} - }), - content_type='application/json' - ) - assert response.status_code == 200 - - @patch('SNMP.snmp_crud.get_elastic_connection') - def test_deploy_handles_elasticsearch_errors(self, mock_es_conn, authenticated_client, test_network, test_device): - """Test that deploy handles Elasticsearch errors gracefully""" - mock_es_conn.side_effect = Exception("Connection failed") - - response = authenticated_client.post('/SNMP/DeployConfiguration/') - # Should return error but not crash - assert response.status_code in [400, 500] - - -# ============================================================================ -# Pure-function unit tests for snmp_crud.py helpers -# ============================================================================ - -@pytest.mark.django_db -class TestGetPipelineName: - """Tests for _get_pipeline_name() helper""" - - def test_basic_name_generation(self, test_network): - from SNMP.snmp_crud import _get_pipeline_name - name = _get_pipeline_name(test_network) - assert name.startswith('snmp-') - # network name is 'Test Network' — spaces become underscores via sanitizer - assert 'test_network' in name - - def test_special_chars_sanitized(self, test_connection, test_credential_v2c): - """Special chars in network name are sanitized""" - from SNMP.snmp_crud import _get_pipeline_name - network = Network.objects.create( - name='My Network (prod)!', - network_range='10.0.0.0/24', - connection=test_connection, - ) - name = _get_pipeline_name(network) - # Pipeline names must not contain special chars - import re - assert re.match(r'^[a-z0-9_\-]+$', name), f"Bad pipeline name: {name}" - - -@pytest.mark.django_db -class TestCreateOrUpdatePipeline: - """Tests for _create_or_update_pipeline() helper""" - - def test_creates_new_pipeline(self): - from SNMP.snmp_crud import _create_or_update_pipeline - mock_es = MagicMock() - mock_es.logstash.get_pipeline.side_effect = Exception("not found") - mock_es.logstash.put_pipeline.return_value = {} - - success, is_new, error, was_updated = _create_or_update_pipeline( - mock_es, 'test-pipe', 'input {} filter {} output {}' - ) - assert success is True - assert is_new is True - assert error is None - assert was_updated is True - mock_es.logstash.put_pipeline.assert_called_once() - - def test_updates_existing_pipeline_when_content_changed(self): - from SNMP.snmp_crud import _create_or_update_pipeline - mock_es = MagicMock() - mock_es.logstash.get_pipeline.return_value = { - 'test-pipe': { - 'pipeline': 'input {} filter {} output { old_output }', - 'pipeline_settings': {'queue.type': 'memory'}, - 'pipeline_metadata': {'version': 2, 'type': 'logstash_pipeline'}, - } - } - mock_es.logstash.put_pipeline.return_value = {} - - success, is_new, error, was_updated = _create_or_update_pipeline( - mock_es, 'test-pipe', 'input {} filter {} output { new_output }' - ) - assert success is True - assert is_new is False - assert was_updated is True - mock_es.logstash.put_pipeline.assert_called_once() - - def test_skips_update_when_content_identical(self): - from SNMP.snmp_crud import _create_or_update_pipeline - content = 'input {} filter {} output {}' - mock_es = MagicMock() - mock_es.logstash.get_pipeline.return_value = { - 'test-pipe': { - 'pipeline': content, - 'pipeline_settings': {}, - 'pipeline_metadata': {}, - } - } - - success, is_new, error, was_updated = _create_or_update_pipeline( - mock_es, 'test-pipe', content - ) - assert success is True - assert is_new is False - assert was_updated is False - mock_es.logstash.put_pipeline.assert_not_called() - - def test_returns_false_on_put_exception(self): - from SNMP.snmp_crud import _create_or_update_pipeline - mock_es = MagicMock() - mock_es.logstash.get_pipeline.side_effect = Exception("not found") - mock_es.logstash.put_pipeline.side_effect = Exception("ES write error") - - success, is_new, error, was_updated = _create_or_update_pipeline( - mock_es, 'test-pipe', 'input {} filter {} output {}' - ) - assert success is False - assert error is not None - assert 'ES write error' in error - - def test_new_pipeline_uses_default_settings(self): - from SNMP.snmp_crud import _create_or_update_pipeline - mock_es = MagicMock() - mock_es.logstash.get_pipeline.side_effect = Exception("not found") - mock_es.logstash.put_pipeline.return_value = {} - - _create_or_update_pipeline(mock_es, 'new-pipe', 'input {}') - call_body = mock_es.logstash.put_pipeline.call_args[1]['body'] - assert 'pipeline_settings' in call_body - assert call_body['pipeline_settings']['queue.type'] == 'memory' - - def test_existing_pipeline_preserves_settings(self): - from SNMP.snmp_crud import _create_or_update_pipeline - custom_settings = {'queue.type': 'persisted', 'pipeline.workers': 4} - mock_es = MagicMock() - mock_es.logstash.get_pipeline.return_value = { - 'test-pipe': { - 'pipeline': 'old content', - 'pipeline_settings': custom_settings, - 'pipeline_metadata': {'version': 5}, - } - } - mock_es.logstash.put_pipeline.return_value = {} - - _create_or_update_pipeline(mock_es, 'test-pipe', 'new content') - call_body = mock_es.logstash.put_pipeline.call_args[1]['body'] - assert call_body['pipeline_settings'] == custom_settings - - -@pytest.mark.django_db -class TestGetDeviceProfiles: - """Tests for _get_device_profiles() helper (lives in snmp_pipeline_generator)""" - - def test_no_template_returns_empty(self, test_network, test_credential_v2c): - from SNMP.snmp_pipeline_generator import _get_device_profiles - device = Device.objects.create( - name='No Template Device', ip_address='10.0.0.1', - credential=test_credential_v2c, network=test_network - ) - profile_ids, merged, normalizers = _get_device_profiles(device, {}) - assert profile_ids == tuple() - assert merged == {'get': {}, 'walk': {}, 'table': {}} - assert normalizers == [] - - def test_custom_profile_oids_merged(self, test_network, test_credential_v2c): - from SNMP.snmp_pipeline_generator import _get_device_profiles - profile = Profile.objects.create( - name='custom_test', - vendor='Generic', - profile_data={ - 'get': {'system.name': '1.3.6.1.2.1.1.5.0'}, - 'walk': {}, - 'table': {} - } - ) - template = DeviceTemplate.objects.create(name='Test Template', vendor='Generic') - template.profiles.add(profile) - device = Device.objects.create( - name='Profile Device', ip_address='10.0.0.2', - credential=test_credential_v2c, network=test_network, - device_template=template - ) - - profile_ids, merged, normalizers = _get_device_profiles(device, {}) - assert len(profile_ids) == 1 - assert '1.3.6.1.2.1.1.5.0' in merged['get'].values() - - def test_official_placeholder_loaded_from_file(self, test_network, test_credential_v2c): - from SNMP.snmp_pipeline_generator import _get_device_profiles - profile = Profile.objects.create( - name='test_official.json', - vendor='Generic', - profile_data={'is_official_placeholder': True}, - ) - template = DeviceTemplate.objects.create(name='Official Template', vendor='Generic') - template.profiles.add(profile) - device = Device.objects.create( - name='Official Device', ip_address='10.0.0.3', - credential=test_credential_v2c, network=test_network, - device_template=template - ) - - fake_data = {'get': {'system.desc': '1.3.6.1.2.1.1.1.0'}, 'walk': {}, 'table': {}} - with patch('SNMP.snmp_pipeline_generator.os.path.exists', return_value=True), \ - patch('builtins.open', create=True) as mock_open, \ - patch('SNMP.snmp_pipeline_generator.json.load', return_value=fake_data): - mock_open.return_value.__enter__ = lambda s: s - mock_open.return_value.__exit__ = MagicMock(return_value=False) - profile_ids, merged, normalizers = _get_device_profiles(device, {}) - - assert '1.3.6.1.2.1.1.1.0' in merged['get'].values() - - def test_official_placeholder_file_missing_skipped(self, test_network, test_credential_v2c): - from SNMP.snmp_pipeline_generator import _get_device_profiles - profile = Profile.objects.create( - name='missing_official.json', - vendor='Generic', - profile_data={'is_official_placeholder': True}, - ) - template = DeviceTemplate.objects.create(name='Missing File Template', vendor='Generic') - template.profiles.add(profile) - device = Device.objects.create( - name='Missing File Device', ip_address='10.0.0.4', - credential=test_credential_v2c, network=test_network, - device_template=template - ) - - with patch('SNMP.snmp_pipeline_generator.os.path.exists', return_value=False): - profile_ids, merged, normalizers = _get_device_profiles(device, {}) - - assert merged == {'get': {}, 'walk': {}, 'table': {}} - - def test_oid_conflict_gets_suffixed(self, test_network, test_credential_v2c): - """When two profiles define the same OID key with different values, a suffix is added""" - from SNMP.snmp_pipeline_generator import _get_device_profiles - profile_a = Profile.objects.create( - name='profile_a', vendor='Generic', - profile_data={'get': {'metric': 'oid.1'}, 'walk': {}, 'table': {}} - ) - profile_b = Profile.objects.create( - name='profile_b', vendor='Generic', - profile_data={'get': {'metric': 'oid.2'}, 'walk': {}, 'table': {}} - ) - template = DeviceTemplate.objects.create(name='Conflict Template', vendor='Generic') - template.profiles.add(profile_a, profile_b) - device = Device.objects.create( - name='Conflict Device', ip_address='10.0.0.5', - credential=test_credential_v2c, network=test_network, - device_template=template - ) - - _, merged, _ = _get_device_profiles(device, {}) - assert len(merged['get']) == 2 - - -@pytest.mark.django_db -class TestFormatFieldName: - """Tests for _format_field_name() pure function""" - - def test_already_bracket_notation_unchanged(self): - from SNMP.snmp_pipeline_generator import _format_field_name - assert _format_field_name('[system][cpu]') == '[system][cpu]' - - def test_dotted_name_converted(self): - from SNMP.snmp_pipeline_generator import _format_field_name - assert _format_field_name('system.cpu.load') == '[system][cpu][load]' - - def test_plain_name_wrapped_in_brackets(self): - # In snmp_pipeline_generator, plain names without dots are wrapped in [brackets] - from SNMP.snmp_pipeline_generator import _format_field_name - assert _format_field_name('hostname') == '[hostname]' - - def test_single_dot(self): - from SNMP.snmp_pipeline_generator import _format_field_name - assert _format_field_name('a.b') == '[a][b]' - - -@pytest.mark.django_db -class TestGetDiscoveryIpAddresses: - """Tests for _get_discovery_ip_addresses() helper""" - - def test_returns_all_hosts_in_range(self, test_network): - from SNMP.snmp_pipeline_generator import _get_discovery_ip_addresses - # /30 has 2 usable hosts - test_network.network_range = '192.168.100.0/30' - test_network.save() - ips = _get_discovery_ip_addresses(test_network) - assert '192.168.100.1' in ips - assert '192.168.100.2' in ips - assert '192.168.100.0' not in ips # network address - assert '192.168.100.3' not in ips # broadcast - - def test_excludes_existing_device_ips(self, test_network, test_credential_v2c): - from SNMP.snmp_pipeline_generator import _get_discovery_ip_addresses - test_network.network_range = '10.0.0.0/30' - test_network.save() - Device.objects.create( - name='Existing', ip_address='10.0.0.1', - credential=test_credential_v2c, network=test_network - ) - ips = _get_discovery_ip_addresses(test_network) - assert '10.0.0.1' not in ips - assert '10.0.0.2' in ips - - def test_legacy_hostname_ip_values_not_excluded(self, test_network): - """If legacy data has a non-IP value in ip_address, the function skips it gracefully. - - The Device model now validates that ip_address must be a valid IP, so this - scenario can only occur with legacy data. We test it using a mocked queryset. - """ - from SNMP.snmp_pipeline_generator import _get_discovery_ip_addresses - test_network.network_range = '10.0.1.0/30' - test_network.save() - - with patch('SNMP.snmp_pipeline_generator.Device.objects') as mock_objs: - mock_objs.filter.return_value.values_list.return_value = ['router.example.com'] - ips = _get_discovery_ip_addresses(test_network) - - # Hostname in ip_address should be skipped; both usable IPs remain - assert '10.0.1.1' in ips - assert '10.0.1.2' in ips - - def test_invalid_cidr_returns_empty(self): - """Invalid CIDR can't be saved to DB (model validates it), so use a Mock.""" - from SNMP.snmp_pipeline_generator import _get_discovery_ip_addresses - from unittest.mock import MagicMock - fake_network = MagicMock() - fake_network.network_range = 'not-a-cidr' - fake_network.name = 'Fake' - ips = _get_discovery_ip_addresses(fake_network) - assert ips == [] - - -@pytest.mark.django_db -class TestGetCredentialEndpointV3: - """Additional GetCredential tests for v3 fields""" - - def test_get_credential_v3_returns_security_fields(self, authenticated_client, test_credential_v3): - """v3 credential response includes security_name, security_level, auth_protocol""" - response = authenticated_client.get(f'/SNMP/GetCredential/{test_credential_v3.id}/') - assert response.status_code == 200 - data = json.loads(response.content) - assert data['version'] == '3' - assert data['security_name'] == 'snmpuser' - assert data['security_level'] == 'authPriv' - assert data['auth_protocol'] == 'sha' - # Passwords must be masked - assert data['auth_pass'] == '***' - assert data['priv_pass'] == '***' - - def test_get_credential_v3_authnopriv_no_priv_fields(self, authenticated_client): - """authNoPriv credential has no priv fields in response""" - cred = Credential.objects.create( - name='v3_authNoPriv', - version='3', - security_name='user', - security_level='authNoPriv', - auth_protocol='sha', - auth_pass='authpass', - ) - response = authenticated_client.get(f'/SNMP/GetCredential/{cred.id}/') - data = json.loads(response.content) - assert 'priv_pass' not in data - assert 'auth_protocol' in data - - -@pytest.mark.django_db -class TestGetNetworkEndpointEdgeCases: - """Tests for GetNetwork, UpdateNetwork, GetNetworkPipelineName error paths""" - - def test_get_network_not_found(self, authenticated_client): - response = authenticated_client.get('/SNMP/GetNetwork/99999/') - assert response.status_code == 404 - assert 'error' in json.loads(response.content) - - def test_update_network_not_found(self, authenticated_client): - response = authenticated_client.post('/SNMP/UpdateNetwork/99999/', { - 'name': 'Ghost', 'network_range': '10.0.0.0/24' - }) - assert response.status_code == 404 - - def test_get_network_pipeline_name_not_found(self, authenticated_client): - response = authenticated_client.get('/SNMP/GetNetworkPipelineName/99999/') - assert response.status_code == 404 - data = json.loads(response.content) - assert data['success'] is False - - def test_update_network_clears_optional_fields_when_empty(self, authenticated_client, test_network): - """Passing empty connection/credential nullifies those FK fields""" - response = authenticated_client.post(f'/SNMP/UpdateNetwork/{test_network.id}/', { - 'name': test_network.name, - 'network_range': test_network.network_range, - 'connection': '', - 'discovery_credential': '', - 'credential': '', - }) - assert response.status_code == 200 - test_network.refresh_from_db() - assert test_network.connection is None - assert test_network.discovery_credential is None - assert test_network.credential is None - - -@pytest.mark.django_db -class TestDeleteNetworkPipelinePaths: - """Tests for DeleteNetwork pipeline deletion branches""" - - @patch('SNMP.snmp_crud.get_elastic_connection') - def test_delete_network_with_pipeline_deleted_reports_it(self, mock_get_es, authenticated_client, test_network): - """When both pipelines exist and are deleted, success message mentions them""" - mock_es = MagicMock() - # make get_pipeline return the pipeline as existing - def get_pipeline_side_effect(id): - return {id: {'pipeline': 'content'}} - mock_es.logstash.get_pipeline.side_effect = get_pipeline_side_effect - mock_es.logstash.delete_pipeline.return_value = {} - mock_get_es.return_value = mock_es - - response = authenticated_client.post(f'/SNMP/DeleteNetwork/{test_network.id}/') - assert response.status_code == 200 - data = json.loads(response.content) - assert data['success'] is True - assert 'pipeline' in data['message'].lower() - - @patch('SNMP.snmp_crud.get_elastic_connection') - def test_delete_network_connection_error_still_deletes_db_record( - self, mock_get_es, authenticated_client, test_network): - """Even if ES connection fails, the DB record is deleted and success=True returned""" - mock_get_es.side_effect = Exception("ES connection failed") - network_id = test_network.id - - response = authenticated_client.post(f'/SNMP/DeleteNetwork/{network_id}/') - assert response.status_code == 200 - data = json.loads(response.content) - assert data['success'] is True - # DB record should be gone - assert not Network.objects.filter(id=network_id).exists() - - def test_delete_network_without_connection_skips_es(self, authenticated_client, test_credential_v2c): - """Network with no connection skips ES interaction and deletes cleanly""" - network = Network.objects.create( - name='No Conn Network', - network_range='172.16.0.0/24', - ) - network_id = network.id - response = authenticated_client.post(f'/SNMP/DeleteNetwork/{network_id}/') - assert response.status_code == 200 - data = json.loads(response.content) - assert data['success'] is True - assert not Network.objects.filter(id=network_id).exists() - - -class TestDefaultTemplateAssignment: - """Tests for automatic Default template assignment (Device.save / DeviceTemplate.delete)""" - - @pytest.fixture - def default_template(self, db): - """Create the official default template as synced from default.json""" - return DeviceTemplate.objects.create( - name='default', - description='Fallback template applied when no other template matches.', - vendor='Any', - official=True - ) - - def test_device_without_template_gets_default(self, default_template, test_network, test_credential_v2c): - """A device saved with no template is auto-assigned the official default template""" - device = Device.objects.create( - name='No Template Device', - ip_address='192.168.1.150', - credential=test_credential_v2c, - network=test_network - ) - assert device.device_template == default_template - - def test_device_keeps_explicit_template(self, default_template, test_network, test_credential_v2c): - """A device saved with an explicit template is not reassigned to default""" - other_template = DeviceTemplate.objects.create( - name='custom_template', - vendor='Any', - official=False - ) - device = Device.objects.create( - name='Templated Device', - ip_address='192.168.1.151', - credential=test_credential_v2c, - network=test_network, - device_template=other_template - ) - assert device.device_template == other_template - - def test_default_template_cannot_be_deleted(self, default_template): - """The official default template is protected from deletion""" - from django.core.exceptions import ValidationError - with pytest.raises(ValidationError): - default_template.delete() - - def test_deleting_template_reassigns_devices_to_default(self, default_template, test_network, test_credential_v2c): - """Deleting a template moves its devices onto the default template""" - doomed_template = DeviceTemplate.objects.create( - name='doomed_template', - vendor='Any', - official=False - ) - device = Device.objects.create( - name='Orphaned Device', - ip_address='192.168.1.152', - credential=test_credential_v2c, - network=test_network, - device_template=doomed_template - ) - doomed_template.delete() - device.refresh_from_db() - assert device.device_template == default_template - - -# ============================================================================ -# DeviceTemplate CRUD Endpoint Tests -# ============================================================================ - -@pytest.fixture -def test_device_template(db): - """Create a custom (non-official) device template.""" - return DeviceTemplate.objects.create( - name='custom_template', - description='A custom test template', - vendor='Cisco', - model='9300', - product='Catalyst', - official=False, - matching_rules=['cisco', 'catalyst'], - ) - - -@pytest.fixture -def official_template(db): - """Create an official (read-only) device template.""" - return DeviceTemplate.objects.create( - name='official_template', - description='Official template', - vendor='Dell', - official=True, - ) - - -@pytest.mark.django_db -class TestDeviceTemplateCRUD: - """Test DeviceTemplate Create, Read, Update, Delete operations via API endpoints.""" - - # ── GetDeviceTemplates ──────────────────────────────────────────────────── - - def test_get_device_templates_returns_list(self, authenticated_client, test_device_template): - response = authenticated_client.get('/SNMP/GetDeviceTemplates/') - assert response.status_code == 200 - data = json.loads(response.content) - assert 'templates' in data - names = [t['name'] for t in data['templates']] - assert 'custom_template' in names - - def test_get_device_templates_includes_required_fields(self, authenticated_client, test_device_template): - response = authenticated_client.get('/SNMP/GetDeviceTemplates/') - data = json.loads(response.content) - template = next(t for t in data['templates'] if t['name'] == 'custom_template') - for field in ('id', 'name', 'display_name', 'vendor', 'model', 'product', 'official'): - assert field in template - - def test_get_device_templates_display_name_formatted(self, authenticated_client, test_device_template): - response = authenticated_client.get('/SNMP/GetDeviceTemplates/') - data = json.loads(response.content) - template = next(t for t in data['templates'] if t['name'] == 'custom_template') - assert template['display_name'] == 'Custom Template' - - # ── GetDeviceTemplate ──────────────────────────────────────────────────── - - def test_get_device_template_by_id(self, authenticated_client, test_device_template): - response = authenticated_client.get(f'/SNMP/GetDeviceTemplate/{test_device_template.id}/') - assert response.status_code == 200 - data = json.loads(response.content) - assert data['name'] == 'custom_template' - assert data['vendor'] == 'Cisco' - assert 'profiles' in data - assert 'matching_rules' in data - - def test_get_device_template_not_found(self, authenticated_client): - response = authenticated_client.get('/SNMP/GetDeviceTemplate/99999/') - # Falls back to GetOfficialDeviceTemplate which returns 404 for unknown names - assert response.status_code in (404, 200) - - def test_get_device_template_includes_profiles(self, authenticated_client, test_device_template, test_profile): - test_device_template.profiles.add(test_profile) - response = authenticated_client.get(f'/SNMP/GetDeviceTemplate/{test_device_template.id}/') - data = json.loads(response.content) - assert any(p['name'] == test_profile.name for p in data['profiles']) - - # ── AddDeviceTemplate ──────────────────────────────────────────────────── - - def test_add_device_template_requires_admin(self, readonly_client): - response = readonly_client.post('/SNMP/AddDeviceTemplate/', { - 'name': 'new_tmpl', - 'vendor': 'Cisco', - }) - assert response.status_code == 403 - - def test_add_device_template_success(self, authenticated_client): - import json as _json - response = authenticated_client.post('/SNMP/AddDeviceTemplate/', { - 'name': 'brand_new_template', - 'description': 'Test', - 'vendor': 'Juniper', - 'model': 'EX2300', - 'product': 'EX', - 'matching_rules': _json.dumps(['juniper', 'ex']), - 'profiles': _json.dumps([]), - }) - assert response.status_code == 200 - data = json.loads(response.content) - assert 'template_id' in data - assert DeviceTemplate.objects.filter(name='brand_new_template').exists() - - def test_add_device_template_missing_name(self, authenticated_client): - import json as _json - response = authenticated_client.post('/SNMP/AddDeviceTemplate/', { - 'vendor': 'Cisco', - 'matching_rules': _json.dumps([]), - 'profiles': _json.dumps([]), - }) - assert response.status_code == 400 - - def test_add_device_template_missing_vendor(self, authenticated_client): - import json as _json - response = authenticated_client.post('/SNMP/AddDeviceTemplate/', { - 'name': 'no_vendor_tmpl', - 'matching_rules': _json.dumps([]), - 'profiles': _json.dumps([]), - }) - assert response.status_code == 400 - - def test_add_device_template_with_profile_ids(self, authenticated_client, test_profile): - import json as _json - response = authenticated_client.post('/SNMP/AddDeviceTemplate/', { - 'name': 'with_profiles', - 'vendor': 'Generic', - 'matching_rules': _json.dumps([]), - 'profiles': _json.dumps([test_profile.id]), - }) - assert response.status_code == 200 - tmpl = DeviceTemplate.objects.get(name='with_profiles') - assert tmpl.profiles.filter(id=test_profile.id).exists() - - # ── UpdateDeviceTemplate ───────────────────────────────────────────────── - - def test_update_device_template_requires_admin(self, readonly_client, test_device_template): - import json as _json - response = readonly_client.post( - f'/SNMP/UpdateDeviceTemplate/{test_device_template.id}/', - { - 'name': 'hacked_name', - 'vendor': 'X', - 'matching_rules': _json.dumps([]), - 'profiles': _json.dumps([]), - }, - ) - assert response.status_code == 403 - - def test_update_device_template_success(self, authenticated_client, test_device_template): - import json as _json - response = authenticated_client.post( - f'/SNMP/UpdateDeviceTemplate/{test_device_template.id}/', - { - 'name': 'updated_template', - 'vendor': 'HPE', - 'model': 'ProLiant', - 'description': 'Updated desc', - 'matching_rules': _json.dumps(['hpe']), - 'profiles': _json.dumps([]), - }, - ) - assert response.status_code == 200 - test_device_template.refresh_from_db() - assert test_device_template.name == 'updated_template' - assert test_device_template.vendor == 'HPE' - - def test_update_official_template_rejected(self, authenticated_client, official_template): - import json as _json - response = authenticated_client.post( - f'/SNMP/UpdateDeviceTemplate/{official_template.id}/', - { - 'name': 'hacked_official', - 'vendor': 'X', - 'matching_rules': _json.dumps([]), - 'profiles': _json.dumps([]), - }, - ) - assert response.status_code == 403 - - def test_update_device_template_not_found(self, authenticated_client): - import json as _json - response = authenticated_client.post( - '/SNMP/UpdateDeviceTemplate/99999/', - { - 'name': 'ghost', - 'vendor': 'X', - 'matching_rules': _json.dumps([]), - 'profiles': _json.dumps([]), - }, - ) - assert response.status_code == 404 - - # ── DeleteDeviceTemplate ────────────────────────────────────────────────── - - def test_delete_device_template_requires_admin(self, readonly_client, test_device_template): - response = readonly_client.post(f'/SNMP/DeleteDeviceTemplate/{test_device_template.id}/') - assert response.status_code == 403 - - def test_delete_device_template_success(self, authenticated_client, test_device_template): - tmpl_id = test_device_template.id - response = authenticated_client.post(f'/SNMP/DeleteDeviceTemplate/{tmpl_id}/') - assert response.status_code == 200 - assert not DeviceTemplate.objects.filter(id=tmpl_id).exists() - - def test_delete_official_template_rejected(self, authenticated_client, official_template): - response = authenticated_client.post(f'/SNMP/DeleteDeviceTemplate/{official_template.id}/') - assert response.status_code == 403 - - def test_delete_device_template_not_found(self, authenticated_client): - response = authenticated_client.post('/SNMP/DeleteDeviceTemplate/99999/') - assert response.status_code == 404 - - -# ============================================================================ -# suggest_device_template -# ============================================================================ - -@pytest.mark.django_db -class TestSuggestDeviceTemplate: - """Test the suggest_device_template pure function in snmp_crud.""" - - @pytest.fixture(autouse=True) - def clear_templates(self, db): - """Ensure no leftover templates pollute suggestion results.""" - DeviceTemplate.objects.all().delete() - - def _make_template(self, name, rules): - return DeviceTemplate.objects.create( - name=name, vendor='Any', matching_rules=rules - ) - - def test_empty_device_info_returns_empty(self): - from SNMP.snmp_crud import suggest_device_template - assert suggest_device_template('') == [] - assert suggest_device_template(None) == [] - - def test_all_rules_match_returns_full_match(self): - from SNMP.snmp_crud import suggest_device_template - tmpl = self._make_template('cisco_cat', ['cisco', 'catalyst']) - result = suggest_device_template('Cisco Catalyst 9300 switch') - assert tmpl.id in result - assert result.index(tmpl.id) == 0 # full match first - - def test_partial_match_returned(self): - from SNMP.snmp_crud import suggest_device_template - tmpl = self._make_template('cisco_any', ['cisco', 'nexus']) - result = suggest_device_template('Cisco Catalyst switch') # 'cisco' matches, 'nexus' doesn't - assert tmpl.id in result - - def test_no_match_excluded(self): - from SNMP.snmp_crud import suggest_device_template - self._make_template('juniper_tmpl', ['juniper', 'ex']) - result = suggest_device_template('Cisco Catalyst 9300') - assert result == [] - - def test_full_match_ranked_before_partial(self): - from SNMP.snmp_crud import suggest_device_template - full = self._make_template('full_match', ['cisco', 'catalyst']) - partial = self._make_template('partial_match', ['cisco', 'nexus']) - result = suggest_device_template('Cisco Catalyst switch') - assert result.index(full.id) < result.index(partial.id) - - def test_case_insensitive_matching(self): - from SNMP.snmp_crud import suggest_device_template - tmpl = self._make_template('caps_tmpl', ['CISCO']) - result = suggest_device_template('cisco ios router') - assert tmpl.id in result - - def test_template_without_rules_excluded(self): - from SNMP.snmp_crud import suggest_device_template - self._make_template('no_rules', []) - result = suggest_device_template('cisco ios') - assert result == [] - - -# ============================================================================ -# GetDeviceLocationData -# ============================================================================ - -@pytest.mark.django_db -class TestGetDeviceLocationData: - """Test the /SNMP/GetDeviceLocationData/ endpoint.""" - - def test_requires_authentication(self, client): - response = client.get('/SNMP/GetDeviceLocationData/') - assert response.status_code == 302 - - def test_returns_empty_lists_when_no_devices(self, authenticated_client): - Device.objects.all().delete() - response = authenticated_client.get('/SNMP/GetDeviceLocationData/') - assert response.status_code == 200 - data = json.loads(response.content) - assert data['sites'] == [] - assert data['site_building'] == [] - assert data['full'] == [] - - def test_returns_sites(self, authenticated_client, test_credential_v2c, test_network): - Device.objects.create( - name='dev_site1', - ip_address='10.0.0.1', - credential=test_credential_v2c, - network=test_network, - site='HQ', - ) - response = authenticated_client.get('/SNMP/GetDeviceLocationData/') - data = json.loads(response.content) - assert 'HQ' in data['sites'] - - def test_site_building_pairs(self, authenticated_client, test_credential_v2c, test_network): - Device.objects.create( - name='dev_sb', - ip_address='10.0.0.2', - credential=test_credential_v2c, - network=test_network, - site='Campus A', - building='Bldg 1', - ) - response = authenticated_client.get('/SNMP/GetDeviceLocationData/') - data = json.loads(response.content) - assert any( - sb['site'] == 'Campus A' and sb['building'] == 'Bldg 1' - for sb in data['site_building'] - ) - - def test_full_entries_with_coordinates(self, authenticated_client, test_credential_v2c, test_network): - Device.objects.create( - name='dev_full', - ip_address='10.0.0.3', - credential=test_credential_v2c, - network=test_network, - site='Site B', - building='Bldg 2', - room='Room 101', - latitude='37.774929', - longitude='-122.419418', - ) - response = authenticated_client.get('/SNMP/GetDeviceLocationData/') - data = json.loads(response.content) - entry = next((e for e in data['full'] if e['room'] == 'Room 101'), None) - assert entry is not None - # lat/lon must be serialised as strings (Decimal-safe) - assert isinstance(entry['latitude'], str) - assert isinstance(entry['longitude'], str) - - def test_devices_without_site_excluded_from_sites(self, authenticated_client, test_credential_v2c, test_network): - Device.objects.create( - name='dev_no_site', - ip_address='10.0.0.4', - credential=test_credential_v2c, - network=test_network, - site=None, - ) - response = authenticated_client.get('/SNMP/GetDeviceLocationData/') - data = json.loads(response.content) - assert None not in data['sites'] - - -# ============================================================================ -# GetDevices – additional coverage (pagination, sorting) -# ============================================================================ - -@pytest.mark.django_db -class TestGetDevicesAdditional: - """Additional GetDevices tests not covered by TestDeviceCRUD.""" - - def test_sort_by_name(self, authenticated_client, test_network, test_credential_v2c): - Device.objects.create(name='Zebra Device', ip_address='10.1.1.1', - credential=test_credential_v2c, network=test_network) - Device.objects.create(name='Alpha Device', ip_address='10.1.1.2', - credential=test_credential_v2c, network=test_network) - response = authenticated_client.get('/SNMP/GetDevices/?sort_by=name') - data = json.loads(response.content) - names = [d['name'] for d in data['devices']] - assert names == sorted(names) - - def test_sort_by_name_descending(self, authenticated_client, test_network, test_credential_v2c): - Device.objects.create(name='ZZZ Device', ip_address='10.1.2.1', - credential=test_credential_v2c, network=test_network) - Device.objects.create(name='AAA Device', ip_address='10.1.2.2', - credential=test_credential_v2c, network=test_network) - response = authenticated_client.get('/SNMP/GetDevices/?sort_by=-name') - data = json.loads(response.content) - names = [d['name'] for d in data['devices']] - assert names == sorted(names, reverse=True) - - def test_pagination_has_next(self, authenticated_client, test_network, test_credential_v2c): - """When more devices than page_size exist, has_next is True.""" - for i in range(5): - Device.objects.create( - name=f'Paged Device {i}', - ip_address=f'10.2.0.{i + 1}', - credential=test_credential_v2c, - network=test_network, - ) - response = authenticated_client.get('/SNMP/GetDevices/?page=1&page_size=2') - data = json.loads(response.content) - assert data['has_next'] is True - assert len(data['devices']) == 2 - - def test_pagination_page_2(self, authenticated_client, test_network, test_credential_v2c): - """Page 2 returns the next slice and has_previous is True.""" - for i in range(4): - Device.objects.create( - name=f'Page2 Device {i}', - ip_address=f'10.3.0.{i + 1}', - credential=test_credential_v2c, - network=test_network, - ) - response = authenticated_client.get('/SNMP/GetDevices/?page=2&page_size=2') - data = json.loads(response.content) - assert data['has_previous'] is True - - def test_search_by_ip(self, authenticated_client, test_network, test_credential_v2c): - Device.objects.create(name='IP Search Device', ip_address='172.16.100.1', - credential=test_credential_v2c, network=test_network) - response = authenticated_client.get('/SNMP/GetDevices/?search=172.16.100') - data = json.loads(response.content) - assert any(d['name'] == 'IP Search Device' for d in data['devices']) - - def test_get_device_returns_location_fields(self, authenticated_client, test_network, test_credential_v2c): - """GetDevice includes location and metadata fields.""" - device = Device.objects.create( - name='Location Device', - ip_address='10.5.5.5', - credential=test_credential_v2c, - network=test_network, - site='HQ', - building='Main', - room='A1', - metadata={'rack': '12'}, - ) - response = authenticated_client.get(f'/SNMP/GetDevice/{device.id}/') - data = json.loads(response.content) - assert data['site'] == 'HQ' - assert data['building'] == 'Main' - assert data['room'] == 'A1' - assert data['metadata'] == {'rack': '12'} - - def test_add_device_with_location_fields(self, authenticated_client, test_network, test_credential_v2c): - """AddDevice persists location and metadata fields.""" - import json as _json - response = authenticated_client.post('/SNMP/AddDevice/', { - 'name': 'Located Device', - 'ip_address': '10.6.6.6', - 'network': test_network.id, - 'credential': test_credential_v2c.id, - 'site': 'West Campus', - 'building': 'B1', - 'room': 'R2', - 'metadata': _json.dumps({'owner': 'infra'}), - }) - assert response.status_code == 200 - device = Device.objects.get(name='Located Device') - assert device.site == 'West Campus' - assert device.metadata == {'owner': 'infra'} - - def test_add_device_with_hostname(self, authenticated_client, test_network, test_credential_v2c): - """AddDevice accepts hostname-only devices (no IP).""" - response = authenticated_client.post('/SNMP/AddDevice/', { - 'name': 'Hostname Only Device', - 'hostname': 'myswitch.example.com', - 'network': test_network.id, - 'credential': test_credential_v2c.id, - }) - assert response.status_code == 200 - device = Device.objects.get(name='Hostname Only Device') - assert device.hostname == 'myswitch.example.com' - assert device.ip_address is None - - -# ============================================================================ -# Unit tests for _build_trap_components helper -# ============================================================================ - -@pytest.mark.django_db -class TestBuildTrapComponents: - """Direct unit tests for the _build_trap_components() internal helper""" - - def test_v2c_trap_components_have_correct_structure( - self, test_connection, test_credential_v2c - ): - from SNMP.snmp_crud import _build_trap_components - network = Network.objects.create( - name='Trap V2c Network', - network_range='10.10.0.0/24', - connection=test_connection, - credential=test_credential_v2c, - traps_enabled=True, - credential_mode='PLAINTEXT', - ) - result = _build_trap_components(network) - assert 'input' in result - assert 'filter' in result - assert 'output' in result - assert len(result['input']) == 1 - assert result['input'][0]['plugin'] == 'snmptrap' - trap_cfg = result['input'][0]['config'] - assert '2c' in trap_cfg.get('supported_versions', []) - - def test_v1_trap_components_include_v1_version( - self, test_connection, test_credential_v2c - ): - from SNMP.snmp_crud import _build_trap_components - cred = Credential.objects.create(name='V1 Trap Cred', version='1', community='public') - network = Network.objects.create( - name='Trap V1 Network', - network_range='10.11.0.0/24', - connection=test_connection, - credential=cred, - traps_enabled=True, - credential_mode='PLAINTEXT', - ) - result = _build_trap_components(network) - trap_cfg = result['input'][0]['config'] - assert '1' in trap_cfg.get('supported_versions', []) - - def test_v3_trap_components_include_security_fields( - self, test_connection, test_credential_v3 - ): - from SNMP.snmp_crud import _build_trap_components - network = Network.objects.create( - name='Trap V3 Network', - network_range='10.12.0.0/24', - connection=test_connection, - credential=test_credential_v3, - traps_enabled=True, - credential_mode='PLAINTEXT', - ) - result = _build_trap_components(network) - trap_cfg = result['input'][0]['config'] - assert '3' in trap_cfg.get('supported_versions', []) - assert 'security_name' in trap_cfg - - def test_keystore_mode_emits_keystore_references( - self, test_connection, test_credential_v2c - ): - from SNMP.snmp_crud import _build_trap_components - network = Network.objects.create( - name='Trap Keystore Network', - network_range='10.13.0.0/24', - connection=test_connection, - credential=test_credential_v2c, - traps_enabled=True, - credential_mode='KEYSTORE', - ) - result = _build_trap_components(network) - trap_cfg = result['input'][0]['config'] - # In KEYSTORE mode the community string should be a ${...} reference - community = trap_cfg.get('community', []) - assert community and community[0].startswith('${') - - def test_plaintext_mode_emits_decrypted_community( - self, test_connection, test_credential_v2c - ): - from SNMP.snmp_crud import _build_trap_components - network = Network.objects.create( - name='Trap Plaintext Network', - network_range='10.14.0.0/24', - connection=test_connection, - credential=test_credential_v2c, - traps_enabled=True, - credential_mode='PLAINTEXT', - ) - result = _build_trap_components(network) - trap_cfg = result['input'][0]['config'] - community = trap_cfg.get('community', []) - assert community and not community[0].startswith('${') - - def test_filter_adds_event_category_traps(self, test_connection, test_credential_v2c): - from SNMP.snmp_crud import _build_trap_components - network = Network.objects.create( - name='Trap Filter Check', - network_range='10.15.0.0/24', - connection=test_connection, - credential=test_credential_v2c, - traps_enabled=True, - credential_mode='PLAINTEXT', - ) - result = _build_trap_components(network) - mutate_filter = next( - (f for f in result['filter'] if f.get('plugin') == 'mutate'), None - ) - assert mutate_filter is not None - add_field = mutate_filter['config'].get('add_field', {}) - assert add_field.get('[event][category]') == 'traps' - - -# ============================================================================ -# Unit tests for _build_network_pipeline_configs helper -# ============================================================================ - -@pytest.mark.django_db -class TestBuildNetworkPipelineConfigs: - """Direct unit tests for the _build_network_pipeline_configs() internal helper""" - - def test_returns_empty_when_no_devices(self, test_connection, test_credential_v2c): - from SNMP.snmp_crud import _build_network_pipeline_configs - network = Network.objects.create( - name='Empty Network Configs', - network_range='10.20.0.0/24', - connection=test_connection, - traps_enabled=False, - discovery_enabled=False, - ) - results = _build_network_pipeline_configs(network) - assert results == [] - - def test_returns_polling_pipeline_for_v2c_device( - self, test_connection, test_credential_v2c - ): - from SNMP.snmp_crud import _build_network_pipeline_configs - network = Network.objects.create( - name='Polling V2c Configs', - network_range='10.21.0.0/24', - connection=test_connection, - traps_enabled=False, - discovery_enabled=False, - ) - Device.objects.create( - name='Config Test Device', - ip_address='10.21.0.10', - credential=test_credential_v2c, - network=network, - ) - results = _build_network_pipeline_configs(network) - assert len(results) >= 1 - pipeline_types = [r['pipeline_type'] for r in results] - assert 'polling' in pipeline_types - - def test_trap_pipeline_included_when_enabled( - self, test_connection, test_credential_v2c - ): - from SNMP.snmp_crud import _build_network_pipeline_configs - network = Network.objects.create( - name='Trap Enabled Configs', - network_range='10.22.0.0/24', - connection=test_connection, - credential=test_credential_v2c, - traps_enabled=True, - discovery_enabled=False, - credential_mode='PLAINTEXT', - ) - Device.objects.create( - name='Trap Config Device', - ip_address='10.22.0.10', - credential=test_credential_v2c, - network=network, - ) - results = _build_network_pipeline_configs(network) - pipeline_types = [r['pipeline_type'] for r in results] - assert 'trap' in pipeline_types - - def test_discovery_pipeline_included_when_enabled( - self, test_connection, test_credential_v2c - ): - from SNMP.snmp_crud import _build_network_pipeline_configs - network = Network.objects.create( - name='Discovery Enabled Configs', - network_range='10.23.0.0/24', - connection=test_connection, - discovery_credential=test_credential_v2c, - traps_enabled=False, - discovery_enabled=True, - credential_mode='PLAINTEXT', - ) - Device.objects.create( - name='Discovery Config Device', - ip_address='10.23.0.10', - credential=test_credential_v2c, - network=network, - ) - results = _build_network_pipeline_configs(network) - pipeline_types = [r['pipeline_type'] for r in results] - assert 'discovery' in pipeline_types - - def test_pipeline_name_contains_network_name( - self, test_connection, test_credential_v2c - ): - from SNMP.snmp_crud import _build_network_pipeline_configs - network = Network.objects.create( - name='Name Check Network', - network_range='10.24.0.0/24', - connection=test_connection, - traps_enabled=False, - discovery_enabled=False, - ) - Device.objects.create( - name='Name Check Device', - ip_address='10.24.0.10', - credential=test_credential_v2c, - network=network, - ) - results = _build_network_pipeline_configs(network) - for r in results: - assert 'name_check_network' in r['pipeline_name'] - - def test_config_is_valid_logstash_syntax( - self, test_connection, test_credential_v2c - ): - from SNMP.snmp_crud import _build_network_pipeline_configs - network = Network.objects.create( - name='Syntax Check Network', - network_range='10.25.0.0/24', - connection=test_connection, - traps_enabled=False, - discovery_enabled=False, - ) - Device.objects.create( - name='Syntax Device', - ip_address='10.25.0.10', - credential=test_credential_v2c, - network=network, - ) - results = _build_network_pipeline_configs(network) - for r in results: - config = r['config'] - assert 'input {' in config - assert 'output {' in config - - -@pytest.mark.django_db -class TestDeviceVisualizationData: - """Regression tests for device visualization data shaping. - - Covers the defects that left the device detail panel blank or wrong while the - underlying data was present: interface values nested under OpenConfig's - ``state``, and memory being sourced from the wrong place. - """ - - def _search_stub(self, by_category, captured=None): - """Build an es.search side_effect that dispatches on event.category.""" - def search(**kwargs): - filters = kwargs['query']['bool']['filter'] - if captured is not None: - captured.append(filters) - categories = [f['term']['event.category'] for f in filters - if 'term' in f and 'event.category' in f['term']] - for category in categories: - if category in by_category: - return {'hits': {'hits': by_category[category]}} - return {'hits': {'hits': []}} - return search - - # ---- interface shaping ---- - - def test_interfaces_flatten_openconfig_state_and_counters(self, test_device): - """state.* AND state.counters.* are lifted to where the UI reads them.""" - from SNMP.snmp_crud import _get_device_interfaces - - mock_es = MagicMock() - mock_es.search.return_value = { - 'aggregations': {'fans': {'buckets': [ - {'top_if_doc': {'hits': {'hits': [{'_source': {'interface': { - 'name': 'Ethernet1', - 'index': 1, - 'state': { - 'admin_status': 'UP', - 'oper_status': 'DOWN', - 'speed': 1000000000.0, - 'counters': {'in_octets': 42, 'out_errors': 7}, - }, - }}}]}}}, - ]}} - } - - iface = _get_device_interfaces(test_device, mock_es)['interfaces'][0] - - assert iface['admin_status'] == 'UP' - assert iface['oper_status'] == 'DOWN' - assert iface['speed'] == 1000000000.0 - # Counters must be flat — createInterfaceCard reads iface.in_octets, so - # leaving them at iface.counters.in_octets renders 0 B on a busy link. - assert iface['in_octets'] == 42 - assert iface['out_errors'] == 7 - assert iface['name'] == 'Ethernet1' - assert iface['index'] == 1 - assert 'state' not in iface - - def test_normalized_top_level_status_wins_over_raw_state(self, test_device): - """A raw state value must not clobber the pipeline-normalized one.""" - from SNMP.snmp_crud import _get_device_interfaces - - mock_es = MagicMock() - mock_es.search.return_value = { - 'aggregations': {'fans': {'buckets': [ - {'top_if_doc': {'hits': {'hits': [{'_source': {'interface': { - 'name': 'Ethernet1', - # Translate normalizer already decoded 1 -> UP here. - 'oper_status': 'UP', - # ...while the raw enum survives under state. - 'state': {'oper_status': 1, 'admin_status': 'UP'}, - }}}]}}}, - ]}} - } - - iface = _get_device_interfaces(test_device, mock_es)['interfaces'][0] - - # The UI compares strictly against 'UP'; the raw 1 would render "Unknown". - assert iface['oper_status'] == 'UP' - # Keys only present under state still come through. - assert iface['admin_status'] == 'UP' - - def test_interfaces_without_state_are_passed_through(self, test_device): - """A device that already reports flat status is left alone.""" - from SNMP.snmp_crud import _get_device_interfaces - - mock_es = MagicMock() - mock_es.search.return_value = { - 'aggregations': {'fans': {'buckets': [ - {'top_if_doc': {'hits': {'hits': [{'_source': {'interface': { - 'name': 'Ethernet1', 'admin_status': 'UP', 'oper_status': 'UP', - }}}]}}}, - ]}} - } - - iface = _get_device_interfaces(test_device, mock_es)['interfaces'][0] - assert iface['admin_status'] == 'UP' - assert iface['oper_status'] == 'UP' - - # ---- metrics sourcing ---- - - def test_canonical_memory_field_is_preferred(self, test_device): - """Profiles deriving system.memory.actual.used.pct keep working.""" - from SNMP.snmp_crud import _get_device_metrics - - captured = [] - mock_es = MagicMock() - mock_es.search.side_effect = self._search_stub({ - 'metrics': [{'_source': { - '@timestamp': '2026-08-04T01:00:00Z', - 'system': { - 'cpu': {'total': {'norm': {'pct': 0.25}}}, - 'memory': {'actual': {'used': {'pct': 0.19}}}, - }, - 'host': {'uptime': 12345}, - }}], - }, captured) - - metrics = _get_device_metrics(test_device, mock_es) - - assert metrics['CPU'] == [0.25] - assert metrics['Memory'] == [0.19] - assert metrics['MemorySource'] == 'system.memory.actual.used.pct' - assert metrics['Uptime'] == 12345 - # The storage-table fallback must not be queried when the canonical - # field is present — that query is pure overhead here. - queried = [f['term']['event.category'] for fl in captured for f in fl - if 'term' in f and 'event.category' in f['term']] - assert 'system.filesystem' not in queried - - def test_cpu_returned_when_memory_is_absent(self, test_device): - """CPU must survive on its own — it used to be dropped with memory.""" - from SNMP.snmp_crud import _get_device_metrics - - mock_es = MagicMock() - mock_es.search.side_effect = self._search_stub({ - 'metrics': [{'_source': { - '@timestamp': '2026-08-04T01:00:00Z', - 'system': {'cpu': {'total': {'norm': {'pct': 0.25}}}}, - 'host': {'uptime': 12345}, - }}], - }) - - metrics = _get_device_metrics(test_device, mock_es) - - assert metrics['CPU'] == [0.25] - assert metrics['CPUTime'] == ['2026-08-04T01:00:00Z'] - assert metrics['Memory'] == [] - assert metrics['MemorySource'] is None - - def test_memory_falls_back_to_physical_hrstorage_ram_row(self, test_device): - """Without the canonical field, use hrStorageRam — physical, not cache.""" - from SNMP.snmp_crud import _get_device_metrics - - captured = [] - - def search(**kwargs): - filters = kwargs['query']['bool']['filter'] - captured.append(filters) - categories = [f['term']['event.category'] for f in filters - if 'term' in f and 'event.category' in f['term']] - if 'metrics' in categories: - return {'hits': {'hits': [{'_source': { - '@timestamp': '2026-08-04T01:00:00Z', - 'system': {'cpu': {'total': {'norm': {'pct': 0.25}}}}, - }}]}} - # The aggregation asks for the lowest hrStorageIndex per poll, so the - # physical row is what comes back — cache/buffers rows are ranked out - # by the sort, which is the behaviour being pinned here. - return {'aggregations': {'by_poll': {'buckets': [ - {'key': 1, 'physical': {'hits': {'hits': [{'_source': { - '@timestamp': '2026-08-04T01:00:00Z', - 'system': {'filesystem': {'used': {'pct': 0.98}}}, - }}]}}}, - ]}}} - - mock_es = MagicMock() - mock_es.search.side_effect = search - - metrics = _get_device_metrics(test_device, mock_es) - - assert metrics['Memory'] == [0.98] - assert metrics['MemoryTime'] == ['2026-08-04T01:00:00Z'] - assert metrics['MemorySource'] == 'hrStorageRam' - - fs_filters = [fl for fl in captured - if any('term' in f and f['term'].get('event.category') == 'system.filesystem' - for f in fl)] - assert fs_filters, "expected a system.filesystem query" - - # Rows are selected by hrStorageType, not by a locale-specific description - # ("RAM" on EOS vs "Physical memory" on net-snmp), and matched under both - # possible mappings of that field. - shoulds = [f['bool']['should'] for f in fs_filters[0] if 'bool' in f] - assert shoulds, "expected a bool/should type filter" - fields = {list(clause['term'].keys())[0] for clause in shoulds[0]} - assert fields == {'system.filesystem.type', 'system.filesystem.type.keyword'} - assert all(list(c['term'].values())[0] == '1.3.6.1.2.1.25.2.1.2' for c in shoulds[0]) - assert not any('mount_point' in str(f) for f in fs_filters[0]) - - # All RAM rows report the same total, so the row must be disambiguated by - # lowest hrStorageIndex — ranking by total silently picks an arbitrary row. - agg = mock_es.search.call_args_list[-1].kwargs['aggregations'] - top_hits = agg['by_poll']['aggregations']['physical']['top_hits'] - assert top_hits['sort'] == [{'system.filesystem.index': {'order': 'asc'}}] - assert top_hits['size'] == 1 - - def test_metrics_queries_are_scoped_to_the_device(self, test_device): - """Every metrics query must filter to this device, not the whole fleet.""" - from SNMP.snmp_crud import _get_device_metrics - - captured = [] - mock_es = MagicMock() - mock_es.search.side_effect = self._search_stub({ - 'metrics': [{'_source': { - '@timestamp': '2026-08-04T01:00:00Z', - 'system': {'cpu': {'total': {'norm': {'pct': 0.25}}}}, - }}], - 'system.filesystem': [], - }, captured) - - _get_device_metrics(test_device, mock_es) - - assert captured, "expected at least one query" - for filters in captured: - assert {"term": {"host.polled_address": test_device.ip_address}} in filters - assert {"range": {"@timestamp": {"gte": "now-6h"}}} in filters - - def test_uptime_defaults_to_zero_without_metrics_docs(self, test_device): - """No metrics documents must not raise.""" - from SNMP.snmp_crud import _get_device_metrics - - mock_es = MagicMock() - mock_es.search.side_effect = self._search_stub({}) - - metrics = _get_device_metrics(test_device, mock_es) - - assert metrics['Uptime'] == 0 - assert metrics['CPU'] == [] - assert metrics['Memory'] == [] diff --git a/src/logstashui/SNMP/tests/test_snmp_grounding.py b/src/logstashui/SNMP/tests/test_snmp_grounding.py deleted file mode 100644 index 961682a..0000000 --- a/src/logstashui/SNMP/tests/test_snmp_grounding.py +++ /dev/null @@ -1,263 +0,0 @@ -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. - -""" -Tests for SNMP.snmp_grounding — all pure functions, no DB or network required. -""" - -import json -import os -import tempfile -import pytest - -from SNMP.snmp_grounding import ( - build_grounding, - load_grounding, - reduce_and_ground, -) - - -# =========================================================================== -# Helpers -# =========================================================================== - -def _write_mib(directory, filename, data): - path = os.path.join(directory, filename) - with open(path, 'w') as f: - json.dump(data, f) - return path - - -def _minimal_mib(obj_name, oid, nodetype='scalar', typ='Integer32', enum=None, units=None): - """ - Return a minimal pysmi-style MIB dict for one object. - build_grounding iterates the file dict directly as {obj_name: obj_body}, - so the file should contain {obj_name: {...}} at the top level. - """ - syntax = {'type': typ} - if enum: - syntax['constraints'] = {'enumeration': enum} - obj = { - 'oid': oid, - 'nodetype': nodetype, - 'syntax': syntax, - 'maxaccess': 'read-only', - } - if units: - obj['units'] = units - return {obj_name: obj} - - -# =========================================================================== -# build_grounding -# =========================================================================== - -class TestBuildGrounding: - - def test_empty_directory_returns_empty_dict(self): - with tempfile.TemporaryDirectory() as d: - result = build_grounding(d) - assert result == {} - - def test_scalar_object_included(self): - with tempfile.TemporaryDirectory() as d: - mib = _minimal_mib('sysDescr', '1.3.6.1.2.1.1.1', nodetype='scalar') - _write_mib(d, 'RFC1213-MIB.json', mib) - result = build_grounding(d) - assert '1.3.6.1.2.1.1.1' in result - entry = result['1.3.6.1.2.1.1.1'] - assert entry['name'] == 'sysDescr' - assert entry['nodetype'] == 'scalar' - - def test_column_object_included(self): - with tempfile.TemporaryDirectory() as d: - mib = _minimal_mib('ifDescr', '1.3.6.1.2.1.2.2.1.2', nodetype='column') - _write_mib(d, 'IF-MIB.json', mib) - result = build_grounding(d) - assert '1.3.6.1.2.1.2.2.1.2' in result - - def test_enum_is_inverted(self): - """pysmi stores enums as {name: int}; build_grounding inverts to {int: name}.""" - with tempfile.TemporaryDirectory() as d: - mib = _minimal_mib( - 'ifOperStatus', '1.3.6.1.2.1.2.2.1.8', - nodetype='scalar', - enum={'up': 1, 'down': 2}, - ) - _write_mib(d, 'IF-MIB.json', mib) - result = build_grounding(d) - assert result['1.3.6.1.2.1.2.2.1.8']['enum'] == {1: 'up', 2: 'down'} - - def test_invalid_json_file_skipped(self): - with tempfile.TemporaryDirectory() as d: - bad_path = os.path.join(d, 'bad.json') - with open(bad_path, 'w') as f: - f.write('this is not json {{{') - result = build_grounding(d) - assert result == {} - - def test_non_dict_json_file_skipped(self): - with tempfile.TemporaryDirectory() as d: - _write_mib(d, 'list.json', [1, 2, 3]) - result = build_grounding(d) - assert result == {} - - def test_non_scalar_column_nodetype_excluded(self): - with tempfile.TemporaryDirectory() as d: - mib = {'someRow': {'oid': '1.2.3', 'nodetype': 'row', 'syntax': {}}} - _write_mib(d, 'TEST-MIB.json', mib) - result = build_grounding(d) - assert '1.2.3' not in result - - def test_units_included_when_defined(self): - with tempfile.TemporaryDirectory() as d: - mib = _minimal_mib('sysUpTime', '1.3.6.1.2.1.1.3', - nodetype='scalar', typ='TimeTicks', - units='hundredths of a second') - _write_mib(d, 'MIB.json', mib) - result = build_grounding(d) - assert result['1.3.6.1.2.1.1.3']['units'] == 'hundredths of a second' - - def test_multiple_files_merged(self): - with tempfile.TemporaryDirectory() as d: - mib1 = _minimal_mib('sysDescr', '1.3.6.1.2.1.1.1', nodetype='scalar') - mib2 = _minimal_mib('ifDescr', '1.3.6.1.2.1.2.2.1.2', nodetype='column') - _write_mib(d, 'MIB1.json', mib1) - _write_mib(d, 'MIB2.json', mib2) - result = build_grounding(d) - assert '1.3.6.1.2.1.1.1' in result - assert '1.3.6.1.2.1.2.2.1.2' in result - - -# =========================================================================== -# load_grounding -# =========================================================================== - -class TestLoadGrounding: - - def test_valid_json_file_loaded(self): - with tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False) as f: - json.dump({'1.2.3': {'name': 'sysDescr'}}, f) - path = f.name - try: - result = load_grounding(path) - assert '1.2.3' in result - finally: - os.unlink(path) - - def test_missing_file_returns_empty_dict(self): - result = load_grounding('/nonexistent/path/grounding.json') - assert result == {} - - def test_invalid_json_returns_empty_dict(self): - with tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False) as f: - f.write('not valid json !!!') - path = f.name - try: - result = load_grounding(path) - assert result == {} - finally: - os.unlink(path) - - -# =========================================================================== -# reduce_and_ground -# =========================================================================== - -class TestReduceAndGround: - """ - reduce_and_ground is pure; we supply a synthetic grounding dict so tests - don't depend on the compiled grounding.json file. - """ - - GROUNDING = { - '1.3.6.1.2.1.1.1': { - 'name': 'sysDescr', 'mib': 'RFC1213-MIB', - 'type': 'OctetString', 'enum': None, - 'nodetype': 'scalar', 'access': 'read-only', - }, - '1.3.6.1.2.1.2.2.1.2': { - 'name': 'ifDescr', 'mib': 'IF-MIB', - 'type': 'DisplayString', 'enum': None, - 'nodetype': 'column', 'access': 'read-only', - }, - '1.3.6.1.2.1.2.2.1.8': { - 'name': 'ifOperStatus', 'mib': 'IF-MIB', - 'type': 'Integer32', - 'enum': {1: 'up', 2: 'down'}, - 'nodetype': 'column', 'access': 'read-only', - }, - } - - def test_empty_walk_returns_empty_results(self): - grounded, ungrounded = reduce_and_ground('', self.GROUNDING) - assert grounded == [] - assert ungrounded == [] - - def test_known_scalar_oid_grounded(self): - walk = '1.3.6.1.2.1.1.1.0 = Linux router' - grounded, ungrounded = reduce_and_ground(walk, self.GROUNDING) - assert len(grounded) == 1 - assert grounded[0]['name'] == 'sysDescr' - - def test_known_table_oid_with_instance_grounded(self): - # ifDescr.1 — the '.1' is the instance arc (row index) - walk = '1.3.6.1.2.1.2.2.1.2.1 = GigabitEthernet0/0' - grounded, ungrounded = reduce_and_ground(walk, self.GROUNDING) - names = [r['name'] for r in grounded] - assert 'ifDescr' in names - - def test_multiple_rows_of_same_column_counted(self): - walk = ( - '1.3.6.1.2.1.2.2.1.2.1 = GigabitEthernet0/0\n' - '1.3.6.1.2.1.2.2.1.2.2 = GigabitEthernet0/1\n' - ) - grounded, _ = reduce_and_ground(walk, self.GROUNDING) - ifdescr = next(r for r in grounded if r['name'] == 'ifDescr') - assert ifdescr['instances'] == 2 - - def test_unknown_oid_goes_to_ungrounded(self): - walk = '9.9.9.9.9.9 = SomeValue' - _, ungrounded = reduce_and_ground(walk, self.GROUNDING) - assert len(ungrounded) > 0 - - def test_grounded_sorted_by_oid_numerically(self): - walk = ( - '1.3.6.1.2.1.2.2.1.8.1 = 1\n' - '1.3.6.1.2.1.1.1.0 = router\n' - ) - grounded, _ = reduce_and_ground(walk, self.GROUNDING) - oids = [r['oid'] for r in grounded] - assert oids == sorted(oids, key=lambda o: [int(x) for x in o.split('.')]) - - def test_sample_value_truncated_to_50_chars(self): - long_value = 'X' * 200 - walk = f'1.3.6.1.2.1.1.1.0 = {long_value}' - grounded, _ = reduce_and_ground(walk, self.GROUNDING) - assert len(grounded[0]['sample']) <= 50 - - def test_malformed_lines_skipped(self): - walk = 'this is not a valid walk line\n1.3.6.1.2.1.1.1.0 = Linux' - grounded, _ = reduce_and_ground(walk, self.GROUNDING) - assert len(grounded) == 1 - - def test_enum_propagated_to_grounded_entry(self): - walk = '1.3.6.1.2.1.2.2.1.8.1 = 1' - grounded, _ = reduce_and_ground(walk, self.GROUNDING) - entry = next(r for r in grounded if r['name'] == 'ifOperStatus') - assert entry['enum'] == {1: 'up', 2: 'down'} - - def test_uses_module_level_grounding_when_none_passed(self): - # Should not raise even when the module-level GROUNDING may be empty. - walk = '1.3.6.1.2.1.1.1.0 = test' - grounded, ungrounded = reduce_and_ground(walk) # no grounding arg - # Result depends on the compiled file; just verify it doesn't crash - assert isinstance(grounded, list) - assert isinstance(ungrounded, list) - - def test_tab_separated_walk_format(self): - walk = '1.3.6.1.2.1.1.1.0\tLinux router' - grounded, _ = reduce_and_ground(walk, self.GROUNDING) - assert len(grounded) == 1 - assert grounded[0]['name'] == 'sysDescr' diff --git a/src/logstashui/SNMP/tests/test_snmp_normalizers.py b/src/logstashui/SNMP/tests/test_snmp_normalizers.py deleted file mode 100644 index 5945aa8..0000000 --- a/src/logstashui/SNMP/tests/test_snmp_normalizers.py +++ /dev/null @@ -1,545 +0,0 @@ -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. - -""" -Tests for SNMP.snmp_normalizers — all pure functions, no DB or network required. -""" - -import pytest - -from SNMP.snmp_normalizers import ( - _apply_normalizers, - _generate_multiply_get_filter, - _generate_ratio_get_filter, - _generate_translate_filter, -) - - -# =========================================================================== -# _generate_multiply_get_filter -# =========================================================================== - -class TestGenerateMultiplyGetFilter: - - def test_returns_none_for_empty_list(self): - assert _generate_multiply_get_filter([]) is None - - def test_returns_comment_and_ruby_filter(self): - normalizers = [ - { - 'operation': 'multiply', - 'target': {'scope': 'get', 'field': 'system.cpu.total.norm.pct'}, - 'params': {'multiply_value': 0.01} - } - ] - result = _generate_multiply_get_filter(normalizers) - assert isinstance(result, list) - assert len(result) == 2 - comment, ruby = result - assert comment['plugin'] == 'comment' - assert ruby['plugin'] == 'ruby' - - def test_ruby_code_contains_field_path(self): - normalizers = [ - { - 'operation': 'multiply', - 'target': {'scope': 'get', 'field': 'system.cpu.total.norm.pct'}, - 'params': {'multiply_value': 0.01} - } - ] - result = _generate_multiply_get_filter(normalizers) - ruby_code = result[1]['config']['code'] - assert '[system][cpu][total][norm][pct]' in ruby_code - - def test_ruby_code_contains_multiply_value(self): - normalizers = [ - { - 'operation': 'multiply', - 'target': {'scope': 'get', 'field': 'some.metric'}, - 'params': {'multiply_value': 100} - } - ] - result = _generate_multiply_get_filter(normalizers) - ruby_code = result[1]['config']['code'] - assert '100' in ruby_code - - def test_skips_normalizer_missing_field(self): - normalizers = [ - { - 'operation': 'multiply', - 'target': {'scope': 'get'}, # no 'field' - 'params': {'multiply_value': 0.01} - } - ] - result = _generate_multiply_get_filter(normalizers) - assert result is None - - def test_skips_normalizer_missing_multiply_value(self): - normalizers = [ - { - 'operation': 'multiply', - 'target': {'scope': 'get', 'field': 'some.metric'}, - 'params': {} # no 'multiply_value' - } - ] - result = _generate_multiply_get_filter(normalizers) - assert result is None - - def test_multiple_normalizers_in_single_filter(self): - normalizers = [ - { - 'operation': 'multiply', - 'target': {'scope': 'get', 'field': 'metric.a'}, - 'params': {'multiply_value': 2} - }, - { - 'operation': 'multiply', - 'target': {'scope': 'get', 'field': 'metric.b'}, - 'params': {'multiply_value': 3} - } - ] - result = _generate_multiply_get_filter(normalizers) - # Both fields consolidated into single ruby filter - assert isinstance(result, list) - ruby_code = result[1]['config']['code'] - assert '[metric][a]' in ruby_code - assert '[metric][b]' in ruby_code - - def test_comment_mentions_multiply(self): - normalizers = [ - { - 'operation': 'multiply', - 'target': {'scope': 'get', 'field': 'metric.a'}, - 'params': {'multiply_value': 0.5} - } - ] - result = _generate_multiply_get_filter(normalizers) - comment_text = result[0]['config']['text'] - assert 'Multiply' in comment_text - - -# =========================================================================== -# _generate_ratio_get_filter -# =========================================================================== - -class TestGenerateRatioGetFilter: - - def test_returns_none_for_empty_list(self): - assert _generate_ratio_get_filter([]) is None - - def test_returns_comment_and_ruby_filter(self): - normalizers = [ - { - 'operation': 'ratio', - 'target': {'scope': 'get'}, - 'params': { - 'value1_field': 'memory.used', - 'value2_field': 'memory.free', - 'total_output_field': 'memory.total', - } - } - ] - result = _generate_ratio_get_filter(normalizers) - assert isinstance(result, list) - assert len(result) == 2 - comment, ruby = result - assert comment['plugin'] == 'comment' - assert ruby['plugin'] == 'ruby' - - def test_skips_normalizer_missing_value_fields(self): - normalizers = [ - { - 'operation': 'ratio', - 'target': {'scope': 'get'}, - 'params': { - 'value1_field': 'memory.used', - # no value2_field - } - } - ] - result = _generate_ratio_get_filter(normalizers) - assert result is None - - def test_ruby_code_contains_field_paths(self): - normalizers = [ - { - 'operation': 'ratio', - 'target': {'scope': 'get'}, - 'params': { - 'value1_field': 'memory.used', - 'value2_field': 'memory.free', - } - } - ] - result = _generate_ratio_get_filter(normalizers) - ruby_code = result[1]['config']['code'] - assert '[memory][used]' in ruby_code - assert '[memory][free]' in ruby_code - - def test_optional_output_fields_included_when_specified(self): - normalizers = [ - { - 'operation': 'ratio', - 'target': {'scope': 'get'}, - 'params': { - 'value1_field': 'mem.used', - 'value2_field': 'mem.free', - 'total_output_field': 'mem.total', - 'ratio1_output_field': 'mem.used_pct', - 'ratio2_output_field': 'mem.free_pct', - 'complement_ratio_output_field': 'mem.complement', - 'divide_output_field': 'mem.divided', - } - } - ] - result = _generate_ratio_get_filter(normalizers) - ruby_code = result[1]['config']['code'] - assert '[mem][total]' in ruby_code - assert '[mem][used_pct]' in ruby_code - assert '[mem][free_pct]' in ruby_code - assert '[mem][complement]' in ruby_code - assert '[mem][divided]' in ruby_code - - def test_multiple_ratio_normalizers_use_unique_variable_names(self): - normalizers = [ - { - 'operation': 'ratio', - 'target': {'scope': 'get'}, - 'params': {'value1_field': 'a.used', 'value2_field': 'a.free'} - }, - { - 'operation': 'ratio', - 'target': {'scope': 'get'}, - 'params': {'value1_field': 'b.used', 'value2_field': 'b.free'} - } - ] - result = _generate_ratio_get_filter(normalizers) - ruby_code = result[1]['config']['code'] - # With multiple normalizers, suffixes are added to variable names - assert 'value1_0' in ruby_code - assert 'value1_1' in ruby_code - - def test_comment_mentions_ratio(self): - normalizers = [ - { - 'operation': 'ratio', - 'target': {'scope': 'get'}, - 'params': {'value1_field': 'a.used', 'value2_field': 'a.free'} - } - ] - result = _generate_ratio_get_filter(normalizers) - comment_text = result[0]['config']['text'] - assert 'Ratio' in comment_text - - -# =========================================================================== -# _generate_translate_filter -# =========================================================================== - -class TestGenerateTranslateFilter: - - def test_returns_none_for_empty_list(self): - assert _generate_translate_filter([]) is None - - def test_returns_comment_and_translate_filter(self): - normalizers = [ - { - 'operation': 'translate', - 'target': {'scope': 'table', 'field': 'interface.admin_status'}, - 'params': { - 'mapping': {'1': 'UP', '2': 'DOWN', '3': 'TESTING'} - } - } - ] - result = _generate_translate_filter(normalizers) - assert isinstance(result, list) - assert len(result) == 2 - comment, translate = result - assert comment['plugin'] == 'comment' - assert translate['plugin'] == 'translate' - - def test_translate_config_has_correct_source_and_destination(self): - normalizers = [ - { - 'operation': 'translate', - 'target': {'scope': 'table', 'field': 'interface.oper_status'}, - 'params': {'mapping': {'1': 'UP', '2': 'DOWN'}} - } - ] - result = _generate_translate_filter(normalizers) - translate_config = result[1]['config'] - assert translate_config['source'] == '[interface][oper_status]' - assert translate_config['destination'] == '[interface][oper_status]' - - def test_translate_config_has_override_true(self): - normalizers = [ - { - 'operation': 'translate', - 'target': {'scope': 'table', 'field': 'interface.admin_status'}, - 'params': {'mapping': {'1': 'UP'}} - } - ] - result = _generate_translate_filter(normalizers) - assert result[1]['config']['override'] is True - - def test_translate_config_contains_mapping(self): - mapping = {'1': 'UP', '2': 'DOWN', '3': 'TESTING'} - normalizers = [ - { - 'operation': 'translate', - 'target': {'scope': 'table', 'field': 'interface.admin_status'}, - 'params': {'mapping': mapping} - } - ] - result = _generate_translate_filter(normalizers) - assert result[1]['config']['dictionary'] == mapping - - def test_skips_normalizer_without_field(self): - normalizers = [ - { - 'operation': 'translate', - 'target': {'scope': 'table'}, # no 'field' - 'params': {'mapping': {'1': 'UP'}} - } - ] - result = _generate_translate_filter(normalizers) - assert result is None - - def test_skips_normalizer_without_mapping(self): - normalizers = [ - { - 'operation': 'translate', - 'target': {'scope': 'table', 'field': 'interface.admin_status'}, - 'params': {} # no 'mapping' - } - ] - result = _generate_translate_filter(normalizers) - assert result is None - - def test_multiple_translate_normalizers_each_get_own_filter(self): - normalizers = [ - { - 'operation': 'translate', - 'target': {'scope': 'table', 'field': 'interface.admin_status'}, - 'params': {'mapping': {'1': 'UP', '2': 'DOWN'}} - }, - { - 'operation': 'translate', - 'target': {'scope': 'table', 'field': 'interface.oper_status'}, - 'params': {'mapping': {'1': 'UP', '2': 'DOWN', '3': 'TESTING'}} - } - ] - result = _generate_translate_filter(normalizers) - # Two normalizers → 2 comment + 2 translate = 4 total - assert len(result) == 4 - - -# =========================================================================== -# _apply_normalizers -# =========================================================================== - -class TestApplyNormalizers: - - def test_returns_empty_list_for_none(self): - assert _apply_normalizers(None) == [] - - def test_returns_empty_list_for_empty_list(self): - assert _apply_normalizers([]) == [] - - def test_skips_normalizer_missing_operation(self): - normalizers = [ - { - 'target': {'scope': 'get', 'field': 'some.field'}, - 'params': {'multiply_value': 2} - } - ] - result = _apply_normalizers(normalizers) - assert result == [] - - def test_skips_normalizer_missing_scope(self): - normalizers = [ - { - 'operation': 'multiply', - 'target': {'field': 'some.field'}, # no 'scope' - 'params': {'multiply_value': 2} - } - ] - result = _apply_normalizers(normalizers) - assert result == [] - - def test_applies_multiply_normalizer(self): - normalizers = [ - { - 'operation': 'multiply', - 'target': {'scope': 'get', 'field': 'system.cpu.total.norm.pct'}, - 'params': {'multiply_value': 0.01} - } - ] - result = _apply_normalizers(normalizers) - assert len(result) > 0 - plugin_types = [c['plugin'] for c in result] - assert 'ruby' in plugin_types - - def test_applies_ratio_normalizer(self): - normalizers = [ - { - 'operation': 'ratio', - 'target': {'scope': 'get'}, - 'params': { - 'value1_field': 'memory.used', - 'value2_field': 'memory.free', - } - } - ] - result = _apply_normalizers(normalizers) - assert len(result) > 0 - plugin_types = [c['plugin'] for c in result] - assert 'ruby' in plugin_types - - def test_applies_translate_normalizer(self): - normalizers = [ - { - 'operation': 'translate', - 'target': {'scope': 'table', 'field': 'interface.admin_status'}, - 'params': {'mapping': {'1': 'UP', '2': 'DOWN'}} - } - ] - result = _apply_normalizers(normalizers) - assert len(result) > 0 - plugin_types = [c['plugin'] for c in result] - assert 'translate' in plugin_types - - def test_applies_multiple_normalizer_types(self): - normalizers = [ - { - 'operation': 'multiply', - 'target': {'scope': 'get', 'field': 'metric.a'}, - 'params': {'multiply_value': 0.01} - }, - { - 'operation': 'translate', - 'target': {'scope': 'table', 'field': 'interface.admin_status'}, - 'params': {'mapping': {'1': 'UP'}} - } - ] - result = _apply_normalizers(normalizers) - plugin_types = [c['plugin'] for c in result] - assert 'ruby' in plugin_types - assert 'translate' in plugin_types - - def test_ignores_unknown_operation(self): - normalizers = [ - { - 'operation': 'unknown_op', - 'target': {'scope': 'get', 'field': 'some.field'}, - 'params': {} - } - ] - result = _apply_normalizers(normalizers) - assert result == [] - - def test_table_scope_multiply_is_handled(self): - normalizers = [ - { - 'operation': 'multiply', - 'target': {'scope': 'table', 'field': 'interface.speed'}, - 'params': {'multiply_value': 1000} - } - ] - result = _apply_normalizers(normalizers) - assert len(result) > 0 - - -# =========================================================================== -# Scope-qualified filter IDs (regression: duplicate Logstash plugin IDs) -# =========================================================================== - -class TestScopeQualifiedFilterIds: - """ - _generate_multiply_get_filter / _generate_ratio_get_filter run for BOTH - 'get' and 'table' scope normalizers, so their generated Logstash filter - component IDs must be scope-qualified. Otherwise a profile pairing a - get-scope op with a table-scope op of the same type emits two components - with an identical plugin ID, and Logstash rejects pipelines with duplicate - plugin IDs at compile time (the merged pipeline never builds). - """ - - # Mirrors cisco_system_metrics.json (get scope) combined with a Cisco - # OpenConfig interface-table profile (table scope). - GET_MULTIPLY = { - 'operation': 'multiply', - 'target': {'scope': 'get', 'field': 'system.cpu.total.norm.pct'}, - 'params': {'multiply_value': 0.01}, - } - TABLE_MULTIPLY = { - 'operation': 'multiply', - 'target': {'scope': 'table', 'field': 'interface.state.speed'}, - 'params': {'multiply_value': 1000000}, - } - GET_RATIO = { - 'operation': 'ratio', - 'target': {'scope': 'get'}, - 'params': { - 'value1_field': 'system.memory.actual.used.bytes', - 'value2_field': 'system.memory.actual.free.bytes', - 'total_output_field': 'system.memory.total.bytes', - 'ratio1_output_field': 'system.memory.actual.used.pct', - }, - } - TABLE_RATIO = { - 'operation': 'ratio', - 'target': {'scope': 'table'}, - 'params': { - 'value1_field': 'interface.state.counters.in_octets', - 'value2_field': 'interface.state.counters.out_octets', - 'total_output_field': 'interface.state.counters.total_octets', - }, - } - - @staticmethod - def _assert_unique(components): - ids = [c['id'] for c in components] - dupes = sorted({i for i in ids if ids.count(i) > 1}) - assert not dupes, f"duplicate filter component IDs: {dupes}" - - def test_multiply_get_and_table_scope_ids_are_scope_qualified(self): - components = _apply_normalizers([self.GET_MULTIPLY, self.TABLE_MULTIPLY]) - self._assert_unique(components) - ids = [c['id'] for c in components] - assert 'normalizer_multiply_get_1' in ids - assert 'normalizer_multiply_table_1' in ids - assert 'normalizer_multiply_get_comment_1' in ids - assert 'normalizer_multiply_table_comment_1' in ids - - def test_ratio_get_and_table_scope_ids_are_scope_qualified(self): - components = _apply_normalizers([self.GET_RATIO, self.TABLE_RATIO]) - self._assert_unique(components) - ids = [c['id'] for c in components] - assert 'normalizer_ratio_get_1' in ids - assert 'normalizer_ratio_table_1' in ids - assert 'normalizer_ratio_get_comment_1' in ids - assert 'normalizer_ratio_table_comment_1' in ids - - def test_combined_profile_all_component_ids_unique(self): - # The exact scenario that triggered the bug: cisco_system_metrics - # (get-scope CPU multiply + memory ratio) merged with a Cisco - # OpenConfig interface-table profile (table-scope multiply + ratio). - components = _apply_normalizers( - [self.GET_MULTIPLY, self.GET_RATIO, self.TABLE_MULTIPLY, self.TABLE_RATIO] - ) - self._assert_unique(components) - ids = [c['id'] for c in components] - for expected in ( - 'normalizer_multiply_get_1', - 'normalizer_multiply_table_1', - 'normalizer_ratio_get_1', - 'normalizer_ratio_table_1', - ): - assert expected in ids, f"missing generated component: {expected}" - - def test_single_get_scope_multiply_keeps_stable_id(self): - # IDs are always suffixed with _N by _next_id, even on the first call. - components = _apply_normalizers([self.GET_MULTIPLY]) - ids = [c['id'] for c in components] - assert ids == ['normalizer_multiply_get_comment_1', 'normalizer_multiply_get_1'] diff --git a/src/logstashui/SNMP/tests/test_snmp_pipeline_generator.py b/src/logstashui/SNMP/tests/test_snmp_pipeline_generator.py deleted file mode 100644 index 7ce51b5..0000000 --- a/src/logstashui/SNMP/tests/test_snmp_pipeline_generator.py +++ /dev/null @@ -1,789 +0,0 @@ -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. - -""" -Tests for SNMP.snmp_pipeline_generator — pure utility functions that require -no database or network access. -""" - -import pytest -from unittest.mock import MagicMock - -from SNMP.snmp_pipeline_generator import ( - _normalize_template_name, - _deduplicate_normalizers, - _uses_keystore, - _community_key_name, - _auth_pass_key_name, - _priv_pass_key_name, - _es_api_key_name, - _es_user_key_name, - _es_password_key_name, - _ref, - snmp_credential_keystore_entries, - snmp_credential_keystore_key_names, - es_connection_keystore_entries, - es_connection_keystore_key_names, - _ruby_table_nested_entry, - _ruby_row_rename_statements, - _ruby_row_value_expr, - _ruby_keep_when_statements, - _avg_var_name, - _ruby_avg_pre_loop, - _ruby_avg_in_loop, - _ruby_avg_post_loop, - _generate_table_split_filters, - _generate_snmp_error_cleanup_filter, -) - - -# =========================================================================== -# _normalize_template_name -# =========================================================================== - -class TestNormalizeTemplateName: - """Tests for the ES-index-safe name normalizer.""" - - def test_empty_string_returns_fallback(self): - assert _normalize_template_name('') == 'unknown_template' - - def test_none_returns_fallback(self): - assert _normalize_template_name(None) == 'unknown_template' - - def test_plain_name_lowercased(self): - assert _normalize_template_name('Cisco') == 'cisco' - - def test_spaces_replaced_with_underscore(self): - assert _normalize_template_name('My Template') == 'my_template' - - def test_multiple_spaces_collapsed(self): - assert _normalize_template_name('My Template') == 'my_template' - - def test_tabs_and_newlines_replaced(self): - result = _normalize_template_name('a\tb\nc') - assert result == 'a_b_c' - - def test_es_illegal_chars_replaced(self): - # Characters: * : / \ ? " < > | , # space - for char in ['*', ':', '/', '\\', '?', '"', '<', '>', '|', ',', '#']: - result = _normalize_template_name(f'name{char}value') - assert result == 'name_value', f"Failed for char {char!r}" - - def test_consecutive_underscores_collapsed(self): - assert _normalize_template_name('a__b') == 'a_b' - - def test_consecutive_hyphens_collapsed(self): - assert _normalize_template_name('a--b') == 'a_b' - - def test_mixed_separators_collapsed(self): - assert _normalize_template_name('a-_b') == 'a_b' - - def test_leading_hyphen_stripped(self): - assert _normalize_template_name('-name') == 'name' - - def test_leading_underscore_stripped(self): - assert _normalize_template_name('_name') == 'name' - - def test_leading_plus_stripped(self): - assert _normalize_template_name('+name') == 'name' - - def test_leading_dot_stripped(self): - assert _normalize_template_name('.name') == 'name' - - def test_already_clean_name_unchanged(self): - assert _normalize_template_name('dell_idrac') == 'dell_idrac' - - def test_alphanumeric_with_single_hyphen_preserved(self): - # Single hyphens are NOT replaced — only runs of 2+ hyphens/underscores are collapsed - assert _normalize_template_name('cisco-catalyst-9300') == 'cisco-catalyst-9300' - - def test_numeric_only_name(self): - assert _normalize_template_name('1234') == '1234' - - def test_surrounding_whitespace_stripped(self): - assert _normalize_template_name(' cisco ') == 'cisco' - - def test_all_forbidden_leading_chars_stripped(self): - # Multiple leading forbidden chars - result = _normalize_template_name('---___name') - assert result == 'name' - - def test_result_is_255_bytes_max(self): - # Build a name that would exceed 255 bytes when encoded - long_name = 'a' * 300 - result = _normalize_template_name(long_name) - assert len(result.encode('utf-8')) <= 255 - - def test_name_that_becomes_empty_after_stripping_returns_fallback(self): - # A name consisting only of forbidden leading chars - result = _normalize_template_name('---') - assert result == 'unknown_template' - - def test_real_world_dell_idrac(self): - assert _normalize_template_name('Dell iDRAC') == 'dell_idrac' - - def test_real_world_ubiquiti(self): - assert _normalize_template_name('Ubiquiti UniFi AP') == 'ubiquiti_unifi_ap' - - def test_real_world_brocade(self): - assert _normalize_template_name('Brocade FC Switch') == 'brocade_fc_switch' - - -# =========================================================================== -# _deduplicate_normalizers -# =========================================================================== - -class TestDeduplicateNormalizers: - - def _make_normalizer(self, operation, field, param_value): - return { - 'operation': operation, - 'target': {'scope': 'get', 'field': field}, - 'params': {'multiply_value': param_value} - } - - def test_returns_empty_for_none(self): - assert _deduplicate_normalizers(None) == [] - - def test_returns_empty_for_empty_list(self): - assert _deduplicate_normalizers([]) == [] - - def test_single_normalizer_returned_unchanged(self): - n = self._make_normalizer('multiply', 'metric.a', 0.01) - result = _deduplicate_normalizers([n]) - assert result == [n] - - def test_identical_normalizers_deduplicated(self): - n = self._make_normalizer('multiply', 'metric.a', 0.01) - result = _deduplicate_normalizers([n, n]) - assert len(result) == 1 - - def test_different_normalizers_both_kept(self): - n1 = self._make_normalizer('multiply', 'metric.a', 0.01) - n2 = self._make_normalizer('multiply', 'metric.b', 0.01) - result = _deduplicate_normalizers([n1, n2]) - assert len(result) == 2 - - def test_different_operations_both_kept(self): - n1 = { - 'operation': 'multiply', - 'target': {'scope': 'get', 'field': 'metric.a'}, - 'params': {'multiply_value': 0.01} - } - n2 = { - 'operation': 'ratio', - 'target': {'scope': 'get'}, - 'params': {'value1_field': 'metric.a', 'value2_field': 'metric.b'} - } - result = _deduplicate_normalizers([n1, n2]) - assert len(result) == 2 - - def test_duplicate_with_different_param_value_both_kept(self): - n1 = self._make_normalizer('multiply', 'metric.a', 0.01) - n2 = self._make_normalizer('multiply', 'metric.a', 100) - result = _deduplicate_normalizers([n1, n2]) - assert len(result) == 2 - - def test_three_duplicates_only_one_kept(self): - n = self._make_normalizer('multiply', 'metric.a', 0.01) - result = _deduplicate_normalizers([n, n, n]) - assert len(result) == 1 - - def test_mixed_duplicates_and_unique(self): - n1 = self._make_normalizer('multiply', 'metric.a', 0.01) - n2 = self._make_normalizer('multiply', 'metric.b', 0.01) - result = _deduplicate_normalizers([n1, n1, n2]) - assert len(result) == 2 - - def test_order_preserved_for_unique_normalizers(self): - n1 = self._make_normalizer('multiply', 'metric.a', 0.01) - n2 = self._make_normalizer('multiply', 'metric.b', 0.01) - n3 = self._make_normalizer('multiply', 'metric.c', 0.01) - result = _deduplicate_normalizers([n1, n2, n3]) - assert result[0] == n1 - assert result[1] == n2 - assert result[2] == n3 - - def test_first_occurrence_kept_on_duplicate(self): - n1 = self._make_normalizer('multiply', 'metric.a', 0.01) - n2 = self._make_normalizer('multiply', 'metric.a', 0.01) - result = _deduplicate_normalizers([n1, n2]) - assert result[0] is n1 - - -# =========================================================================== -# _uses_keystore -# =========================================================================== - -class TestUsesKeystore: - - def _network(self, deployment_mode='CENTRALIZED', credential_mode='KEYSTORE'): - n = MagicMock() - n.deployment_mode = deployment_mode - n.credential_mode = credential_mode - return n - - def test_agent_mode_always_uses_keystore(self): - assert _uses_keystore(self._network(deployment_mode='AGENT')) is True - - def test_agent_mode_ignores_credential_mode(self): - assert _uses_keystore(self._network(deployment_mode='AGENT', credential_mode='PLAINTEXT')) is True - - def test_centralized_keystore_mode_uses_keystore(self): - assert _uses_keystore(self._network(deployment_mode='CENTRALIZED', credential_mode='KEYSTORE')) is True - - def test_centralized_plaintext_mode_no_keystore(self): - assert _uses_keystore(self._network(deployment_mode='CENTRALIZED', credential_mode='PLAINTEXT')) is False - - def test_missing_deployment_mode_attr_defaults_to_centralized(self): - n = MagicMock(spec=[]) # no attributes at all → getattr returns default - n.credential_mode = 'KEYSTORE' - assert _uses_keystore(n) is True - - def test_missing_credential_mode_defaults_to_keystore(self): - n = MagicMock(spec=[]) - n.deployment_mode = 'CENTRALIZED' - assert _uses_keystore(n) is True - - -# =========================================================================== -# Keystore key-name helpers -# =========================================================================== - -class TestKeystoreKeyNameHelpers: - - def _cred(self, cred_id, version): - c = MagicMock() - c.id = cred_id - c.version = version - return c - - def _conn(self, conn_id): - c = MagicMock() - c.id = conn_id - return c - - # _community_key_name - def test_community_key_v1(self): - assert _community_key_name(self._cred(5, '1')) == 'snmp_5_v1' - - def test_community_key_v2c(self): - assert _community_key_name(self._cred(7, '2c')) == 'snmp_7_v2' - - # _auth_pass_key_name - def test_auth_pass_key_name(self): - assert _auth_pass_key_name(self._cred(3, '3')) == 'snmp_3_v3_auth' - - # _priv_pass_key_name - def test_priv_pass_key_name(self): - assert _priv_pass_key_name(self._cred(3, '3')) == 'snmp_3_v3_priv' - - # _es_api_key_name - def test_es_api_key_name(self): - assert _es_api_key_name(self._conn(10)) == 'snmp_es_10_api_key' - - # _es_user_key_name - def test_es_user_key_name(self): - assert _es_user_key_name(self._conn(10)) == 'snmp_es_10_user' - - # _es_password_key_name - def test_es_password_key_name(self): - assert _es_password_key_name(self._conn(10)) == 'snmp_es_10_password' - - # _ref - def test_ref_wraps_in_dollar_braces(self): - assert _ref('my_key') == '${my_key}' - - def test_ref_preserves_underscores(self): - assert _ref('snmp_5_v2') == '${snmp_5_v2}' - - -# =========================================================================== -# snmp_credential_keystore_entries -# =========================================================================== - -class TestSnmpCredentialKeystoreEntries: - - def _cred(self, cred_id, version, **kwargs): - c = MagicMock() - c.id = cred_id - c.version = version - for k, v in kwargs.items(): - setattr(c, k, v) - return c - - def test_none_credential_returns_empty(self): - assert snmp_credential_keystore_entries(None) == {} - - def test_v2c_community_included(self): - cred = self._cred(1, '2c') - cred.get_community.return_value = 'public' - entries = snmp_credential_keystore_entries(cred) - assert 'snmp_1_v2' in entries - assert entries['snmp_1_v2'] == 'public' - - def test_v1_community_uses_v1_suffix(self): - cred = self._cred(2, '1') - cred.get_community.return_value = 'private' - entries = snmp_credential_keystore_entries(cred) - assert 'snmp_2_v1' in entries - - def test_v2c_empty_community_not_included(self): - cred = self._cred(3, '2c') - cred.get_community.return_value = None - entries = snmp_credential_keystore_entries(cred) - assert entries == {} - - def test_v3_authpriv_includes_auth_and_priv(self): - cred = self._cred(4, '3', security_level='authPriv') - cred.get_auth_pass.return_value = 'authsecret' - cred.get_priv_pass.return_value = 'privsecret' - entries = snmp_credential_keystore_entries(cred) - assert 'snmp_4_v3_auth' in entries - assert 'snmp_4_v3_priv' in entries - assert entries['snmp_4_v3_auth'] == 'authsecret' - assert entries['snmp_4_v3_priv'] == 'privsecret' - - def test_v3_authnopriv_includes_only_auth(self): - cred = self._cred(5, '3', security_level='authNoPriv') - cred.get_auth_pass.return_value = 'authsecret' - entries = snmp_credential_keystore_entries(cred) - assert 'snmp_5_v3_auth' in entries - assert 'snmp_5_v3_priv' not in entries - - def test_v3_noauthnopriv_returns_empty(self): - cred = self._cred(6, '3', security_level='noAuthNoPriv') - entries = snmp_credential_keystore_entries(cred) - assert entries == {} - - -# =========================================================================== -# snmp_credential_keystore_key_names -# =========================================================================== - -class TestSnmpCredentialKeystoreKeyNames: - - def _cred(self, cred_id, version, **kwargs): - c = MagicMock() - c.id = cred_id - c.version = version - for k, v in kwargs.items(): - setattr(c, k, v) - return c - - def test_none_returns_empty_set(self): - assert snmp_credential_keystore_key_names(None) == set() - - def test_v2c_with_community_returns_one_key(self): - cred = self._cred(1, '2c', community='public') - names = snmp_credential_keystore_key_names(cred) - assert names == {'snmp_1_v2'} - - def test_v2c_empty_community_returns_empty(self): - cred = self._cred(2, '2c', community='') - names = snmp_credential_keystore_key_names(cred) - assert names == set() - - def test_v3_authpriv_returns_auth_and_priv_keys(self): - cred = self._cred(3, '3', security_level='authPriv', auth_pass='x', priv_pass='y') - names = snmp_credential_keystore_key_names(cred) - assert 'snmp_3_v3_auth' in names - assert 'snmp_3_v3_priv' in names - - def test_v3_authnopriv_returns_only_auth_key(self): - cred = self._cred(4, '3', security_level='authNoPriv', auth_pass='x', priv_pass='') - names = snmp_credential_keystore_key_names(cred) - assert 'snmp_4_v3_auth' in names - assert 'snmp_4_v3_priv' not in names - - -# =========================================================================== -# es_connection_keystore_entries -# =========================================================================== - -class TestEsConnectionKeystoreEntries: - - def _conn(self, conn_id, **kwargs): - c = MagicMock() - c.id = conn_id - for k, v in kwargs.items(): - setattr(c, k, v) - return c - - def test_none_returns_empty(self): - assert es_connection_keystore_entries(None) == {} - - def test_api_key_preferred(self): - conn = self._conn(1, api_key='encrypted_key', username='user', password='pass') - conn.get_api_key.return_value = 'myapikey' - entries = es_connection_keystore_entries(conn) - assert 'snmp_es_1_api_key' in entries - assert entries['snmp_es_1_api_key'] == 'myapikey' - assert 'snmp_es_1_user' not in entries - - def test_username_password_used_when_no_api_key(self): - conn = self._conn(2, api_key=None, username='elastic', password='encrypted_pass') - conn.get_password.return_value = 'secret' - entries = es_connection_keystore_entries(conn) - assert 'snmp_es_2_user' in entries - assert 'snmp_es_2_password' in entries - assert entries['snmp_es_2_user'] == 'elastic' - assert entries['snmp_es_2_password'] == 'secret' - - def test_no_credentials_returns_empty(self): - conn = self._conn(3, api_key=None, username='', password='') - entries = es_connection_keystore_entries(conn) - assert entries == {} - - -# =========================================================================== -# es_connection_keystore_key_names -# =========================================================================== - -class TestEsConnectionKeystoreKeyNames: - - def _conn(self, conn_id, **kwargs): - c = MagicMock() - c.id = conn_id - for k, v in kwargs.items(): - setattr(c, k, v) - return c - - def test_none_returns_empty_set(self): - assert es_connection_keystore_key_names(None) == set() - - def test_api_key_returns_api_key_name(self): - conn = self._conn(1, api_key='something', username='user', password='pass') - names = es_connection_keystore_key_names(conn) - assert names == {'snmp_es_1_api_key'} - - def test_username_password_returns_user_and_password_names(self): - conn = self._conn(2, api_key=None, username='elastic', password='secret') - names = es_connection_keystore_key_names(conn) - assert 'snmp_es_2_user' in names - assert 'snmp_es_2_password' in names - - def test_no_credentials_returns_empty_set(self): - conn = self._conn(3, api_key=None, username='', password='') - names = es_connection_keystore_key_names(conn) - assert names == set() - - -# =========================================================================== -# _ruby_table_nested_entry -# =========================================================================== - -class TestRubyTableNestedEntry: - - def test_flat_table_name(self): - result = _ruby_table_nested_entry('ifTable', 'row') - assert result == '"ifTable" => row' - - def test_dotted_table_name_two_levels(self): - result = _ruby_table_nested_entry('component.fan', 'row') - assert result == '"component" => { "fan" => row }' - - def test_dotted_table_name_three_levels(self): - result = _ruby_table_nested_entry('a.b.c', 'val') - assert result == '"a" => { "b" => { "c" => val } }' - - def test_value_expr_is_preserved(self): - result = _ruby_table_nested_entry('ifTable', 'event.get("[myfield]")') - assert 'event.get("[myfield]")' in result - - -# =========================================================================== -# _ruby_row_rename_statements -# =========================================================================== - -class TestRubyRowRenameStatements: - - def test_empty_columns_returns_empty_string(self): - assert _ruby_row_rename_statements({}) == '' - - def test_flat_column_rename(self): - result = _ruby_row_rename_statements({'col_a': 'oid1'}) - assert 'row["col_a"] = row.delete("oid1")' in result - - def test_dotted_column_initializes_parent(self): - result = _ruby_row_rename_statements({'component.speed': 'oid2'}) - assert 'row["component"] ||= {}' in result - assert 'row["component"]["speed"] = row.delete("oid2")' in result - - def test_two_columns_sharing_parent_initializes_parent_once(self): - result = _ruby_row_rename_statements({ - 'iface.in_octets': 'oid1', - 'iface.out_octets': 'oid2', - }) - assert result.count('row["iface"] ||= {}') == 1 - - def test_multiple_flat_columns(self): - result = _ruby_row_rename_statements({'a': '1', 'b': '2'}) - assert 'row["a"] = row.delete("1")' in result - assert 'row["b"] = row.delete("2")' in result - - -# =========================================================================== -# _avg_var_name -# =========================================================================== - -class TestAvgVarName: - - def _normalizer(self, output_field='', target_field='unknown'): - return { - 'params': {'output_field': output_field}, - 'target': {'field': target_field}, - } - - def test_output_field_used_when_set(self): - n = self._normalizer(output_field='interface.avg_in_octets') - assert _avg_var_name(n) == 'avg_interface_avg_in_octets' - - def test_dots_replaced_with_underscores(self): - n = self._normalizer(output_field='a.b.c') - assert _avg_var_name(n) == 'avg_a_b_c' - - def test_hyphens_replaced_with_underscores(self): - n = self._normalizer(output_field='some-field') - assert _avg_var_name(n) == 'avg_some_field' - - def test_fallback_to_target_field_when_no_output_field(self): - n = self._normalizer(output_field='', target_field='interface.load') - result = _avg_var_name(n) - assert result.startswith('avg_') - assert 'interface' in result - - -# =========================================================================== -# _ruby_avg_pre_loop / _ruby_avg_in_loop / _ruby_avg_post_loop -# =========================================================================== - -class TestRubyAvgLoops: - - def _avg_normalizer(self, output_field, target_field='interface.in_octets'): - return { - 'operation': 'average', - 'target': {'scope': 'table', 'field': target_field}, - 'params': {'output_field': output_field}, - } - - def test_pre_loop_empty_returns_empty_string(self): - assert _ruby_avg_pre_loop([]) == '' - - def test_pre_loop_declares_sum_and_count(self): - n = self._avg_normalizer('interface.avg_in_octets', 'interface.in_octets') - result = _ruby_avg_pre_loop([n]) - assert '_sum = 0.0' in result - assert '_count = 0' in result - - def test_pre_loop_multiple_normalizers(self): - n1 = self._avg_normalizer('interface.avg_in', 'interface.in_octets') - n2 = self._avg_normalizer('interface.avg_out', 'interface.out_octets') - result = _ruby_avg_pre_loop([n1, n2]) - assert result.count('_sum = 0.0') == 2 - - def test_in_loop_empty_returns_empty_string(self): - assert _ruby_avg_in_loop([], 'interface') == '' - - def test_in_loop_generates_accumulation_statements(self): - n = self._avg_normalizer('interface.avg_in_octets', 'interface.in_octets') - result = _ruby_avg_in_loop([n], 'interface') - assert 'Float(_avg_v)' in result - assert '_count +=' in result - assert 'row["in_octets"]' in result - assert 'is_a?(Numeric)' not in result - - def test_in_loop_strips_dotted_table_prefix(self): - n = self._avg_normalizer('system.cpu.total.norm.pct', 'component.cpu.load_pct') - result = _ruby_avg_in_loop([n], 'component.cpu') - assert 'row["load_pct"]' in result - - def test_post_loop_empty_returns_empty_string(self): - assert _ruby_avg_post_loop([]) == '' - - def test_post_loop_generates_event_set(self): - n = self._avg_normalizer('interface.avg_in_octets', 'interface.in_octets') - result = _ruby_avg_post_loop([n]) - assert 'event.set(' in result - assert '_count > 0' in result - - def test_post_loop_skips_normalizer_without_output_field(self): - n = {'operation': 'average', 'target': {}, 'params': {'output_field': ''}} - assert _ruby_avg_post_loop([n]) == '' - - def test_post_loop_includes_multiply_when_set(self): - n = self._avg_normalizer('interface.avg_in_octets', 'interface.in_octets') - n['params']['multiply_value'] = 8 - result = _ruby_avg_post_loop([n]) - assert '* 8' in result - - -class TestRubyRowValueExpr: - - def test_single_level_column(self): - assert _ruby_row_value_expr('component.cpu', 'component.cpu.load_pct') == 'row["load_pct"]' - - def test_nested_column(self): - assert _ruby_row_value_expr( - 'system.filesystem', 'system.filesystem.total.bytes' - ) == 'row.dig("total", "bytes")' - - -class TestHostSystemMetricsSplit: - - def test_cpu_average_writes_ecs_field_on_metrics_doc(self): - oid_mappings = { - 'table': { - 'component.cpu': { - 'columns': {'load_pct': '1.3.6.1.2.1.25.3.3.1.2'} - }, - } - } - averages = [{ - 'operation': 'average', - 'target': { - 'scope': 'table', - 'table': 'component.cpu', - 'field': 'component.cpu.load_pct', - }, - 'params': { - 'output_field': 'system.cpu.total.norm.pct', - 'multiply_value': 0.01, - }, - }] - filters = _generate_table_split_filters(oid_mappings, averages) - cpu = filters[0]['config']['code'] - assert '[system][cpu][total][norm][pct]' in cpu - assert 'Float(_avg_v)' in cpu - assert '* 0.01' in cpu - - def test_official_profile_averages_cores_to_ecs_cpu_field(self): - import json - from pathlib import Path - - path = ( - Path(__file__).resolve().parents[1] - / 'data' / 'official_profiles' / 'generic_host_system_metrics.json' - ) - profile = json.loads(path.read_text(encoding='utf-8')) - average = next(n for n in profile['normalizers'] if n['operation'] == 'average') - assert average['params']['output_field'] == 'system.cpu.total.norm.pct' - assert average['params']['multiply_value'] == 0.01 - assert 'promote_output_field' not in average.get('params', {}) - ratio = next(n for n in profile['normalizers'] if n['operation'] == 'ratio') - assert 'promote_output_field' not in ratio['params'] - assert 'promote_when_type' not in ratio['params'] - - -class TestRubyKeepWhenStatements: - - def test_empty_table_returns_empty_string(self): - assert _ruby_keep_when_statements({}) == '' - assert _ruby_keep_when_statements(None) == '' - - def test_emits_delete_and_include_check(self): - result = _ruby_keep_when_statements({ - 'keep_when': {'column': 'type', 'equals': ['10']}, - }) - assert 'row.delete("type")' in result - assert '["10"].include?(_keep_v.to_s)' in result - - def test_rejects_unsafe_column_names(self): - result = _ruby_keep_when_statements({ - 'keep_when': {'column': 'type"; exit', 'equals': ['10']}, - }) - assert result == '' - - def test_table_split_injects_keep_when_before_event_emit(self): - oid_mappings = { - 'table': { - 'component.fan': { - 'columns': { - 'description': '1.3.6.1.2.1.47.1.1.1.1.2', - 'type': '1.3.6.1.2.1.99.1.1.1.1', - }, - 'keep_when': {'column': 'type', 'equals': ['10']}, - } - } - } - filters = _generate_table_split_filters(oid_mappings) - code = filters[0]['config']['code'] - assert 'row.delete("type")' in code - assert '["10"].include?(_keep_v.to_s)' in code - assert code.index('row.delete("type")') < code.index('LogStash::Event.new') - - def test_tables_without_keep_when_are_unchanged(self): - oid_mappings = { - 'table': { - 'component.cpu': { - 'columns': {'load_pct': '1.3.6.1.2.1.25.3.3.1.2'} - } - } - } - filters = _generate_table_split_filters(oid_mappings) - code = filters[0]['config']['code'] - assert 'row.delete("type")' not in code - assert '_keep_v' not in code - - -class TestPaloaltoComponentsProfile: - - def _profile(self): - import json - from pathlib import Path - path = ( - Path(__file__).resolve().parents[1] - / 'data' / 'official_profiles' / 'paloalto_components.json' - ) - return json.loads(path.read_text(encoding='utf-8')) - - def test_maps_entity_sensor_onto_cisco_schema(self): - profile = self._profile() - fans = profile['table']['component.fan'] - sensors = profile['table']['component.sensor'] - assert fans['columns']['description'] == '1.3.6.1.2.1.47.1.1.1.1.2' - assert fans['columns']['state'] == '1.3.6.1.2.1.99.1.1.1.5' - assert fans['columns']['rpm'] == '1.3.6.1.2.1.99.1.1.1.4' - assert fans['keep_when'] == {'column': 'type', 'equals': ['10']} - assert sensors['columns']['description'] == '1.3.6.1.2.1.47.1.1.1.1.2' - assert sensors['columns']['temp.celsius'] == '1.3.6.1.2.1.99.1.1.1.4' - assert sensors['columns']['state'] == '1.3.6.1.2.1.99.1.1.1.5' - assert sensors['keep_when'] == {'column': 'type', 'equals': ['8']} - assert 'temp.threshold' not in sensors['columns'] - assert 'temp.last_shutdown' not in sensors['columns'] - - def test_firewall_template_uses_components_not_generic_entity_sensor(self): - import json - from pathlib import Path - path = ( - Path(__file__).resolve().parents[1] - / 'data' / 'official_device_templates' / 'palo_alto_firewall.json' - ) - template = json.loads(path.read_text(encoding='utf-8')) - assert 'paloalto_components' in template['profiles'] - assert 'generic_entity_sensor' not in template['profiles'] - - -# =========================================================================== -# _generate_snmp_error_cleanup_filter -# =========================================================================== - -class TestGenerateSnmpErrorCleanupFilter: - - def test_returns_a_dict(self): - result = _generate_snmp_error_cleanup_filter() - assert isinstance(result, dict) - - def test_plugin_is_ruby(self): - result = _generate_snmp_error_cleanup_filter() - assert result['plugin'] == 'ruby' - - def test_has_id(self): - result = _generate_snmp_error_cleanup_filter() - assert 'id' in result and result['id'] - - def test_code_contains_error_cleanup_logic(self): - result = _generate_snmp_error_cleanup_filter() - code = result['config']['code'] - assert 'error' in code.lower() diff --git a/src/logstashui/SNMP/tests/test_snmp_test.py b/src/logstashui/SNMP/tests/test_snmp_test.py deleted file mode 100644 index f9c32c1..0000000 --- a/src/logstashui/SNMP/tests/test_snmp_test.py +++ /dev/null @@ -1,920 +0,0 @@ -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. - -""" -Tests for SNMP.snmp_test — format helpers, auth data creation, and the -RunSNMPTest / RunSNMPWalk view endpoints (SNMP network I/O mocked). -""" - -import json -import socket -import pytest -from unittest.mock import patch, MagicMock -from django.test import Client -from django.contrib.auth.models import User - -from SNMP.snmp_test import ( - _create_auth_data, - _device_poll_address, - _device_response_data, - _format_snmp_value, - _load_profile_data, - _merge_profile_oids, - _resolve_device_poll_address, -) -from SNMP.models import Device, DeviceTemplate, Profile, Credential, Network -from PipelineManager.models import Connection -from Management.models import UserProfile - - -# =========================================================================== -# Fixtures -# =========================================================================== - -@pytest.fixture -def admin_user(db): - user = User.objects.create_user( - username='snmp_test_admin', - password='testpass123', - email='snmp_test_admin@example.com' - ) - profile, created = UserProfile.objects.get_or_create(user=user, defaults={'role': 'admin'}) - if not created: - profile.role = 'admin' - profile.save() - return user - - -@pytest.fixture -def authenticated_client(admin_user): - client = Client() - client.force_login(admin_user) - return client - - -@pytest.fixture -def test_connection(db): - return Connection.objects.create( - name='SNMP Test Connection', - connection_type='CENTRALIZED', - host='https://localhost:9200', - username='elastic', - password='changeme' - ) - - -@pytest.fixture -def test_credential_v2c(db): - return Credential.objects.create( - name='snmp_test_cred_v2c', - version='2c', - community='public', - description='Test v2c credential for snmp_test tests' - ) - - -@pytest.fixture -def test_credential_v3_auth_priv(db): - return Credential.objects.create( - name='snmp_test_cred_v3', - version='3', - security_name='testuser', - security_level='authPriv', - auth_protocol='sha', - auth_pass='authpass123', - priv_protocol='aes', - priv_pass='privpass123', - ) - - -@pytest.fixture -def test_credential_v3_auth_no_priv(db): - return Credential.objects.create( - name='snmp_test_cred_v3_anp', - version='3', - security_name='testuser2', - security_level='authNoPriv', - auth_protocol='md5', - auth_pass='authpass123', - ) - - -@pytest.fixture -def test_credential_v3_no_auth(db): - return Credential.objects.create( - name='snmp_test_cred_v3_noanp', - version='3', - security_name='testuser3', - security_level='noAuthNoPriv', - ) - - -@pytest.fixture -def test_network(db, test_connection, test_credential_v2c): - return Network.objects.create( - name='SNMP Test Network', - network_range='10.0.0.0/24', - connection=test_connection, - discovery_credential=test_credential_v2c, - interval=30 - ) - - -@pytest.fixture -def test_custom_profile(db): - return Profile.objects.create( - name='snmp_test_custom_profile', - description='Custom profile for snmp_test tests', - vendor='Generic', - profile_data={ - 'get': {'sysDescr': '1.3.6.1.2.1.1.1.0'}, - 'walk': {}, - 'table': {} - } - ) - - -@pytest.fixture -def test_device_template(db, test_custom_profile): - template = DeviceTemplate.objects.create( - name='snmp_test_template', - description='Test template', - vendor='Generic', - ) - template.profiles.add(test_custom_profile) - return template - - -@pytest.fixture -def test_device(db, test_network, test_credential_v2c, test_device_template): - return Device.objects.create( - name='snmp_test_device', - ip_address='10.0.0.1', - port=161, - retries=1, - timeout=500, - credential=test_credential_v2c, - network=test_network, - device_template=test_device_template, - ) - - -# =========================================================================== -# _format_snmp_value — pure function -# =========================================================================== - -class TestFormatSnmpValue: - - def test_printable_string_returned_as_is(self): - assert _format_snmp_value('Hello World') == 'Hello World' - - def test_empty_string_returned_as_is(self): - assert _format_snmp_value('') == '' - - def test_numeric_string_returned_as_is(self): - assert _format_snmp_value('12345') == '12345' - - def test_mostly_printable_string_returned_as_is(self): - # All printable ASCII - value = 'Linux router 2.6.32' - assert _format_snmp_value(value) == value - - def test_six_byte_binary_value_formatted_as_mac(self): - # Simulate a 6-character string with non-printable bytes → MAC-like hex - binary = '\x00\x11\x22\x33\x44\x55' - result = _format_snmp_value(binary) - # Should be hex-formatted - assert ':' in result - - def test_four_byte_binary_value_formatted_as_hex(self): - binary = '\xc0\xa8\x01\x01' # 192.168.1.1 as binary - result = _format_snmp_value(binary) - assert ':' in result - - -# =========================================================================== -# _load_profile_data -# =========================================================================== - -class TestLoadProfileData: - - def test_returns_profile_data_for_custom_profile(self, test_custom_profile): - data = _load_profile_data(test_custom_profile) - assert data == test_custom_profile.profile_data - - def test_official_placeholder_loads_from_file(self, settings, tmp_path): - import os, json as jsonlib - # Point BASE_DIR at tmp_path and create the official profile file - settings.BASE_DIR = str(tmp_path) - profile_dir = tmp_path / 'SNMP' / 'data' / 'official_profiles' - profile_dir.mkdir(parents=True) - - profile_content = { - 'get': {'sysDescr': '1.3.6.1.2.1.1.1.0'}, - 'walk': {}, - 'table': {} - } - profile_file = profile_dir / 'test_official.json' - profile_file.write_text(jsonlib.dumps(profile_content)) - - official_profile = Profile( - name='test_official.json', - profile_data={'is_official_placeholder': True}, - vendor='Generic' - ) - data = _load_profile_data(official_profile) - assert data['get']['sysDescr'] == '1.3.6.1.2.1.1.1.0' - - def test_official_placeholder_missing_file_returns_empty(self, settings, tmp_path): - settings.BASE_DIR = str(tmp_path) - (tmp_path / 'SNMP' / 'data' / 'official_profiles').mkdir(parents=True) - - official_profile = Profile( - name='nonexistent.json', - profile_data={'is_official_placeholder': True}, - vendor='Generic' - ) - data = _load_profile_data(official_profile) - assert data == {'get': {}, 'walk': {}, 'table': {}} - - -# =========================================================================== -# _merge_profile_oids -# =========================================================================== - -class TestMergeProfileOids: - - def test_empty_profiles_returns_empty_structure(self): - result = _merge_profile_oids([]) - assert result == {'get': {}, 'walk': {}, 'table': {}} - - def test_single_profile_merged_correctly(self, test_custom_profile): - result = _merge_profile_oids([test_custom_profile]) - assert 'sysDescr' in result['get'] - - def test_multiple_profiles_oids_merged(self, db): - p1 = Profile.objects.create( - name='merge_test_p1', - vendor='Generic', - profile_data={'get': {'oid_a': '1.3.6.1.2.1.1.1.0'}, 'walk': {}, 'table': {}} - ) - p2 = Profile.objects.create( - name='merge_test_p2', - vendor='Generic', - profile_data={'get': {'oid_b': '1.3.6.1.2.1.1.2.0'}, 'walk': {}, 'table': {}} - ) - result = _merge_profile_oids([p1, p2]) - assert 'oid_a' in result['get'] - assert 'oid_b' in result['get'] - - def test_later_profile_overwrites_duplicate_oid_key(self, db): - p1 = Profile.objects.create( - name='merge_test_dup1', - vendor='Generic', - profile_data={'get': {'oid_x': '1.3.6.1.2.1.1.1.0'}, 'walk': {}, 'table': {}} - ) - p2 = Profile.objects.create( - name='merge_test_dup2', - vendor='Generic', - profile_data={'get': {'oid_x': '1.3.6.1.2.1.1.2.0'}, 'walk': {}, 'table': {}} - ) - result = _merge_profile_oids([p1, p2]) - # p2's value wins - assert result['get']['oid_x'] == '1.3.6.1.2.1.1.2.0' - - -# =========================================================================== -# Device poll address -# =========================================================================== - -class TestDevicePollAddress: - - def test_hostname_is_preferred_over_ip_address(self): - device = MagicMock( - id=1, - name='switch-1', - hostname='switch-1.example.com', - ip_address='10.0.0.1', - port=161, - ) - - assert _device_poll_address(device) == 'switch-1.example.com' - assert _device_response_data(device)['address'] == 'switch-1.example.com' - - def test_ip_address_is_used_when_hostname_is_empty(self): - device = MagicMock(hostname=None, ip_address='10.0.0.1') - - assert _device_poll_address(device) == '10.0.0.1' - - @patch('SNMP.snmp_test.socket.getaddrinfo') - def test_resolvable_hostname_is_used(self, mock_getaddrinfo): - mock_getaddrinfo.return_value = [(socket.AF_INET, socket.SOCK_DGRAM, 17, '', ('10.0.0.1', 161))] - device = MagicMock( - hostname='switch-1.example.com', - ip_address='10.0.0.1', - port=161, - ) - - address, warning = _resolve_device_poll_address(device) - - assert address == 'switch-1.example.com' - assert warning is None - - @patch('SNMP.snmp_test.socket.getaddrinfo', side_effect=socket.gaierror) - def test_unresolvable_hostname_falls_back_to_ip(self, mock_getaddrinfo): - device = MagicMock( - hostname='switch-1.example.com', - ip_address='10.0.0.1', - port=161, - ) - - address, warning = _resolve_device_poll_address(device) - - assert address == '10.0.0.1' - assert 'cannot resolve' in warning - assert 'Falling back' in warning - - @patch('SNMP.snmp_test.socket.getaddrinfo', side_effect=socket.gaierror) - def test_unresolvable_hostname_without_ip_raises_clear_error(self, mock_getaddrinfo): - device = MagicMock( - hostname='switch-1.example.com', - ip_address=None, - port=161, - ) - - with pytest.raises(ValueError, match='No fallback IP address'): - _resolve_device_poll_address(device) - - -# =========================================================================== -# _create_auth_data -# =========================================================================== - -class TestCreateAuthData: - - def test_v2c_creates_community_data(self, test_credential_v2c): - from pysnmp.hlapi.v3arch.asyncio import CommunityData - auth = _create_auth_data(test_credential_v2c) - assert isinstance(auth, CommunityData) - - def test_v3_no_auth_no_priv_creates_usm(self, test_credential_v3_no_auth): - from pysnmp.hlapi.v3arch.asyncio import UsmUserData - auth = _create_auth_data(test_credential_v3_no_auth) - assert isinstance(auth, UsmUserData) - - def test_v3_auth_no_priv_creates_usm_with_auth(self, test_credential_v3_auth_no_priv): - from pysnmp.hlapi.v3arch.asyncio import UsmUserData - auth = _create_auth_data(test_credential_v3_auth_no_priv) - assert isinstance(auth, UsmUserData) - - def test_v3_auth_priv_creates_usm_with_auth_and_priv(self, test_credential_v3_auth_priv): - from pysnmp.hlapi.v3arch.asyncio import UsmUserData - auth = _create_auth_data(test_credential_v3_auth_priv) - assert isinstance(auth, UsmUserData) - - def test_v2c_missing_community_raises_value_error(self): - # Use an unsaved model instance to bypass model-level validation - # and test the _create_auth_data logic directly - cred = Credential( - name='snmp_test_cred_nocommunity', - version='2c', - community='', # explicitly empty, overriding default='public' - ) - with pytest.raises(ValueError, match='no community string'): - _create_auth_data(cred) - - def test_v3_auth_no_priv_missing_auth_protocol_raises(self): - # Use unsaved instance: model validates auth_protocol at save() time, - # but _create_auth_data validates it independently at runtime - cred = Credential( - version='3', - security_name='user', - security_level='authNoPriv', - auth_protocol='', # no protocol - auth_pass='', - ) - with pytest.raises(ValueError, match='auth protocol'): - _create_auth_data(cred) - - def test_v3_auth_priv_missing_priv_protocol_raises(self): - # Use unsaved instance to bypass model validation - cred = Credential( - version='3', - security_name='user', - security_level='authPriv', - auth_protocol='sha', - auth_pass='authpass123', - priv_protocol='', # no priv protocol - priv_pass='', - ) - with pytest.raises(ValueError, match='privacy protocol'): - _create_auth_data(cred) - - def test_unknown_version_raises_value_error(self): - cred = Credential( - name='bad_version_cred', - version='9', - security_name='user', - ) - with pytest.raises(ValueError, match='Unknown SNMP version'): - _create_auth_data(cred) - - -# =========================================================================== -# RunSNMPTest view -# =========================================================================== - -@pytest.mark.django_db -class TestRunSNMPTestView: - - def test_missing_device_id_returns_400(self, authenticated_client): - response = authenticated_client.post( - '/SNMP/RunSNMPTest/', - data=json.dumps({}), - content_type='application/json' - ) - assert response.status_code == 400 - data = json.loads(response.content) - assert data['success'] is False - assert 'device_id' in data['error'] - - def test_device_not_found_returns_404(self, authenticated_client): - response = authenticated_client.post( - '/SNMP/RunSNMPTest/', - data=json.dumps({'device_id': 999999}), - content_type='application/json' - ) - assert response.status_code == 404 - data = json.loads(response.content) - assert data['success'] is False - - def test_device_without_credential_returns_400(self, authenticated_client, test_network, test_device_template, db): - # Create device with no credential by bypassing model validation - # We'll use a credential initially then remove it - cred = Credential.objects.create( - name='temp_cred_for_removal', - version='2c', - community='public' - ) - device = Device.objects.create( - name='device_no_cred', - ip_address='10.0.0.99', - port=161, - retries=1, - timeout=500, - credential=cred, - network=test_network, - device_template=test_device_template, - ) - # Remove credential by setting it to None directly in DB - Device.objects.filter(pk=device.pk).update(credential=None) - - response = authenticated_client.post( - '/SNMP/RunSNMPTest/', - data=json.dumps({'device_id': device.pk}), - content_type='application/json' - ) - assert response.status_code == 400 - data = json.loads(response.content) - assert data['success'] is False - assert 'credential' in data['error'].lower() - - def test_template_not_found_returns_404(self, authenticated_client, test_device): - response = authenticated_client.post( - '/SNMP/RunSNMPTest/', - data=json.dumps({'device_id': test_device.pk, 'template_id': 999999}), - content_type='application/json' - ) - assert response.status_code == 404 - data = json.loads(response.content) - assert data['success'] is False - - def test_template_with_no_profiles_returns_400(self, authenticated_client, test_network, - test_credential_v2c, db): - empty_template = DeviceTemplate.objects.create( - name='snmp_test_empty_template', - vendor='Generic', - ) - device = Device.objects.create( - name='device_empty_template', - ip_address='10.0.0.50', - port=161, - retries=1, - timeout=500, - credential=test_credential_v2c, - network=test_network, - device_template=empty_template, - ) - - response = authenticated_client.post( - '/SNMP/RunSNMPTest/', - data=json.dumps({'device_id': device.pk, 'template_id': empty_template.pk}), - content_type='application/json' - ) - assert response.status_code == 400 - data = json.loads(response.content) - assert data['success'] is False - assert 'profiles' in data['error'].lower() - - @patch('SNMP.snmp_test._perform_snmp_get') - @patch('SNMP.snmp_test._perform_snmp_walk') - @patch('SNMP.snmp_test._perform_snmp_table') - def test_successful_snmp_test_returns_200_with_results( - self, mock_table, mock_walk, mock_get, - authenticated_client, test_device - ): - mock_get.return_value = {'sysDescr': 'Linux router'} - mock_walk.return_value = {} - mock_table.return_value = {} - - response = authenticated_client.post( - '/SNMP/RunSNMPTest/', - data=json.dumps({'device_id': test_device.pk}), - content_type='application/json' - ) - assert response.status_code == 200 - data = json.loads(response.content) - assert data['success'] is True - assert 'results' in data - assert data['device']['id'] == test_device.pk - assert data['device']['address'] == test_device.ip_address - assert data['template']['name'] == test_device.device_template.name - - @patch('SNMP.snmp_test._perform_snmp_get') - @patch('SNMP.snmp_test._perform_snmp_walk') - @patch('SNMP.snmp_test._perform_snmp_table') - def test_hostname_device_response_uses_hostname_as_poll_address( - self, mock_table, mock_walk, mock_get, - authenticated_client, test_device - ): - test_device.hostname = 'switch-1.example.com' - test_device.save(update_fields=['hostname']) - mock_get.return_value = {'sysDescr': 'Network switch'} - mock_walk.return_value = {} - mock_table.return_value = {} - - with patch( - 'SNMP.snmp_test._resolve_device_poll_address', - return_value=('switch-1.example.com', None), - ): - response = authenticated_client.post( - '/SNMP/RunSNMPTest/', - data=json.dumps({'device_id': test_device.pk}), - content_type='application/json' - ) - - assert response.status_code == 200 - data = json.loads(response.content) - assert data['success'] is True - assert data['device']['address'] == 'switch-1.example.com' - assert data['device']['hostname'] == 'switch-1.example.com' - assert data['device']['ip_address'] == '10.0.0.1' - - @patch('SNMP.snmp_test._perform_snmp_get') - @patch('SNMP.snmp_test._perform_snmp_walk') - @patch('SNMP.snmp_test._perform_snmp_table') - def test_unresolvable_hostname_falls_back_to_ip_and_returns_warning( - self, mock_table, mock_walk, mock_get, - authenticated_client, test_device - ): - test_device.hostname = 'switch-1.example.com' - test_device.save(update_fields=['hostname']) - mock_get.return_value = {'sysDescr': 'Network switch'} - mock_walk.return_value = {} - mock_table.return_value = {} - - warning = ( - "The machine running LogstashUI cannot resolve hostname " - "'switch-1.example.com'. Falling back to IP address 10.0.0.1." - ) - with patch( - 'SNMP.snmp_test._resolve_device_poll_address', - return_value=('10.0.0.1', warning), - ): - response = authenticated_client.post( - '/SNMP/RunSNMPTest/', - data=json.dumps({'device_id': test_device.pk}), - content_type='application/json' - ) - - data = json.loads(response.content) - assert data['success'] is True - assert data['device']['address'] == '10.0.0.1' - assert data['address_warning'] == warning - mock_get.assert_called_once_with( - test_device, test_device.credential, - test_device.device_template.profiles.first().profile_data['get'], - '10.0.0.1' - ) - - @patch('SNMP.snmp_test.socket.getaddrinfo', side_effect=socket.gaierror) - def test_unresolvable_hostname_only_device_returns_clear_error( - self, mock_getaddrinfo, authenticated_client, test_device - ): - test_device.hostname = 'switch-1.example.com' - test_device.ip_address = None - test_device.save(update_fields=['hostname', 'ip_address']) - - response = authenticated_client.post( - '/SNMP/RunSNMPTest/', - data=json.dumps({'device_id': test_device.pk}), - content_type='application/json' - ) - - assert response.status_code == 400 - data = json.loads(response.content) - assert data['success'] is False - assert data['error'] == ( - "The machine running LogstashUI cannot resolve hostname " - "'switch-1.example.com'. No fallback IP address is configured " - "for this device." - ) - - @patch('SNMP.snmp_test._perform_snmp_get') - @patch('SNMP.snmp_test._perform_snmp_walk') - @patch('SNMP.snmp_test._perform_snmp_table') - def test_auth_failure_returns_success_false_with_auth_error( - self, mock_table, mock_walk, mock_get, - authenticated_client, test_device - ): - auth_error = {'error': 'Unknown USM user'} - mock_get.return_value = {'sysDescr': auth_error} - mock_walk.return_value = {} - mock_table.return_value = {} - - response = authenticated_client.post( - '/SNMP/RunSNMPTest/', - data=json.dumps({'device_id': test_device.pk}), - content_type='application/json' - ) - assert response.status_code == 200 - data = json.loads(response.content) - assert data['success'] is False - assert 'authentication' in data['error'].lower() or 'auth' in data['error'].lower() - - @patch('SNMP.snmp_test._perform_snmp_get') - @patch('SNMP.snmp_test._perform_snmp_walk') - @patch('SNMP.snmp_test._perform_snmp_table') - def test_all_operations_fail_returns_success_false( - self, mock_table, mock_walk, mock_get, - authenticated_client, test_device - ): - mock_get.return_value = {'sysDescr': {'error': 'No response from device'}} - mock_walk.return_value = {} - mock_table.return_value = {} - - response = authenticated_client.post( - '/SNMP/RunSNMPTest/', - data=json.dumps({'device_id': test_device.pk}), - content_type='application/json' - ) - assert response.status_code == 200 - data = json.loads(response.content) - assert data['success'] is False - - @patch('SNMP.snmp_test._perform_snmp_get') - @patch('SNMP.snmp_test._perform_snmp_walk') - @patch('SNMP.snmp_test._perform_snmp_table') - def test_partial_success_returns_success_true_with_has_errors( - self, mock_table, mock_walk, mock_get, - authenticated_client, test_device, test_custom_profile, db - ): - # Profile has two GET OIDs so we can simulate one success and one failure - test_custom_profile.profile_data = { - 'get': {'sysDescr': '1.3.6.1.2.1.1.1.0', 'sysUpTime': '1.3.6.1.2.1.1.3.0'}, - 'walk': {}, - 'table': {} - } - test_custom_profile.save() - - mock_get.return_value = { - 'sysDescr': 'Linux router', - 'sysUpTime': {'error': 'No such object'} - } - mock_walk.return_value = {} - mock_table.return_value = {} - - response = authenticated_client.post( - '/SNMP/RunSNMPTest/', - data=json.dumps({'device_id': test_device.pk}), - content_type='application/json' - ) - assert response.status_code == 200 - data = json.loads(response.content) - assert data['success'] is True - assert data['has_errors'] is True - - def test_get_method_not_allowed(self, authenticated_client): - response = authenticated_client.get('/SNMP/RunSNMPTest/') - assert response.status_code == 405 - - @patch('SNMP.snmp_test._perform_snmp_get') - @patch('SNMP.snmp_test._perform_snmp_walk') - @patch('SNMP.snmp_test._perform_snmp_table') - def test_explicit_template_id_overrides_device_template( - self, mock_table, mock_walk, mock_get, - authenticated_client, test_device, db - ): - mock_get.return_value = {'sysDescr': 'Linux router'} - mock_walk.return_value = {} - mock_table.return_value = {} - - # Create a second template with a profile - extra_profile = Profile.objects.create( - name='snmp_test_extra_profile', - vendor='Generic', - profile_data={'get': {'sysContact': '1.3.6.1.2.1.1.4.0'}, 'walk': {}, 'table': {}} - ) - extra_template = DeviceTemplate.objects.create( - name='snmp_test_extra_template', - vendor='Generic', - ) - extra_template.profiles.add(extra_profile) - - response = authenticated_client.post( - '/SNMP/RunSNMPTest/', - data=json.dumps({'device_id': test_device.pk, 'template_id': extra_template.pk}), - content_type='application/json' - ) - assert response.status_code == 200 - data = json.loads(response.content) - assert data['template']['name'] == extra_template.name - - @patch('SNMP.snmp_test._perform_snmp_get') - @patch('SNMP.snmp_test._perform_snmp_walk') - @patch('SNMP.snmp_test._perform_snmp_table') - def test_response_contains_execution_time( - self, mock_table, mock_walk, mock_get, - authenticated_client, test_device - ): - mock_get.return_value = {'sysDescr': 'Linux router'} - mock_walk.return_value = {} - mock_table.return_value = {} - - response = authenticated_client.post( - '/SNMP/RunSNMPTest/', - data=json.dumps({'device_id': test_device.pk}), - content_type='application/json' - ) - data = json.loads(response.content) - assert 'execution_time' in data - - -# =========================================================================== -# RunSNMPWalk view -# =========================================================================== - -@pytest.mark.django_db -class TestRunSNMPWalkView: - - def test_missing_host_returns_400(self, authenticated_client, test_credential_v2c): - response = authenticated_client.post( - '/SNMP/RunSNMPWalk/', - data=json.dumps({'credential_id': test_credential_v2c.pk}), - content_type='application/json' - ) - assert response.status_code == 400 - data = json.loads(response.content) - assert data['success'] is False - assert 'host' in data['error'] - - def test_missing_credential_id_returns_400(self, authenticated_client): - response = authenticated_client.post( - '/SNMP/RunSNMPWalk/', - data=json.dumps({'host': '10.0.0.1'}), - content_type='application/json' - ) - assert response.status_code == 400 - data = json.loads(response.content) - assert data['success'] is False - assert 'credential_id' in data['error'] - - def test_credential_not_found_returns_404(self, authenticated_client): - response = authenticated_client.post( - '/SNMP/RunSNMPWalk/', - data=json.dumps({'host': '10.0.0.1', 'credential_id': 999999}), - content_type='application/json' - ) - assert response.status_code == 404 - data = json.loads(response.content) - assert data['success'] is False - - @patch('SNMP.snmp_test._perform_full_walk') - def test_successful_walk_returns_results(self, mock_walk, authenticated_client, test_credential_v2c): - mock_walk.return_value = { - 'results': [ - {'oid': '1.3.6.1.2.1.1.1.0', 'value': 'Linux router'}, - {'oid': '1.3.6.1.2.1.1.2.0', 'value': '1.3.6.1.4.1.8072.3.2.10'}, - ] - } - - response = authenticated_client.post( - '/SNMP/RunSNMPWalk/', - data=json.dumps({'host': '10.0.0.1', 'credential_id': test_credential_v2c.pk}), - content_type='application/json' - ) - assert response.status_code == 200 - data = json.loads(response.content) - assert data['success'] is True - assert data['oid_count'] == 2 - assert len(data['results']) == 2 - assert data['host'] == '10.0.0.1' - - @patch('SNMP.snmp_test._perform_full_walk') - def test_walk_error_without_results_returns_success_false( - self, mock_walk, authenticated_client, test_credential_v2c - ): - mock_walk.return_value = {'error': 'No response from device', 'results': []} - - response = authenticated_client.post( - '/SNMP/RunSNMPWalk/', - data=json.dumps({'host': '10.0.0.1', 'credential_id': test_credential_v2c.pk}), - content_type='application/json' - ) - assert response.status_code == 200 - data = json.loads(response.content) - assert data['success'] is False - assert 'error' in data - - @patch('SNMP.snmp_test._perform_full_walk') - def test_walk_with_partial_error_and_results_returns_success_true( - self, mock_walk, authenticated_client, test_credential_v2c - ): - mock_walk.return_value = { - 'results': [{'oid': '1.3.6.1.2.1.1.1.0', 'value': 'Linux'}], - 'error': 'End of MIB' - } - - response = authenticated_client.post( - '/SNMP/RunSNMPWalk/', - data=json.dumps({'host': '10.0.0.1', 'credential_id': test_credential_v2c.pk}), - content_type='application/json' - ) - assert response.status_code == 200 - data = json.loads(response.content) - assert data['success'] is True - assert data['partial_error'] == 'End of MIB' - - @patch('SNMP.snmp_test._perform_full_walk') - def test_custom_port_and_start_oid_forwarded( - self, mock_walk, authenticated_client, test_credential_v2c - ): - mock_walk.return_value = {'results': []} - - authenticated_client.post( - '/SNMP/RunSNMPWalk/', - data=json.dumps({ - 'host': '10.0.0.1', - 'port': 1161, - 'credential_id': test_credential_v2c.pk, - 'start_oid': '1.3.6.1.2.1.2' - }), - content_type='application/json' - ) - mock_walk.assert_called_once() - call_args = mock_walk.call_args - assert call_args[0][1] == 1161 # port - assert call_args[0][3] == '1.3.6.1.2.1.2' # start_oid - - @patch('SNMP.snmp_test._perform_full_walk') - def test_default_port_is_161(self, mock_walk, authenticated_client, test_credential_v2c): - mock_walk.return_value = {'results': []} - - authenticated_client.post( - '/SNMP/RunSNMPWalk/', - data=json.dumps({'host': '10.0.0.1', 'credential_id': test_credential_v2c.pk}), - content_type='application/json' - ) - call_args = mock_walk.call_args - assert call_args[0][1] == 161 - - @patch('SNMP.snmp_test._perform_full_walk') - def test_default_start_oid_is_1_3_6_1(self, mock_walk, authenticated_client, test_credential_v2c): - mock_walk.return_value = {'results': []} - - authenticated_client.post( - '/SNMP/RunSNMPWalk/', - data=json.dumps({'host': '10.0.0.1', 'credential_id': test_credential_v2c.pk}), - content_type='application/json' - ) - call_args = mock_walk.call_args - assert call_args[0][3] == '1.3.6.1' - - @patch('SNMP.snmp_test._perform_full_walk') - def test_response_includes_execution_time( - self, mock_walk, authenticated_client, test_credential_v2c - ): - mock_walk.return_value = {'results': []} - - response = authenticated_client.post( - '/SNMP/RunSNMPWalk/', - data=json.dumps({'host': '10.0.0.1', 'credential_id': test_credential_v2c.pk}), - content_type='application/json' - ) - data = json.loads(response.content) - assert 'execution_time' in data - - def test_get_method_not_allowed(self, authenticated_client): - response = authenticated_client.get('/SNMP/RunSNMPWalk/') - assert response.status_code == 405 diff --git a/src/logstashui/SNMP/tests/test_views.py b/src/logstashui/SNMP/tests/test_views.py deleted file mode 100644 index d76d794..0000000 --- a/src/logstashui/SNMP/tests/test_views.py +++ /dev/null @@ -1,1098 +0,0 @@ -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. - -import pytest -from django.contrib.auth.models import User -from django.test import Client -from unittest.mock import patch, MagicMock -import json -import os - -from SNMP.models import Network, Device, Credential, Profile -from PipelineManager.models import Connection -from Management.models import UserProfile - - -@pytest.fixture -def admin_user(db): - """Create a user with admin profile""" - user = User.objects.create_user( - username='admin_user', - password='testpass123', - email='admin@example.com' - ) - profile, created = UserProfile.objects.get_or_create(user=user, defaults={'role': 'admin'}) - if not created: - profile.role = 'admin' - profile.save() - return user - - -@pytest.fixture -def readonly_user(db): - """Create a user with readonly profile""" - user = User.objects.create_user( - username='readonly_user', - password='testpass123', - email='readonly@example.com' - ) - profile = UserProfile.objects.get(user=user) - profile.role = 'readonly' - profile.save() - user.refresh_from_db() - return user - - -@pytest.fixture -def authenticated_client(admin_user): - """Create an authenticated client with admin user""" - client = Client() - client.force_login(admin_user) - return client - - -@pytest.fixture -def readonly_client(readonly_user): - """Create an authenticated client with readonly user""" - client = Client() - client.force_login(readonly_user) - return client - - -@pytest.fixture -def test_connection(db): - """Create a test Elasticsearch connection""" - return Connection.objects.create( - name='Test Connection', - connection_type='CENTRALIZED', - host='https://localhost:9200', - username='elastic', - password='changeme' - ) - - -@pytest.fixture -def test_credential(db): - """Create a test SNMP credential""" - return Credential.objects.create( - name='Test Credential', - version='2c', - community='public', - description='Test SNMP v2c credential' - ) - - -@pytest.fixture -def test_network(db, test_connection, test_credential): - """Create a test SNMP network""" - return Network.objects.create( - name='Test Network', - network_range='192.168.1.0/24', - connection=test_connection, - discovery_credential=test_credential, - discovery_enabled=True, - traps_enabled=False, - interval=30 - ) - - -@pytest.fixture -def test_device(db, test_network, test_credential): - """Create a test SNMP device""" - return Device.objects.create( - name='Test Device', - ip_address='192.168.1.100', - port=161, - retries=2, - timeout=1000, - credential=test_credential, - network=test_network - ) - - -@pytest.fixture -def test_profile(db): - """Create a test user profile""" - return Profile.objects.create( - name='custom_profile', - description='Custom test profile', - vendor='Generic', - profile_data={ - 'get': { - 'test.metric': '1.3.6.1.2.1.1.1.0' - }, - 'walk': {}, - 'table': {} - } - ) - - -# ============================================================================ -# View Tests - Read-Only Pages -# ============================================================================ - -@pytest.mark.django_db -class TestNetworksView: - """Test Networks page view""" - - def test_networks_view_requires_authentication(self, client): - """Test that Networks view requires authentication""" - response = client.get('/SNMP/Networks/') - assert response.status_code == 302 - assert '/Management/Login/' in response.url - - def test_networks_view_accessible_to_admin(self, authenticated_client): - """Test that admin users can access Networks view""" - response = authenticated_client.get('/SNMP/Networks/') - assert response.status_code == 200 - assert b'Networks' in response.content or b'networks' in response.content - - def test_networks_view_accessible_to_readonly(self, readonly_client): - """Test that readonly users can access Networks view""" - response = readonly_client.get('/SNMP/Networks/') - assert response.status_code == 200 - - def test_networks_view_displays_networks(self, authenticated_client, test_network): - """Test that Networks view displays existing networks""" - response = authenticated_client.get('/SNMP/Networks/') - assert response.status_code == 200 - # Networks are loaded via AJAX, so just verify the page loads and has the networks context - assert 'networks' in response.context - - def test_networks_view_with_connection_form(self, authenticated_client): - """Test that Networks view includes connection form""" - response = authenticated_client.get('/SNMP/Networks/') - assert response.status_code == 200 - # Should have form context - assert 'form' in response.context - - -@pytest.mark.django_db -class TestDevicesView: - """Test Devices page view""" - - def test_devices_view_requires_authentication(self, client): - """Test that Devices view requires authentication""" - response = client.get('/SNMP/Devices/') - assert response.status_code == 302 - assert '/Management/Login/' in response.url - - def test_devices_view_accessible_to_admin(self, authenticated_client): - """Test that admin users can access Devices view""" - response = authenticated_client.get('/SNMP/Devices/') - assert response.status_code == 200 - - def test_devices_view_accessible_to_readonly(self, readonly_client): - """Test that readonly users can access Devices view""" - response = readonly_client.get('/SNMP/Devices/') - assert response.status_code == 200 - - def test_devices_view_displays_devices(self, authenticated_client, test_device): - """Test that Devices view displays existing devices""" - response = authenticated_client.get('/SNMP/Devices/') - assert response.status_code == 200 - # Device data is loaded via AJAX, so just check page loads - assert b'devices' in response.content.lower() - - -@pytest.mark.django_db -class TestProfilesView: - """Test DeviceTemplates page (profiles now live there; /SNMP/Profiles/ no longer exists)""" - - def test_device_templates_accessible_to_admin(self, authenticated_client): - """Admin users can access DeviceTemplates (the new home for profiles)""" - response = authenticated_client.get('/SNMP/DeviceTemplates/') - assert response.status_code == 200 - - def test_device_templates_accessible_to_readonly(self, readonly_client): - """Readonly users can access DeviceTemplates""" - response = readonly_client.get('/SNMP/DeviceTemplates/') - assert response.status_code == 200 - - def test_device_templates_exposes_profiles_in_context(self, authenticated_client): - """DeviceTemplates view includes 'profiles' list in its context""" - response = authenticated_client.get('/SNMP/DeviceTemplates/') - assert response.status_code == 200 - assert 'profiles' in response.context - - def test_device_templates_displays_user_profiles(self, authenticated_client, test_profile): - """User-created profiles appear in the DeviceTemplates context""" - response = authenticated_client.get('/SNMP/DeviceTemplates/') - assert response.status_code == 200 - profiles = response.context['profiles'] - assert any(p['name'] == 'custom_profile' for p in profiles) - - def test_device_templates_excludes_placeholder_profiles(self, authenticated_client): - """Placeholder profiles are excluded from the profiles list on DeviceTemplates""" - Profile.objects.create( - name='placeholder.json', - vendor='Generic', - profile_data={'is_official_placeholder': True}, - description='Placeholder' - ) - response = authenticated_client.get('/SNMP/DeviceTemplates/') - assert response.status_code == 200 - profiles = response.context['profiles'] - user_profiles = [p for p in profiles if not p['is_official']] - assert not any(p['name'] == 'placeholder.json' for p in user_profiles) - - def test_device_templates_profiles_sorted_alphabetically(self, authenticated_client): - """Profiles are sorted alphabetically by display_name on DeviceTemplates""" - response = authenticated_client.get('/SNMP/DeviceTemplates/') - assert response.status_code == 200 - profiles = response.context['profiles'] - display_names = [p['display_name'] for p in profiles] - assert display_names == sorted(display_names) - - -@pytest.mark.django_db -class TestCredentialsView: - """Test Credentials page view""" - - def test_credentials_view_requires_authentication(self, client): - """Test that Credentials view requires authentication""" - response = client.get('/SNMP/Credentials/') - assert response.status_code == 302 - assert '/Management/Login/' in response.url - - def test_credentials_view_accessible_to_admin(self, authenticated_client): - """Test that admin users can access Credentials view""" - response = authenticated_client.get('/SNMP/Credentials/') - assert response.status_code == 200 - - def test_credentials_view_accessible_to_readonly(self, readonly_client): - """Test that readonly users can access Credentials view""" - response = readonly_client.get('/SNMP/Credentials/') - assert response.status_code == 200 - - def test_credentials_view_displays_credentials(self, authenticated_client, test_credential): - """Test that Credentials view displays existing credentials""" - response = authenticated_client.get('/SNMP/Credentials/') - assert response.status_code == 200 - # Credentials data is loaded via AJAX, so just check page loads - assert b'credentials' in response.content.lower() - - -# ============================================================================ -# Edge Cases and Error Handling -# ============================================================================ - -@pytest.mark.django_db -class TestViewsEdgeCases: - """Test edge cases and error handling in views""" - - def test_networks_view_with_no_networks(self, authenticated_client): - """Test Networks view when no networks exist""" - response = authenticated_client.get('/SNMP/Networks/') - assert response.status_code == 200 - assert 'networks' in response.context - assert len(response.context['networks']) == 0 - - def test_device_templates_with_invalid_json_file(self, authenticated_client, settings): - """DeviceTemplates handles malformed official profile JSON files gracefully""" - official_profiles_dir = os.path.join(settings.BASE_DIR, 'SNMP', 'data', 'official_profiles') - if os.path.exists(official_profiles_dir): - invalid_file = os.path.join(official_profiles_dir, 'test_invalid.json') - try: - with open(invalid_file, 'w') as f: - f.write('{ invalid json }') - response = authenticated_client.get('/SNMP/DeviceTemplates/') - assert response.status_code == 200 - finally: - if os.path.exists(invalid_file): - os.remove(invalid_file) - - def test_view_with_database_error(self, authenticated_client): - """Test views handle database errors gracefully""" - with patch('SNMP.models.Network.objects') as mock_objects: - mock_objects.select_related.side_effect = Exception("Database error") - - # Should return error response, not crash - try: - response = authenticated_client.get('/SNMP/Networks/') - # May return 500 or handle gracefully - assert response.status_code in [200, 500] - except Exception: - # If exception is raised, that's also acceptable for this test - pass - - -# ============================================================================ -# Additional context / content verification tests -# ============================================================================ - -@pytest.mark.django_db -class TestViewContextContent: - """Additional tests verifying the data passed to each template context""" - - def test_networks_context_contains_network_instance(self, authenticated_client, test_network): - """Networks context 'networks' queryset contains our test network""" - response = authenticated_client.get('/SNMP/Networks/') - assert response.status_code == 200 - network_names = [n.name for n in response.context['networks']] - assert 'Test Network' in network_names - - def test_devices_context_has_devices_key(self, authenticated_client, test_device): - """Devices view passes 'devices' queryset to template""" - response = authenticated_client.get('/SNMP/Devices/') - assert response.status_code == 200 - assert 'devices' in response.context - device_names = [d.name for d in response.context['devices']] - assert 'Test Device' in device_names - - def test_credentials_context_has_credentials_key(self, authenticated_client, test_credential): - """Credentials view passes 'credentials' queryset to template""" - response = authenticated_client.get('/SNMP/Credentials/') - assert response.status_code == 200 - assert 'credentials' in response.context - cred_names = [c.name for c in response.context['credentials']] - assert 'Test Credential' in cred_names - - def test_device_templates_user_profile_required_fields(self, authenticated_client, test_profile): - """User profile dicts in DeviceTemplates context contain all required fields""" - response = authenticated_client.get('/SNMP/DeviceTemplates/') - assert response.status_code == 200 - user_profiles = [p for p in response.context['profiles'] if not p['is_official']] - for p in user_profiles: - for key in ('name', 'display_name', 'description', 'vendor', 'is_official'): - assert key in p - - def test_device_templates_official_profile_fields(self, authenticated_client): - """Official profiles in DeviceTemplates context always have is_official=True""" - response = authenticated_client.get('/SNMP/DeviceTemplates/') - assert response.status_code == 200 - official_profiles = [p for p in response.context['profiles'] if p['is_official']] - for p in official_profiles: - assert p['is_official'] is True - for key in ('name', 'display_name', 'description', 'vendor'): - assert key in p - - def test_device_templates_invalid_json_handled_gracefully(self, authenticated_client, settings, tmp_path): - """DeviceTemplates gracefully skips official profile JSON files that cannot be parsed""" - official_dir = tmp_path / 'official_profiles' - official_dir.mkdir() - (official_dir / 'broken.json').write_text('{ not valid json }') - - with patch('SNMP.views.os.path.exists', return_value=True), \ - patch('SNMP.views.os.listdir', return_value=['broken.json']): - response = authenticated_client.get('/SNMP/DeviceTemplates/') - assert response.status_code == 200 - profiles = response.context['profiles'] - official_names = [p['name'] for p in profiles if p['is_official']] - assert 'broken' in official_names - broken = next(p for p in profiles if p['name'] == 'broken') - assert broken['description'] == '' - - def test_networks_view_form_is_connection_form(self, authenticated_client): - """Networks context form is a ConnectionForm instance""" - from PipelineManager.forms import ConnectionForm - response = authenticated_client.get('/SNMP/Networks/') - assert isinstance(response.context['form'], ConnectionForm) - - def test_profiles_alphabetical_sort_among_unpinned(self, authenticated_client): - """User profiles are sorted alphabetically by display_name on DeviceTemplates""" - Profile.objects.all().delete() # start clean for this test - Profile.objects.create(name='zebra_profile', vendor='Generic', description='', profile_data={'get': {}}) - Profile.objects.create(name='alpha_profile', vendor='Generic', description='', profile_data={'get': {}}) - - response = authenticated_client.get('/SNMP/DeviceTemplates/') - assert response.status_code == 200 - user_profiles = [p for p in response.context['profiles'] if not p['is_official']] - display_names = [p['display_name'] for p in user_profiles] - assert display_names == sorted(display_names) - - -# ============================================================================ -# Onboarding View Tests -# ============================================================================ - -@pytest.mark.django_db -class TestOnboardingView: - """Test the SNMP Onboarding page view""" - - def test_onboarding_requires_authentication(self, client): - response = client.get('/SNMP/Onboarding/') - assert response.status_code == 302 - assert '/Management/Login/' in response.url - - def test_onboarding_accessible_to_admin(self, authenticated_client): - response = authenticated_client.get('/SNMP/Onboarding/') - assert response.status_code == 200 - - def test_onboarding_accessible_to_readonly(self, readonly_client): - response = readonly_client.get('/SNMP/Onboarding/') - assert response.status_code == 200 - - def test_onboarding_has_all_required_context_keys(self, authenticated_client, test_network, test_credential, test_device): - response = authenticated_client.get('/SNMP/Onboarding/') - assert response.status_code == 200 - for key in ('connections', 'credentials', 'networks', 'templates', 'devices', 'device_count', 'form'): - assert key in response.context, f"Missing context key: {key}" - - def test_onboarding_connections_include_suggested_kibana_url(self, authenticated_client, test_connection): - response = authenticated_client.get('/SNMP/Onboarding/') - assert response.status_code == 200 - connections = list(response.context['connections']) - assert connections, 'expected at least the test_connection' - assert 'suggested_kibana_url' in connections[0] - - def test_onboarding_device_count_matches_db(self, authenticated_client, test_device): - from SNMP.models import Device as _Device - response = authenticated_client.get('/SNMP/Onboarding/') - assert response.status_code == 200 - assert response.context['device_count'] == _Device.objects.count() - - def test_onboarding_networks_excludes_nothing(self, authenticated_client, test_network): - response = authenticated_client.get('/SNMP/Onboarding/') - assert response.status_code == 200 - network_names = [n.name for n in response.context['networks']] - assert test_network.name in network_names - - def test_onboarding_templates_excludes_default(self, authenticated_client): - from SNMP.models import DeviceTemplate - DeviceTemplate.objects.create(name='default', vendor='Generic', description='default template') - DeviceTemplate.objects.create(name='custom_tpl', vendor='Generic', description='custom') - response = authenticated_client.get('/SNMP/Onboarding/') - assert response.status_code == 200 - template_names = [t.name for t in response.context['templates']] - assert 'default' not in template_names - assert 'custom_tpl' in template_names - - -# ============================================================================ -# CheckDeviceType View Tests -# ============================================================================ - -@pytest.mark.django_db -class TestCheckDeviceTypeView: - """Test the CheckDeviceType view (lightweight SNMP probe)""" - - def test_get_method_returns_405(self, authenticated_client): - response = authenticated_client.get('/SNMP/CheckDeviceType/') - assert response.status_code == 405 - - def test_missing_host_returns_400(self, authenticated_client, test_credential): - data = json.dumps({'credential_id': test_credential.id}) - response = authenticated_client.post('/SNMP/CheckDeviceType/', data, content_type='application/json') - assert response.status_code == 400 - assert 'host is required' in response.json()['error'] - - def test_missing_credential_id_returns_400(self, authenticated_client): - data = json.dumps({'host': '192.168.1.1'}) - response = authenticated_client.post('/SNMP/CheckDeviceType/', data, content_type='application/json') - assert response.status_code == 400 - assert 'credential_id is required' in response.json()['error'] - - def test_nonexistent_credential_returns_404(self, authenticated_client): - data = json.dumps({'host': '192.168.1.1', 'credential_id': 99999}) - response = authenticated_client.post('/SNMP/CheckDeviceType/', data, content_type='application/json') - assert response.status_code == 404 - assert 'not found' in response.json()['error'].lower() - - def test_invalid_json_body_returns_400(self, authenticated_client): - response = authenticated_client.post( - '/SNMP/CheckDeviceType/', 'not valid json', content_type='application/json' - ) - assert response.status_code == 400 - - def test_unreachable_host_returns_error_payload(self, authenticated_client, test_credential): - with patch('SNMP.views._snmp_get_sys_descr', return_value=None): - data = json.dumps({'host': '10.255.255.255', 'credential_id': test_credential.id}) - response = authenticated_client.post('/SNMP/CheckDeviceType/', data, content_type='application/json') - assert response.status_code == 200 - rdata = response.json() - assert rdata['success'] is False - assert 'error' in rdata - - def test_reachable_host_returns_sys_descr(self, authenticated_client, test_credential): - sys_descr = 'Linux router 5.10.0 #1 SMP x86_64' - with patch('SNMP.views._snmp_get_sys_descr', return_value=sys_descr): - data = json.dumps({'host': '192.168.1.1', 'credential_id': test_credential.id}) - response = authenticated_client.post('/SNMP/CheckDeviceType/', data, content_type='application/json') - assert response.status_code == 200 - rdata = response.json() - assert rdata['success'] is True - assert rdata['sys_descr'] == sys_descr - - def test_matched_template_returned_when_found(self, authenticated_client, test_credential): - from SNMP.models import DeviceTemplate, Profile as _Profile - profile = _Profile.objects.create( - name='linux_match_profile', vendor='Linux', - profile_data={'get': {}, 'walk': {}, 'table': {}} - ) - tpl = DeviceTemplate.objects.create( - name='linux_match', vendor='Linux', matching_rules=['Linux'] - ) - tpl.profiles.add(profile) - - with patch('SNMP.views._snmp_get_sys_descr', return_value='Linux router 5.10 SMP x86_64'), \ - patch('SNMP.snmp_crud.suggest_device_template', return_value=[tpl.id]): - data = json.dumps({'host': '192.168.1.1', 'credential_id': test_credential.id}) - response = authenticated_client.post('/SNMP/CheckDeviceType/', data, content_type='application/json') - assert response.status_code == 200 - rdata = response.json() - assert rdata['success'] is True - assert rdata['matched_template'] is not None - assert rdata['matched_template']['name'] == 'linux_match' - - def test_no_match_returns_null_template(self, authenticated_client, test_credential): - with patch('SNMP.views._snmp_get_sys_descr', return_value='Unknown Device XR-9000'), \ - patch('SNMP.snmp_crud.suggest_device_template', return_value=[]): - data = json.dumps({'host': '192.168.1.1', 'credential_id': test_credential.id}) - response = authenticated_client.post('/SNMP/CheckDeviceType/', data, content_type='application/json') - assert response.status_code == 200 - rdata = response.json() - assert rdata['success'] is True - assert rdata['matched_template'] is None - - def test_readonly_user_is_denied(self, readonly_client, test_credential): - data = json.dumps({'host': '192.168.1.1', 'credential_id': test_credential.id}) - response = readonly_client.post('/SNMP/CheckDeviceType/', data, content_type='application/json') - assert response.status_code == 403 - - def test_custom_port_accepted(self, authenticated_client, test_credential): - with patch('SNMP.views._snmp_get_sys_descr', return_value='Device Description') as mock_fn: - data = json.dumps({'host': '10.0.0.1', 'port': 1161, 'credential_id': test_credential.id}) - response = authenticated_client.post('/SNMP/CheckDeviceType/', data, content_type='application/json') - assert response.status_code == 200 - args = mock_fn.call_args[0] - assert args[1] == 1161 - - -# ============================================================================ -# ImportAIGeneratedDefinitions View Tests -# ============================================================================ - -@pytest.mark.django_db -class TestImportAIGeneratedDefinitions: - """Test the ImportAIGeneratedDefinitions view""" - - def test_get_method_returns_405(self, authenticated_client): - response = authenticated_client.get('/SNMP/ImportAIGeneratedDefinitions/') - assert response.status_code == 405 - - def test_invalid_json_body_returns_400(self, authenticated_client): - response = authenticated_client.post( - '/SNMP/ImportAIGeneratedDefinitions/', 'not json', content_type='application/json' - ) - assert response.status_code == 400 - - def test_profiles_not_list_returns_400(self, authenticated_client): - data = json.dumps({'profiles': 'not a list', 'device_template': {'name': 'tpl', 'profiles': []}}) - response = authenticated_client.post('/SNMP/ImportAIGeneratedDefinitions/', data, content_type='application/json') - assert response.status_code == 400 - assert 'profiles' in response.json()['error'] - - def test_template_not_dict_returns_400(self, authenticated_client): - data = json.dumps({'profiles': [], 'device_template': 'not a dict'}) - response = authenticated_client.post('/SNMP/ImportAIGeneratedDefinitions/', data, content_type='application/json') - assert response.status_code == 400 - - def test_template_missing_name_returns_400(self, authenticated_client): - data = json.dumps({'profiles': [], 'device_template': {'profiles': []}}) - response = authenticated_client.post('/SNMP/ImportAIGeneratedDefinitions/', data, content_type='application/json') - assert response.status_code == 400 - assert 'name' in response.json()['error'] - - def test_template_profiles_not_list_returns_400(self, authenticated_client): - data = json.dumps({'profiles': [], 'device_template': {'name': 'tpl', 'profiles': 'bad'}}) - response = authenticated_client.post('/SNMP/ImportAIGeneratedDefinitions/', data, content_type='application/json') - assert response.status_code == 400 - - def test_profile_missing_name_returns_422(self, authenticated_client): - data = json.dumps({ - 'profiles': [{'get': {}, 'walk': {}}], - 'device_template': {'name': 'My Template', 'profiles': []} - }) - response = authenticated_client.post('/SNMP/ImportAIGeneratedDefinitions/', data, content_type='application/json') - assert response.status_code == 422 - rdata = response.json() - assert rdata['success'] is False - assert len(rdata['errors']) > 0 - - def test_creates_new_profile_and_template(self, authenticated_client): - data = json.dumps({ - 'profiles': [ - { - 'name': 'import_test_profile', - 'vendor': 'Cisco', - 'description': 'Test profile', - 'get': {'cpu.0': '1.3.6.1.4.1.9.9.109.1.1.1.1.7.1'}, - 'walk': {}, - 'table': {} - } - ], - 'device_template': { - 'name': 'import_test_template', - 'vendor': 'Cisco', - 'description': 'Test template', - 'profiles': ['import_test_profile'] - } - }) - response = authenticated_client.post('/SNMP/ImportAIGeneratedDefinitions/', data, content_type='application/json') - assert response.status_code == 200 - rdata = response.json() - assert rdata['success'] is True - assert any(p['action'] == 'created' for p in rdata['profiles']) - assert rdata['template']['action'] == 'created' - assert Profile.objects.filter(name='import_test_profile').exists() - - def test_updates_existing_user_profile(self, authenticated_client, test_profile): - data = json.dumps({ - 'profiles': [ - {'name': 'custom_profile', 'vendor': 'Updated', 'get': {}, 'walk': {}, 'table': {}} - ], - 'device_template': { - 'name': 'update_import_tpl', - 'profiles': ['custom_profile'] - } - }) - response = authenticated_client.post('/SNMP/ImportAIGeneratedDefinitions/', data, content_type='application/json') - assert response.status_code == 200 - rdata = response.json() - assert any(p['action'] == 'updated' for p in rdata['profiles']) - - def test_skips_official_profile(self, authenticated_client): - Profile.objects.create( - name='official_cannot_overwrite', - vendor='Vendor', - official_key='vendor.official_cannot_overwrite', - profile_data={'get': {}, 'walk': {}, 'table': {}} - ) - data = json.dumps({ - 'profiles': [{'name': 'official_cannot_overwrite', 'get': {}, 'walk': {}}], - 'device_template': {'name': 'skip_official_tpl', 'profiles': ['official_cannot_overwrite']} - }) - response = authenticated_client.post('/SNMP/ImportAIGeneratedDefinitions/', data, content_type='application/json') - assert response.status_code == 200 - rdata = response.json() - assert any(p['action'] == 'skipped' for p in rdata['profiles']) - - def test_template_links_to_created_profiles(self, authenticated_client): - data = json.dumps({ - 'profiles': [ - {'name': 'linked_prof_a', 'vendor': 'Generic', 'get': {}, 'walk': {}, 'table': {}}, - {'name': 'linked_prof_b', 'vendor': 'Generic', 'get': {}, 'walk': {}, 'table': {}}, - ], - 'device_template': { - 'name': 'linked_template', - 'profiles': ['linked_prof_a', 'linked_prof_b'] - } - }) - response = authenticated_client.post('/SNMP/ImportAIGeneratedDefinitions/', data, content_type='application/json') - assert response.status_code == 200 - rdata = response.json() - assert rdata['success'] is True - from SNMP.models import DeviceTemplate - tpl = DeviceTemplate.objects.get(name='linked_template') - profile_names = set(tpl.profiles.values_list('name', flat=True)) - assert 'linked_prof_a' in profile_names - assert 'linked_prof_b' in profile_names - - def test_readonly_user_is_denied(self, readonly_client): - data = json.dumps({'profiles': [], 'device_template': {'name': 'tpl', 'profiles': []}}) - response = readonly_client.post('/SNMP/ImportAIGeneratedDefinitions/', data, content_type='application/json') - assert response.status_code == 403 - - def test_empty_profiles_list_creates_only_template(self, authenticated_client): - data = json.dumps({ - 'profiles': [], - 'device_template': {'name': 'empty_profiles_tpl', 'profiles': []} - }) - response = authenticated_client.post('/SNMP/ImportAIGeneratedDefinitions/', data, content_type='application/json') - assert response.status_code == 200 - rdata = response.json() - assert rdata['success'] is True - assert rdata['profiles'] == [] - assert rdata['template']['action'] == 'created' - - -# ============================================================================ -# Overview Page -# ============================================================================ - -@pytest.mark.django_db -class TestOverviewView: - """Test the SNMP Overview page view.""" - - def test_overview_requires_authentication(self, client): - response = client.get('/SNMP/Overview/') - assert response.status_code == 302 - assert '/Management/Login/' in response.url - - def test_overview_accessible_to_admin(self, authenticated_client): - response = authenticated_client.get('/SNMP/Overview/') - assert response.status_code == 200 - - def test_overview_accessible_to_readonly(self, readonly_client): - response = readonly_client.get('/SNMP/Overview/') - assert response.status_code == 200 - - -# ============================================================================ -# GetOverviewMetrics API -# ============================================================================ - -@pytest.mark.django_db -class TestGetOverviewMetricsView: - """Test the /SNMP/GetOverviewMetrics/ JSON endpoint.""" - - def test_get_overview_metrics_requires_auth(self, client): - response = client.get('/SNMP/GetOverviewMetrics/') - assert response.status_code == 302 - - def test_get_overview_metrics_success(self, authenticated_client): - """GetOverviewMetrics returns the expected JSON shape when ES helpers succeed.""" - with patch('SNMP.views.get_discovered_devices_count', return_value={'count': 5, 'errors': []}), \ - patch('SNMP.views.get_template_data_categories', return_value={'templates': [], 'errors': []}), \ - patch('SNMP.views.get_high_resource_usage', return_value={'high_cpu': [], 'high_memory': [], 'errors': []}): - response = authenticated_client.get('/SNMP/GetOverviewMetrics/') - - assert response.status_code == 200 - data = json.loads(response.content) - assert data['success'] is True - assert 'metrics' in data - assert data['metrics']['discovered_devices'] == 5 - assert 'high_usage' in data - assert 'data_quality' in data - - def test_get_overview_metrics_total_devices(self, authenticated_client, test_device): - """total_devices counts devices in the database.""" - with patch('SNMP.views.get_discovered_devices_count', return_value={'count': 0, 'errors': []}), \ - patch('SNMP.views.get_template_data_categories', return_value={'templates': [], 'errors': []}), \ - patch('SNMP.views.get_high_resource_usage', return_value={'high_cpu': [], 'high_memory': [], 'errors': []}): - response = authenticated_client.get('/SNMP/GetOverviewMetrics/') - - data = json.loads(response.content) - assert data['metrics']['total_devices'] >= 1 - - def test_get_overview_metrics_propagates_errors(self, authenticated_client): - """Errors from helpers are propagated to the response errors list.""" - with patch('SNMP.views.get_discovered_devices_count', return_value={'count': 0, 'errors': ['ES connection failed']}), \ - patch('SNMP.views.get_template_data_categories', return_value={'templates': [], 'errors': []}), \ - patch('SNMP.views.get_high_resource_usage', return_value={'high_cpu': [], 'high_memory': [], 'errors': []}): - response = authenticated_client.get('/SNMP/GetOverviewMetrics/') - - data = json.loads(response.content) - assert data['success'] is True - assert data['errors'] is not None - assert 'ES connection failed' in data['errors'] - - def test_get_overview_metrics_exception_returns_500(self, authenticated_client): - """An unexpected exception inside GetOverviewMetrics returns HTTP 500.""" - with patch('SNMP.views.get_discovered_devices_count', side_effect=Exception('Boom')): - response = authenticated_client.get('/SNMP/GetOverviewMetrics/') - assert response.status_code == 500 - data = json.loads(response.content) - assert data['success'] is False - - -# ============================================================================ -# CheckSNMPIndexTemplate -# ============================================================================ - -@pytest.mark.django_db -class TestCheckSNMPIndexTemplateView: - """Test the /SNMP/CheckSNMPIndexTemplate/ endpoint.""" - - def test_requires_post(self, authenticated_client): - response = authenticated_client.get('/SNMP/CheckSNMPIndexTemplate/') - assert response.status_code == 405 - - def test_requires_connection_ids(self, authenticated_client): - response = authenticated_client.post( - '/SNMP/CheckSNMPIndexTemplate/', - data=json.dumps({}), - content_type='application/json', - ) - assert response.status_code == 400 - - def test_invalid_json_body(self, authenticated_client): - response = authenticated_client.post( - '/SNMP/CheckSNMPIndexTemplate/', - data='not json', - content_type='application/json', - ) - assert response.status_code == 400 - - def test_connection_not_found(self, authenticated_client): - """A non-existent connection_id returns an error result (not a 500).""" - with patch('SNMP.views._load_snmp_template', return_value={'_meta': {'template_name': 'metrics-snmp.polling'}}), \ - patch('Common.elastic_utils.check_index_template', return_value={'status': 'installed', 'differences': []}): - response = authenticated_client.post( - '/SNMP/CheckSNMPIndexTemplate/', - data=json.dumps({'connection_ids': [99999]}), - content_type='application/json', - ) - assert response.status_code == 200 - data = json.loads(response.content) - result = data['results'][0] - assert result['status'] == 'error' - assert 'not found' in result['error'] - - def test_installed_status(self, authenticated_client, test_connection): - """Returns 'installed' status when template is present and up to date.""" - with patch('SNMP.views._load_snmp_template', return_value={'_meta': {'template_name': 'metrics-snmp.polling'}}), \ - patch('Common.elastic_utils.check_index_template', return_value={'status': 'installed', 'differences': []}): - response = authenticated_client.post( - '/SNMP/CheckSNMPIndexTemplate/', - data=json.dumps({'connection_ids': [test_connection.id]}), - content_type='application/json', - ) - assert response.status_code == 200 - data = json.loads(response.content) - assert data['results'][0]['status'] == 'installed' - assert data['results'][0]['connection_name'] == test_connection.name - - -# ============================================================================ -# InstallSNMPIndexTemplate -# ============================================================================ - -@pytest.mark.django_db -class TestInstallSNMPIndexTemplateView: - """Test the /SNMP/InstallSNMPIndexTemplate/ endpoint.""" - - def test_requires_admin(self, readonly_client, test_connection): - response = readonly_client.post( - '/SNMP/InstallSNMPIndexTemplate/', - data=json.dumps({'connection_ids': [test_connection.id]}), - content_type='application/json', - ) - assert response.status_code == 403 - - def test_requires_post(self, authenticated_client): - response = authenticated_client.get('/SNMP/InstallSNMPIndexTemplate/') - assert response.status_code == 405 - - def test_requires_connection_ids(self, authenticated_client): - response = authenticated_client.post( - '/SNMP/InstallSNMPIndexTemplate/', - data=json.dumps({}), - content_type='application/json', - ) - assert response.status_code == 400 - - def test_success(self, authenticated_client, test_connection): - """Successfully installing a template returns success=True.""" - with patch('SNMP.views._load_snmp_template', return_value={'_meta': {'template_name': 'metrics-snmp.polling'}}), \ - patch('Common.elastic_utils.create_index_template', return_value=None): - response = authenticated_client.post( - '/SNMP/InstallSNMPIndexTemplate/', - data=json.dumps({'connection_ids': [test_connection.id]}), - content_type='application/json', - ) - assert response.status_code == 200 - data = json.loads(response.content) - assert data['success'] is True - assert data['results'][0]['success'] is True - - def test_connection_not_found(self, authenticated_client): - """A non-existent connection_id records failure without crashing.""" - with patch('SNMP.views._load_snmp_template', return_value={'_meta': {'template_name': 'metrics-snmp.polling'}}): - response = authenticated_client.post( - '/SNMP/InstallSNMPIndexTemplate/', - data=json.dumps({'connection_ids': [99999]}), - content_type='application/json', - ) - assert response.status_code == 200 - data = json.loads(response.content) - assert data['success'] is False - assert data['results'][0]['success'] is False - - -# ============================================================================ -# CheckAgentBuilderResources -# ============================================================================ - -@pytest.mark.django_db -class TestCheckAgentBuilderResourcesView: - """Test the /SNMP/CheckAgentBuilderResources/ endpoint.""" - - def test_requires_post(self, authenticated_client): - response = authenticated_client.get('/SNMP/CheckAgentBuilderResources/') - assert response.status_code == 405 - - def test_requires_connection_id(self, authenticated_client): - response = authenticated_client.post( - '/SNMP/CheckAgentBuilderResources/', - data=json.dumps({}), - content_type='application/json', - ) - assert response.status_code == 400 - - def test_invalid_json_body(self, authenticated_client): - response = authenticated_client.post( - '/SNMP/CheckAgentBuilderResources/', - data='{ bad json', - content_type='application/json', - ) - assert response.status_code == 400 - - def test_success(self, authenticated_client): - """Returns result from AgentBuilder.check_resources when successful.""" - mock_result = {'tools': [], 'skills': [], 'agents': []} - with patch('Common.ai.agent_builder.AgentBuilder') as MockBuilder, \ - patch('Common.ai.agent_builder.load_resources_from_directory', return_value=([], [], [])): - MockBuilder.return_value.check_resources.return_value = mock_result - response = authenticated_client.post( - '/SNMP/CheckAgentBuilderResources/', - data=json.dumps({'connection_id': 1}), - content_type='application/json', - ) - assert response.status_code == 200 - - def test_agent_builder_exception_returns_500(self, authenticated_client): - """If AgentBuilder raises, the endpoint returns 500.""" - with patch('Common.ai.agent_builder.AgentBuilder', side_effect=Exception('KB down')), \ - patch('Common.ai.agent_builder.load_resources_from_directory', return_value=([], [], [])): - response = authenticated_client.post( - '/SNMP/CheckAgentBuilderResources/', - data=json.dumps({'connection_id': 1}), - content_type='application/json', - ) - assert response.status_code == 500 - - -# ============================================================================ -# InstallAgentBuilderPackage -# ============================================================================ - -@pytest.mark.django_db -class TestInstallAgentBuilderPackageView: - """Test the /SNMP/InstallAgentBuilderPackage/ endpoint.""" - - def test_requires_admin(self, readonly_client): - response = readonly_client.post( - '/SNMP/InstallAgentBuilderPackage/', - data=json.dumps({'connection_id': 1}), - content_type='application/json', - ) - assert response.status_code == 403 - - def test_requires_post(self, authenticated_client): - response = authenticated_client.get('/SNMP/InstallAgentBuilderPackage/') - assert response.status_code == 405 - - def test_requires_connection_id(self, authenticated_client): - response = authenticated_client.post( - '/SNMP/InstallAgentBuilderPackage/', - data=json.dumps({}), - content_type='application/json', - ) - assert response.status_code == 400 - - def test_success(self, authenticated_client): - """Returns result from AgentBuilder.apply_all_resources when successful.""" - mock_result = {'success': True, 'results': []} - with patch('Common.ai.agent_builder.AgentBuilder') as MockBuilder, \ - patch('Common.ai.agent_builder.load_resources_from_directory', return_value=([], [], [])): - MockBuilder.return_value.apply_all_resources.return_value = mock_result - response = authenticated_client.post( - '/SNMP/InstallAgentBuilderPackage/', - data=json.dumps({'connection_id': 1}), - content_type='application/json', - ) - assert response.status_code == 200 - - def test_exception_returns_500(self, authenticated_client): - """Unexpected exception returns 500 with success=False.""" - with patch('Common.ai.agent_builder.AgentBuilder', side_effect=Exception('Fail')), \ - patch('Common.ai.agent_builder.load_resources_from_directory', return_value=([], [], [])): - response = authenticated_client.post( - '/SNMP/InstallAgentBuilderPackage/', - data=json.dumps({'connection_id': 1}), - content_type='application/json', - ) - assert response.status_code == 500 - data = json.loads(response.content) - assert data['success'] is False - - -class TestGenerateTemplateGroundingInline: - """GenerateTemplateAndProfiles reduces the walk to MIB-grounded columns and passes - them INLINE to the agent. It must NOT stage the walk in a backend ES index — the - record of truth stays local to LogstashUI, which may connect to multiple backends, - so per-backend staging (residue + backend-dependent output) is disallowed.""" - - # Mixed walk: SNMPv2 + IF-MIB columns (grounded) plus an enterprise OID (ungrounded). - WALK = "\n".join([ - "1.3.6.1.2.1.1.1.0 = Cisco IOS Software, C2960X Software", - "1.3.6.1.2.1.1.3.0 = 44266130", - "1.3.6.1.2.1.1.5.0 = homelab-switch1", - "1.3.6.1.2.1.2.2.1.10.1 = 12345", # ifInOctets col (IF-MIB) -> grounded, instances=2 - "1.3.6.1.2.1.2.2.1.10.2 = 67890", - "1.3.6.1.4.1.9.9.999.1.0 = 1", # enterprise -> no compiled MIB -> ungrounded - ]) - - def _post(self, client, connection_id): - resp = client.post( - '/SNMP/GenerateTemplateAndProfiles/', - data=json.dumps({ - 'connection_id': connection_id, - 'walk_text': self.WALK, - 'inference_id': '.rainbow-sprinkles-elastic', - }), - content_type='application/json', - ) - # Drain the SSE stream. - return b''.join(resp.streaming_content).decode() - - @patch('Common.elastic_utils.bulk_index_documents') - @patch('Common.ai.agent_builder.AgentBuilder') - def test_grounded_columns_inline_no_backend_write( - self, MockAgentBuilder, mock_bulk, authenticated_client, test_connection - ): - instance = MockAgentBuilder.return_value - instance._kibana_url = 'https://kb.example' - captured = {} - - def _invoke(agent_id, message, **kwargs): - captured['agent_id'] = agent_id - captured['message'] = message - return iter(()) # empty agent stream is fine for this assertion - - instance.invoke_agent.side_effect = _invoke - - body = self._post(authenticated_client, test_connection.id) - - # 1. Nothing is written to any backend — no bulk index, no temp index name anywhere. - mock_bulk.assert_not_called() - assert 'snmp-template_generation' not in body - assert 'snmp-template_generation' not in captured['message'] - - # 2. The agent received the grounded columns INLINE (not an index to query). - assert 'grounded_columns' in captured['message'] - payload = json.loads(captured['message'][captured['message'].index('{'):]) - names = {c['name'] for c in payload['grounded_columns']} - assert 'sysDescr' in names # SNMPv2-MIB scalar grounded - assert 'ifInOctets' in names # IF-MIB table column grounded (multi-instance) - assert next(c for c in payload['grounded_columns'] if c['name'] == 'ifInOctets')['instances'] == 2 - # The enterprise OID had no compiled MIB -> reported for MIB-loading, not authored. - assert any(u['prefix'].startswith('1.3.6.1.4.1.9') for u in payload['ungrounded_subtrees']) - - # 3. SSE reports the grounding phase and never the old indexing phase. - assert '"phase": "grounding"' in body - assert 'indexing' not in body - - @patch('Common.elastic_utils.bulk_index_documents') - @patch('Common.ai.agent_builder.AgentBuilder') - def test_empty_grounding_errors_without_backend_write( - self, MockAgentBuilder, mock_bulk, authenticated_client, test_connection - ): - # A walk with only un-grounded enterprise OIDs -> no columns to author. - resp = authenticated_client.post( - '/SNMP/GenerateTemplateAndProfiles/', - data=json.dumps({ - 'connection_id': test_connection.id, - 'walk_text': '1.3.6.1.4.1.9999.1.2.3.0 = 5', - 'inference_id': '.rainbow-sprinkles-elastic', - }), - content_type='application/json', - ) - body = b''.join(resp.streaming_content).decode() - - assert '"phase": "error"' in body - mock_bulk.assert_not_called() - MockAgentBuilder.return_value.invoke_agent.assert_not_called() diff --git a/src/logstashui/Site/tests/__init__.py b/src/logstashui/Site/tests/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/src/logstashui/Site/tests/test_views.py b/src/logstashui/Site/tests/test_views.py deleted file mode 100644 index e5670b7..0000000 --- a/src/logstashui/Site/tests/test_views.py +++ /dev/null @@ -1,381 +0,0 @@ -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. - -import pytest -from django.urls import reverse -from unittest.mock import patch -from packaging import version -from Site import views - - -@pytest.mark.django_db -def test_health_check_returns_200(client): - url = reverse('health_check') - response = client.get(url) - assert response.status_code == 200 - assert response.json() == {'status': 'healthy', 'service': 'logstashui'} - - -@pytest.mark.django_db -def test_home_view_returns_200(client, django_user_model): - user = django_user_model.objects.create_user(username='testuser', password='testpass123') - client.force_login(user) - url = reverse('home') - response = client.get(url) - assert response.status_code == 200 - - -def test_parse_version_tag(): - assert views.parse_version_tag('v1.0.0') == version.parse('1.0.0') - assert views.parse_version_tag('2.1.3') == version.parse('2.1.3') - assert views.parse_version_tag('invalid_tag') is None - - -@patch('Site.views.requests.get') -def test_fetch_latest_version_from_docker_hub(mock_get): - class MockResponse: - def json(self): - return { - 'results': [ - {'name': 'v2.0.0'}, - {'name': '1.5.0'}, - {'name': 'latest'} - ] - } - - def raise_for_status(self): - pass - - mock_get.return_value = MockResponse() - - result = views.fetch_latest_version_from_docker_hub() - assert result == '2.0.0' - - -@patch('Site.views.get_latest_version') -@patch('Site.views.settings') -def test_check_for_update_newer_available(mock_settings, mock_get_latest): - mock_settings.__VERSION__ = '1.0.0' - mock_get_latest.return_value = '2.0.0' - - update_info = views.check_for_update() - - assert update_info is not None - assert update_info['update_available'] is True - assert update_info['latest_version'] == '2.0.0' - - -# ============================================================================ -# parse_version_tag — additional edge cases -# ============================================================================ - -def test_parse_version_tag_prerelease_is_parsed(): - """Pre-release tags parse to a version object (but is_prerelease=True)""" - result = views.parse_version_tag('v1.0.0a1') - assert result is not None - assert result.is_prerelease - - -def test_parse_version_tag_strips_whitespace(): - """Leading/trailing whitespace is stripped before parsing""" - result = views.parse_version_tag(' 1.2.3 ') - assert result is not None - from packaging import version as pkg_ver - assert result == pkg_ver.parse('1.2.3') - - -def test_parse_version_tag_empty_string_returns_none(): - """An empty string should return None, not raise""" - result = views.parse_version_tag('') - # An empty string isn't a valid semver — expect None - # (packaging may return a LegacyVersion or raise; either way we handle it) - # The function is supposed to return None on bad input via the except clause - # but packaging may allow it — just assert no exception is raised - # (returns None or the parsed value — both are acceptable) - assert result is None or result is not None # no exception - - -def test_parse_version_tag_v_prefix_stripped(): - """v-prefixed tags are parsed the same as the bare version""" - from packaging import version as pkg_ver - assert views.parse_version_tag('v3.0.1') == pkg_ver.parse('3.0.1') - - -# ============================================================================ -# fetch_latest_version_from_docker_hub — error and edge-case paths -# ============================================================================ - -@patch('Site.views.requests.get') -def test_fetch_latest_version_empty_results(mock_get): - """When results list is empty, returns None""" - mock_get.return_value.raise_for_status.return_value = None - mock_get.return_value.json.return_value = {'results': []} - - result = views.fetch_latest_version_from_docker_hub() - assert result is None - - -@patch('Site.views.requests.get') -def test_fetch_latest_version_only_non_semver_tags(mock_get): - """When no results have valid semver names, returns None""" - mock_get.return_value.raise_for_status.return_value = None - mock_get.return_value.json.return_value = { - 'results': [{'name': 'latest'}, {'name': 'edge'}, {'name': 'nightly'}] - } - - result = views.fetch_latest_version_from_docker_hub() - assert result is None - - -@patch('Site.views.requests.get') -def test_fetch_latest_version_only_prerelease_tags(mock_get): - """When all valid semver tags are pre-releases, returns None""" - mock_get.return_value.raise_for_status.return_value = None - mock_get.return_value.json.return_value = { - 'results': [{'name': 'v1.0.0a1'}, {'name': '2.0.0b3'}] - } - - result = views.fetch_latest_version_from_docker_hub() - assert result is None - - -@patch('Site.views.requests.get') -def test_fetch_latest_version_picks_highest(mock_get): - """Sorting picks the highest version, not just the first returned""" - mock_get.return_value.raise_for_status.return_value = None - mock_get.return_value.json.return_value = { - 'results': [ - {'name': '1.0.0'}, - {'name': '3.0.0'}, - {'name': '2.0.0'}, - ] - } - - result = views.fetch_latest_version_from_docker_hub() - assert result == '3.0.0' - - -@patch('Site.views.requests.get') -def test_fetch_latest_version_strips_v_prefix_from_result(mock_get): - """The returned version string has the leading 'v' stripped""" - mock_get.return_value.raise_for_status.return_value = None - mock_get.return_value.json.return_value = { - 'results': [{'name': 'v4.1.0'}] - } - - result = views.fetch_latest_version_from_docker_hub() - assert result == '4.1.0' - assert not result.startswith('v') - - -@patch('Site.views.requests.get', side_effect=__import__('requests').exceptions.Timeout) -def test_fetch_latest_version_timeout_returns_none(mock_get): - """A Timeout exception returns None gracefully""" - result = views.fetch_latest_version_from_docker_hub() - assert result is None - - -@patch('Site.views.requests.get', - side_effect=__import__('requests').exceptions.ConnectionError("refused")) -def test_fetch_latest_version_request_exception_returns_none(mock_get): - """A generic RequestException returns None gracefully""" - result = views.fetch_latest_version_from_docker_hub() - assert result is None - - -@patch('Site.views.requests.get', side_effect=ValueError("bad JSON")) -def test_fetch_latest_version_generic_exception_returns_none(mock_get): - """Any unexpected exception returns None gracefully""" - result = views.fetch_latest_version_from_docker_hub() - assert result is None - - -# ============================================================================ -# update_latest_version_cache -# ============================================================================ - -@patch('Site.views.cache') -@patch('Site.views.fetch_latest_version_from_docker_hub', return_value='1.2.3') -def test_update_latest_version_cache_sets_cache_on_success(mock_fetch, mock_cache): - """When fetch succeeds, the result is stored in the cache""" - views.update_latest_version_cache() - - mock_cache.set.assert_called_once_with(views.CACHE_KEY, '1.2.3', views.CACHE_TIMEOUT) - - -@patch('Site.views.cache') -@patch('Site.views.fetch_latest_version_from_docker_hub', return_value=None) -def test_update_latest_version_cache_does_not_set_on_failure(mock_fetch, mock_cache): - """When fetch returns None, cache.set is NOT called""" - views.update_latest_version_cache() - - mock_cache.set.assert_not_called() - - -@patch('Site.views.cache') -@patch('Site.views.fetch_latest_version_from_docker_hub', return_value='5.0.0') -def test_update_latest_version_cache_always_releases_lock(mock_fetch, mock_cache): - """Lock is always released via cache.delete in the finally block""" - views.update_latest_version_cache() - - mock_cache.delete.assert_called_once_with(views.CACHE_LOCK_KEY) - - -@patch('Site.views.cache') -@patch('Site.views.fetch_latest_version_from_docker_hub', side_effect=RuntimeError("explode")) -def test_update_latest_version_cache_releases_lock_on_exception(mock_fetch, mock_cache): - """Lock is released even when fetch_latest_version_from_docker_hub raises""" - # fetch raising inside update_latest_version_cache — that exception would propagate - # unless caught. The function doesn't catch it, so the finally still runs. - try: - views.update_latest_version_cache() - except RuntimeError: - pass # expected — the function doesn't swallow fetch exceptions - mock_cache.delete.assert_called_once_with(views.CACHE_LOCK_KEY) - - -# ============================================================================ -# get_latest_version — cache hit/miss and locking -# ============================================================================ - -@patch('Site.views.cache') -def test_get_latest_version_cache_hit_returns_immediately(mock_cache): - """On cache hit, the cached value is returned and no thread is spawned""" - mock_cache.get.return_value = '9.9.9' - - result = views.get_latest_version() - - assert result == '9.9.9' - # cache.add should NOT be called (no lock acquisition needed) - mock_cache.add.assert_not_called() - - -@patch('Site.views.threading.Thread') -@patch('Site.views.cache') -def test_get_latest_version_cache_miss_lock_acquired_spawns_thread(mock_cache, mock_thread): - """On cache miss, when lock is acquired, a background thread is started""" - mock_cache.get.return_value = None # cache miss - mock_cache.add.return_value = True # lock acquired - - mock_thread_instance = mock_thread.return_value - - result = views.get_latest_version() - - assert result is None # returns None synchronously while thread runs - mock_thread.assert_called_once() - mock_thread_instance.start.assert_called_once() - - -@patch('Site.views.threading.Thread') -@patch('Site.views.cache') -def test_get_latest_version_cache_miss_lock_not_acquired_no_thread(mock_cache, mock_thread): - """On cache miss, when lock is already held, no thread is spawned""" - mock_cache.get.return_value = None # cache miss - mock_cache.add.return_value = False # lock already held - - result = views.get_latest_version() - - assert result is None - mock_thread.assert_not_called() - - -# ============================================================================ -# check_for_update — additional edge cases -# ============================================================================ - -@patch('Site.views.settings') -def test_check_for_update_no_version_setting_returns_none(mock_settings): - """When __VERSION__ is not in settings, returns None""" - del mock_settings.__VERSION__ # simulate missing attribute - - result = views.check_for_update() - assert result is None - - -@patch('Site.views.get_latest_version', return_value=None) -@patch('Site.views.settings') -def test_check_for_update_no_latest_returns_none(mock_settings, mock_glv): - """When latest version is not cached yet, returns None""" - mock_settings.__VERSION__ = '1.0.0' - - result = views.check_for_update() - assert result is None - - -@patch('Site.views.get_latest_version', return_value='1.0.0') -@patch('Site.views.settings') -def test_check_for_update_same_version_returns_none(mock_settings, mock_glv): - """When running latest version already, returns None (no update)""" - mock_settings.__VERSION__ = '1.0.0' - - result = views.check_for_update() - assert result is None - - -@patch('Site.views.get_latest_version', return_value='0.9.0') -@patch('Site.views.settings') -def test_check_for_update_running_newer_returns_none(mock_settings, mock_glv): - """When current is newer than latest (dev build), returns None""" - mock_settings.__VERSION__ = '2.0.0' - - result = views.check_for_update() - assert result is None - - -@patch('Site.views.get_latest_version', return_value='not-a-version') -@patch('Site.views.settings') -def test_check_for_update_parse_error_returns_none(mock_settings, mock_glv): - """When version parsing raises, returns None gracefully""" - mock_settings.__VERSION__ = 'also-not-a-version' - - # packaging will raise on truly invalid strings - result = views.check_for_update() - # Either None (graceful error handling) or a valid dict — no exception - assert result is None or isinstance(result, dict) - - -@patch('Site.views.get_latest_version', return_value='5.0.0') -@patch('Site.views.settings') -def test_check_for_update_result_contains_expected_keys(mock_settings, mock_glv): - """The returned dict has all expected keys""" - mock_settings.__VERSION__ = '1.0.0' - - result = views.check_for_update() - - assert result is not None - assert set(result.keys()) == {'current_version', 'latest_version', 'update_available', 'release_url'} - assert result['current_version'] == '1.0.0' - assert result['latest_version'] == '5.0.0' - assert result['update_available'] is True - assert 'v5.0.0' in result['release_url'] - - -# ============================================================================ -# View authentication behaviour -# ============================================================================ - -@pytest.mark.django_db -def test_health_check_unauthenticated_returns_200(client): - """health_check has no login requirement — anonymous requests return 200""" - response = client.get(reverse('health_check')) - assert response.status_code == 200 - - -@pytest.mark.django_db -def test_home_unauthenticated_redirects(client): - """Home requires login — unauthenticated requests are redirected""" - response = client.get(reverse('home')) - # Should redirect to login, not return 200 - assert response.status_code == 302 - assert '/Management/Login/' in response.url - - -@pytest.mark.django_db -def test_home_uses_home_template(client, django_user_model): - """Home view renders the home.html template""" - user = django_user_model.objects.create_user(username='tmpluser', password='pass123') - client.force_login(user) - response = client.get(reverse('home')) - assert response.status_code == 200 - assert any('home.html' in t.name for t in response.templates) diff --git a/src/logstashui/Utilities/tests/__init__.py b/src/logstashui/Utilities/tests/__init__.py deleted file mode 100644 index 81800c0..0000000 --- a/src/logstashui/Utilities/tests/__init__.py +++ /dev/null @@ -1 +0,0 @@ -# Empty file to make tests a Python package diff --git a/src/logstashui/Utilities/tests/test_grok_patterns.py b/src/logstashui/Utilities/tests/test_grok_patterns.py deleted file mode 100644 index 83153f4..0000000 --- a/src/logstashui/Utilities/tests/test_grok_patterns.py +++ /dev/null @@ -1,138 +0,0 @@ -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. - -import pytest -import os -import re -import logging -from django.conf import settings - -logging.basicConfig(level=logging.INFO) -logger = logging.getLogger(__name__) - - -@pytest.fixture -def grok_patterns_file_path(): - """Path to the grok patterns file""" - utilities_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) - patterns_file = os.path.join(utilities_dir, 'data', 'grok-patterns.txt') - - if not os.path.exists(patterns_file): - patterns_file = os.path.join(utilities_dir, 'grok-patterns') - - if not os.path.exists(patterns_file): - patterns_file = os.path.join(utilities_dir, 'static', 'grok-patterns') - - return patterns_file - - -@pytest.mark.django_db -class TestGrokPatternsFile: - """Tests for the grok-patterns.txt file""" - - def test_grok_patterns_file_exists(self, grok_patterns_file_path): - """Verify grok-patterns.txt file exists""" - assert os.path.exists(grok_patterns_file_path), \ - f"Grok patterns file should exist at {grok_patterns_file_path}" - - def test_grok_patterns_file_readable(self, grok_patterns_file_path): - """Verify file is readable""" - try: - with open(grok_patterns_file_path, 'r', encoding='utf-8') as f: - content = f.read() - assert len(content) > 0, "Grok patterns file should not be empty" - except Exception as e: - pytest.fail(f"Failed to read grok patterns file: {e}") - - def test_grok_patterns_file_format(self, grok_patterns_file_path): - """Verify all patterns follow correct format""" - with open(grok_patterns_file_path, 'r', encoding='utf-8') as f: - for line_num, line in enumerate(f, 1): - line = line.strip() - - # Skip empty lines and comments - if not line or line.startswith('#'): - continue - - # Pattern should be: NAME definition - if ' ' not in line: - pytest.fail( - f"Line {line_num} is malformed (no space separator): {line}" - ) - - parts = line.split(None, 1) - if len(parts) != 2: - pytest.fail( - f"Line {line_num} is malformed (expected 2 parts): {line}" - ) - - pattern_name, pattern_def = parts - - # Pattern name should be uppercase alphanumeric with underscores - if not re.match(r'^[A-Z0-9_]+$', pattern_name): - pytest.fail( - f"Line {line_num} has invalid pattern name '{pattern_name}': " - f"should be uppercase alphanumeric with underscores" - ) - - def test_grok_patterns_no_duplicates(self, grok_patterns_file_path): - """Verify no duplicate pattern names""" - pattern_names = [] - - with open(grok_patterns_file_path, 'r', encoding='utf-8') as f: - for line in f: - line = line.strip() - if not line or line.startswith('#'): - continue - - if ' ' in line: - pattern_name = line.split(None, 1)[0] - pattern_names.append(pattern_name) - - duplicates = [name for name in pattern_names if pattern_names.count(name) > 1] - duplicates = list(set(duplicates)) - - assert len(duplicates) == 0, \ - f"Found duplicate pattern names: {duplicates}" - - def test_grok_patterns_contains_essential_patterns(self, grok_patterns_file_path): - """Verify essential patterns are present""" - essential_patterns = [ - 'USERNAME', 'USER', 'INT', 'NUMBER', 'WORD', - 'NOTSPACE', 'SPACE', 'DATA', 'GREEDYDATA', 'IP', 'IPV4' - ] - - with open(grok_patterns_file_path, 'r', encoding='utf-8') as f: - content = f.read() - - missing_patterns = [] - for pattern in essential_patterns: - # Look for pattern at start of line (with word boundary) - if not re.search(rf'^{pattern}\s', content, re.MULTILINE): - missing_patterns.append(pattern) - - assert len(missing_patterns) == 0, \ - f"Missing essential patterns: {missing_patterns}" - - def test_grok_patterns_encoding(self, grok_patterns_file_path): - """Verify file uses UTF-8 encoding""" - try: - with open(grok_patterns_file_path, 'r', encoding='utf-8') as f: - f.read() - except UnicodeDecodeError: - pytest.fail("Grok patterns file should be UTF-8 encoded") - - def test_grok_patterns_no_trailing_whitespace(self, grok_patterns_file_path): - """Verify no lines have trailing whitespace (code quality check)""" - lines_with_trailing_ws = [] - - with open(grok_patterns_file_path, 'r', encoding='utf-8') as f: - for line_num, line in enumerate(f, 1): - # Check for trailing whitespace (but not newlines) - if line.rstrip('\r\n') != line.rstrip(): - lines_with_trailing_ws.append(line_num) - - # This is a soft check - we'll warn but not fail - if lines_with_trailing_ws: - logger.warning(f"Lines with trailing whitespace: {lines_with_trailing_ws[:10]}") diff --git a/src/logstashui/Utilities/tests/test_views.py b/src/logstashui/Utilities/tests/test_views.py deleted file mode 100644 index ff3bff2..0000000 --- a/src/logstashui/Utilities/tests/test_views.py +++ /dev/null @@ -1,672 +0,0 @@ -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. - -import pytest -from django.test import RequestFactory, Client -from django.http import JsonResponse, HttpResponse -from Utilities.views import ( - GrokDebugger, - get_grok_patterns, - simulate_grok, - generate_results_html -) -import json -import os -from django.conf import settings -from Common.test_resources import request_factory, authenticated_client, test_user, client - - -@pytest.fixture -def sample_log_data(): - """Sample log data for testing grok patterns""" - return { - 'simple': '192.168.1.1 - - [01/Jan/2024:12:00:00 +0000] "GET /index.html HTTP/1.1" 200 1234', - 'multiline': 'Line 1\nLine 2\nLine 3', - 'special_chars': '', - 'unicode': 'User José logged in from München' - } - - -@pytest.fixture -def custom_patterns(): - """Custom grok pattern definitions for testing""" - return r"""CUSTOM_EMAIL [A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,} -CUSTOM_DATE \d{4}-\d{2}-\d{2}""" - - -@pytest.fixture -def grok_patterns_file_path(): - """Path to the grok patterns file""" - utilities_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) - patterns_file = os.path.join(utilities_dir, 'data', 'grok-patterns.txt') - - if not os.path.exists(patterns_file): - patterns_file = os.path.join(utilities_dir, 'grok-patterns') - - if not os.path.exists(patterns_file): - patterns_file = os.path.join(utilities_dir, 'static', 'grok-patterns') - - return patterns_file - - -@pytest.mark.django_db -class TestGrokDebuggerView: - """Tests for the main Grok Debugger view""" - - def test_grok_debugger_renders_template(self, request_factory): - """Test that GrokDebugger view renders the correct template""" - request = request_factory.get('/Utilities/GrokDebugger/') - response = GrokDebugger(request) - - assert response.status_code == 200 - - def test_grok_debugger_get_request(self, authenticated_client): - """Test GET request to Grok Debugger""" - response = authenticated_client.get('/Utilities/GrokDebugger/') - assert response.status_code == 200 - - -@pytest.mark.django_db -class TestGetGrokPatternsView: - """Tests for get_grok_patterns view""" - - def test_get_grok_patterns_success(self, request_factory, grok_patterns_file_path): - """Test successful loading of grok patterns""" - request = request_factory.get('/Utilities/GrokDebugger/patterns/') - response = get_grok_patterns(request) - - assert response.status_code == 200 - assert isinstance(response, JsonResponse) - - data = json.loads(response.content) - assert 'patterns' in data - assert isinstance(data['patterns'], dict) - assert len(data['patterns']) > 0 - - def test_get_grok_patterns_contains_common_patterns(self, request_factory): - """Test that common patterns are present""" - request = request_factory.get('/Utilities/GrokDebugger/patterns/') - response = get_grok_patterns(request) - - data = json.loads(response.content) - patterns = data['patterns'] - - # Check for some common patterns - common_patterns = ['USERNAME', 'IP', 'WORD', 'NUMBER', 'DATA'] - for pattern in common_patterns: - assert pattern in patterns, f"Pattern {pattern} should be in grok patterns" - - def test_get_grok_patterns_file_exists(self, grok_patterns_file_path): - """Test that the grok patterns file exists""" - assert os.path.exists(grok_patterns_file_path), "Grok patterns file should exist" - - -@pytest.mark.django_db -class TestSimulateGrokView: - """Tests for simulate_grok view""" - - def test_simulate_grok_single_line_match(self, request_factory, sample_log_data): - """Test successful pattern matching on single line""" - request = request_factory.post('/Utilities/GrokDebugger/simulate/', { - 'sample_data': sample_log_data['simple'], - 'grok_pattern': '%{IP:client_ip}', - 'custom_patterns': '', - 'multiline_mode': 'false' - }) - - response = simulate_grok(request) - assert response.status_code == 200 - assert isinstance(response, HttpResponse) - - content = response.content.decode('utf-8') - assert '192.168.1.1' in content - assert 'Match Found' in content or 'success' in content.lower() - - def test_simulate_grok_single_line_no_match(self, request_factory): - """Test pattern that doesn't match input""" - request = request_factory.post('/Utilities/GrokDebugger/simulate/', { - 'sample_data': 'This is plain text', - 'grok_pattern': '%{IP:client_ip}', - 'custom_patterns': '', - 'multiline_mode': 'false' - }) - - response = simulate_grok(request) - assert response.status_code == 200 - - content = response.content.decode('utf-8') - assert 'No Match' in content or 'did not match' in content.lower() - - def test_simulate_grok_multiline_mode(self, request_factory, sample_log_data): - """Test multiline mode treats entire input as single string""" - request = request_factory.post('/Utilities/GrokDebugger/simulate/', { - 'sample_data': sample_log_data['multiline'], - 'grok_pattern': '%{GREEDYDATA:message}', - 'custom_patterns': '', - 'multiline_mode': 'true' - }) - - response = simulate_grok(request) - assert response.status_code == 200 - - content = response.content.decode('utf-8') - # In multiline mode, should treat as one input - assert 'Line 1' in content - - def test_simulate_grok_multiple_patterns(self, request_factory): - """Test multiple patterns against single input""" - request = request_factory.post('/Utilities/GrokDebugger/simulate/', { - 'sample_data': '192.168.1.1', - 'grok_pattern': '%{IP:ip}\n%{WORD:word}', - 'custom_patterns': '', - 'multiline_mode': 'false' - }) - - response = simulate_grok(request) - assert response.status_code == 200 - - content = response.content.decode('utf-8') - # Should show results for Pattern 1 and Pattern 2 - assert 'Pattern 1' in content - assert 'Pattern 2' in content - - def test_simulate_grok_custom_patterns(self, request_factory, custom_patterns): - """Test with custom pattern definitions""" - request = request_factory.post('/Utilities/GrokDebugger/simulate/', { - 'sample_data': 'test@example.com', - 'grok_pattern': '%{CUSTOM_EMAIL:email}', - 'custom_patterns': custom_patterns, - 'multiline_mode': 'false' - }) - - response = simulate_grok(request) - assert response.status_code == 200 - - content = response.content.decode('utf-8') - assert 'test@example.com' in content - - def test_simulate_grok_dot_notation_fields(self, request_factory): - """Test field names with dots create nested dictionaries""" - request = request_factory.post('/Utilities/GrokDebugger/simulate/', { - 'sample_data': '192.168.1.1', - 'grok_pattern': '%{IP:client.ip.address}', - 'custom_patterns': '', - 'multiline_mode': 'false' - }) - - response = simulate_grok(request) - assert response.status_code == 200 - - content = response.content.decode('utf-8') - # Should show nested structure - assert 'client' in content - assert '192.168.1.1' in content - - def test_simulate_grok_pattern_compilation_error(self, request_factory): - """Test invalid grok pattern syntax""" - request = request_factory.post('/Utilities/GrokDebugger/simulate/', { - 'sample_data': 'test data', - 'grok_pattern': '%{INVALID_PATTERN_THAT_DOES_NOT_EXIST:field}', - 'custom_patterns': '', - 'multiline_mode': 'false' - }) - - response = simulate_grok(request) - assert response.status_code == 200 - - content = response.content.decode('utf-8') - assert 'error' in content.lower() or 'compilation' in content.lower() - - def test_simulate_grok_empty_sample_data(self, request_factory): - """Test with empty sample data""" - request = request_factory.post('/Utilities/GrokDebugger/simulate/', { - 'sample_data': '', - 'grok_pattern': '%{IP:ip}', - 'custom_patterns': '', - 'multiline_mode': 'false' - }) - - response = simulate_grok(request) - assert response.status_code == 200 - - def test_simulate_grok_empty_pattern(self, request_factory): - """Test with empty pattern""" - request = request_factory.post('/Utilities/GrokDebugger/simulate/', { - 'sample_data': 'test data', - 'grok_pattern': '', - 'custom_patterns': '', - 'multiline_mode': 'false' - }) - - response = simulate_grok(request) - assert response.status_code == 200 - - def test_simulate_grok_invalid_request_method(self, request_factory): - """Test GET request to simulate endpoint (should only accept POST)""" - request = request_factory.get('/Utilities/GrokDebugger/simulate/') - response = simulate_grok(request) - - assert response.status_code == 200 - content = response.content.decode('utf-8') - assert 'Invalid request method' in content - - def test_simulate_grok_special_characters(self, request_factory, sample_log_data): - """Test handling of special characters and potential XSS""" - request = request_factory.post('/Utilities/GrokDebugger/simulate/', { - 'sample_data': sample_log_data['special_chars'], - 'grok_pattern': '%{GREEDYDATA:data}', - 'custom_patterns': '', - 'multiline_mode': 'false' - }) - - response = simulate_grok(request) - assert response.status_code == 200 - - content = response.content.decode('utf-8') - # Check that HTML is escaped - assert '<script>' in content or '', - 'pattern_number': 1, - 'matches': [{ - 'line_number': 1, - 'sample': '', - 'success': False, - 'error': '' - }] - }] - - html = generate_results_html(results) - - # All user input should be escaped - assert '<script>' in html - assert '<img' in html - - def test_generate_results_html_nested_data(self): - """Test HTML generation with nested parsed data""" - results = [{ - 'pattern': '%{IP:client.ip}', - 'pattern_number': 1, - 'matches': [{ - 'line_number': 1, - 'sample': '192.168.1.1', - 'success': True, - 'parsed_data': {'client': {'ip': '192.168.1.1'}} - }] - }] - - html = generate_results_html(results) - - assert 'client' in html - assert '192.168.1.1' in html - - -@pytest.mark.django_db -class TestGrokDebuggerIntegration: - """Integration tests for the full Grok Debugger workflow""" - - def test_full_workflow_simple_pattern(self, authenticated_client): - """Test complete workflow from page load to simulation""" - # Load the page - response = authenticated_client.get('/Utilities/GrokDebugger/') - assert response.status_code == 200 - - # Simulate a grok pattern - response = authenticated_client.post('/Utilities/GrokDebugger/simulate/', { - 'sample_data': '192.168.1.1', - 'grok_pattern': '%{IP:ip}', - 'custom_patterns': '', - 'multiline_mode': 'false' - }) - assert response.status_code == 200 - assert b'192.168.1.1' in response.content - - def test_full_workflow_with_custom_patterns(self, authenticated_client, custom_patterns): - """Test workflow with custom patterns""" - response = authenticated_client.post('/Utilities/GrokDebugger/simulate/', { - 'sample_data': '2024-01-15', - 'grok_pattern': '%{CUSTOM_DATE:date}', - 'custom_patterns': custom_patterns, - 'multiline_mode': 'false' - }) - assert response.status_code == 200 - assert b'2024-01-15' in response.content - - -# ============================================================================ -# Additional gap-filling tests -# ============================================================================ - -@pytest.mark.django_db -class TestGetGrokPatternsErrors: - """Test error-handling branches in get_grok_patterns""" - - def test_get_grok_patterns_file_missing_returns_500(self, request_factory): - """When the grok-patterns file does not exist, the view returns 500 with an error key""" - from unittest.mock import patch - request = request_factory.get('/Utilities/GrokDebugger/patterns/') - - with patch('Utilities.views.open', side_effect=FileNotFoundError("no such file")): - response = get_grok_patterns(request) - - assert response.status_code == 500 - data = json.loads(response.content) - assert 'error' in data - - def test_get_grok_patterns_skips_comment_and_blank_lines(self, request_factory): - """Lines starting with # or blank lines are not included as patterns""" - from unittest.mock import patch, mock_open - fake_content = "# This is a comment\n\nWORD \\b\\w+\\b\n" - request = request_factory.get('/Utilities/GrokDebugger/patterns/') - - with patch('builtins.open', mock_open(read_data=fake_content)): - response = get_grok_patterns(request) - - data = json.loads(response.content) - patterns = data['patterns'] - # Only WORD should be loaded; the comment and blank line must be absent - assert 'WORD' in patterns - for key in patterns: - assert not key.startswith('#') - - def test_get_grok_patterns_skips_lines_without_space(self, request_factory): - """Lines with no whitespace (can't be split into name + definition) are silently skipped""" - from unittest.mock import patch, mock_open - fake_content = "BADLINE\nGOOD pattern_def\n" - request = request_factory.get('/Utilities/GrokDebugger/patterns/') - - with patch('builtins.open', mock_open(read_data=fake_content)): - response = get_grok_patterns(request) - - data = json.loads(response.content) - patterns = data['patterns'] - assert 'GOOD' in patterns - assert 'BADLINE' not in patterns - - -@pytest.mark.django_db -class TestSimulateGrokAdditional: - """Additional simulate_grok edge-case tests""" - - def test_whitespace_only_lines_filtered_from_sample(self, request_factory): - """Whitespace-only sample lines are filtered out before matching""" - request = request_factory.post('/Utilities/GrokDebugger/simulate/', { - 'sample_data': ' \n \t ', - 'grok_pattern': '%{IP:ip}', - 'custom_patterns': '', - 'multiline_mode': 'false' - }) - response = simulate_grok(request) - assert response.status_code == 200 - # No sample lines to process → HTML has no match results - content = response.content.decode('utf-8') - assert 'Match Found' not in content - assert 'No Match' not in content - - def test_whitespace_only_pattern_lines_filtered(self, request_factory): - """Whitespace-only pattern lines are filtered; result is empty HTML""" - request = request_factory.post('/Utilities/GrokDebugger/simulate/', { - 'sample_data': '192.168.1.1', - 'grok_pattern': ' \n\t', - 'custom_patterns': '', - 'multiline_mode': 'false' - }) - response = simulate_grok(request) - assert response.status_code == 200 - # No patterns → empty body (no Pattern N headings) - content = response.content.decode('utf-8') - assert 'Pattern 1' not in content - - def test_custom_patterns_blank_and_comment_lines_ignored(self, request_factory): - """Blank lines and lines without a space in custom_patterns are silently skipped""" - custom = "# comment\n\nMY_IP (?:\\d{1,3}\\.){3}\\d{1,3}\n" - request = request_factory.post('/Utilities/GrokDebugger/simulate/', { - 'sample_data': '10.0.0.1', - 'grok_pattern': '%{MY_IP:ip}', - 'custom_patterns': custom, - 'multiline_mode': 'false' - }) - response = simulate_grok(request) - assert response.status_code == 200 - content = response.content.decode('utf-8') - # MY_IP should have been parsed; match should succeed - assert 'Match Found' in content - - def test_multiline_mode_false_splits_on_newlines(self, request_factory): - """When multiline_mode is false, each non-blank line is treated independently""" - request = request_factory.post('/Utilities/GrokDebugger/simulate/', { - 'sample_data': '192.168.1.1\n10.0.0.1', - 'grok_pattern': '%{IP:ip}', - 'custom_patterns': '', - 'multiline_mode': 'false' - }) - response = simulate_grok(request) - assert response.status_code == 200 - content = response.content.decode('utf-8') - # Both IPs should appear in Line 1 and Line 2 results - assert 'Line 1' in content - assert 'Line 2' in content - assert '192.168.1.1' in content - assert '10.0.0.1' in content - - def test_multiline_mode_true_no_split(self, request_factory): - """When multiline_mode is true, the two-line input is treated as a single chunk""" - request = request_factory.post('/Utilities/GrokDebugger/simulate/', { - 'sample_data': '192.168.1.1\n10.0.0.1', - 'grok_pattern': '%{GREEDYDATA:msg}', - 'custom_patterns': '', - 'multiline_mode': 'true' - }) - response = simulate_grok(request) - assert response.status_code == 200 - content = response.content.decode('utf-8') - # Only one entry (Line 1); Line 2 label must not appear - assert 'Line 2' not in content - - -@pytest.mark.django_db -class TestGenerateResultsHtmlAdditional: - """Additional generate_results_html tests""" - - def test_empty_results_list_returns_empty_string(self): - """No results → empty string (no crash, no stray HTML)""" - output = generate_results_html([]) - assert output == '' - - def test_pattern_error_field_shown_in_html(self): - """When a result has pattern_error set, the error information is included""" - results = [{ - 'pattern': '%{BAD_PATTERN:x}', - 'pattern_number': 1, - 'pattern_error': 'Undefined pattern: BAD_PATTERN', - 'matches': [{ - 'line_number': 1, - 'sample': 'anything', - 'success': False, - 'error': 'Pattern compilation error: Undefined pattern: BAD_PATTERN', - 'error_type': 'compilation' - }] - }] - output = generate_results_html(results) - # The pattern header and the failed match entry should both be present - assert 'Pattern 1' in output - assert 'No Match' in output - assert 'compilation' in output.lower() or 'Pattern compilation' in output - - def test_zero_matched_badge_correct(self): - """Badge shows 0 matched when all lines fail""" - results = [{ - 'pattern': '%{IP:ip}', - 'pattern_number': 1, - 'matches': [ - {'line_number': 1, 'sample': 'hello', 'success': False, 'error': 'no match'}, - {'line_number': 2, 'sample': 'world', 'success': False, 'error': 'no match'}, - ] - }] - output = generate_results_html(results) - assert '0 matched' in output - assert '2 failed' in output - - def test_all_matched_badge_correct(self): - """Badge shows 0 failed when all lines succeed""" - results = [{ - 'pattern': '%{IP:ip}', - 'pattern_number': 1, - 'matches': [ - {'line_number': 1, 'sample': '1.1.1.1', 'success': True, 'parsed_data': {'ip': '1.1.1.1'}}, - {'line_number': 2, 'sample': '2.2.2.2', 'success': True, 'parsed_data': {'ip': '2.2.2.2'}}, - ] - }] - output = generate_results_html(results) - assert '2 matched' in output - assert '0 failed' in output - - def test_html_escape_in_error_message(self): - """Error messages containing HTML special chars are escaped""" - results = [{ - 'pattern': 'p', - 'pattern_number': 1, - 'matches': [{ - 'line_number': 1, - 'sample': 'x', - 'success': False, - 'error': 'bad & "error"' - }] - }] - output = generate_results_html(results) - assert '' not in output # raw tag must not appear - assert '<b>' in output # escaped version must appear - - -@pytest.mark.django_db -class TestAuthenticationAndRouting: - """URL-level authentication and routing tests""" - - def test_grok_debugger_requires_authentication(self, client): - """Unauthenticated request to GrokDebugger redirects to login""" - response = client.get('/Utilities/GrokDebugger/') - assert response.status_code == 302 - assert '/Management/Login/' in response.url - - def test_simulate_grok_requires_authentication(self, client): - """Unauthenticated POST to simulate endpoint redirects to login""" - response = client.post('/Utilities/GrokDebugger/simulate/', { - 'sample_data': '192.168.1.1', - 'grok_pattern': '%{IP:ip}', - }) - assert response.status_code == 302 - assert '/Management/Login/' in response.url - - def test_get_grok_patterns_requires_authentication(self, client): - """Unauthenticated GET to patterns endpoint redirects to login""" - response = client.get('/Utilities/GrokDebugger/patterns/') - assert response.status_code == 302 - assert '/Management/Login/' in response.url diff --git a/tests/integration/__init__.py b/tests/integration/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/tests/integration/conftest.py b/tests/integration/conftest.py deleted file mode 100644 index 1ae0172..0000000 --- a/tests/integration/conftest.py +++ /dev/null @@ -1,244 +0,0 @@ -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. - -import os -import shutil -import subprocess -import sys -import uuid - -import pytest - - -# --------------------------------------------------------------------------- -# Docker availability — checked once at module import time -# --------------------------------------------------------------------------- - -def _check_docker() -> tuple[bool, str]: - if not shutil.which("docker"): - return False, "docker binary not found in PATH" - try: - r = subprocess.run( - ["docker", "info"], - capture_output=True, - timeout=10, - ) - if r.returncode != 0: - return False, ( - f"docker info returned {r.returncode}: " - f"{r.stderr.decode()[:200]}" - ) - return True, "" - except (subprocess.TimeoutExpired, OSError) as exc: - return False, str(exc) - - -_DOCKER_OK, _DOCKER_REASON = _check_docker() - - -@pytest.fixture(scope="session", autouse=True) -def skip_if_no_docker(): - """Skip every test in the integration suite when Docker is unavailable.""" - if not _DOCKER_OK: - pytest.skip(f"Docker not available: {_DOCKER_REASON}") - - -# --------------------------------------------------------------------------- -# Container fixtures (session scope — start once, reused across all tests) -# --------------------------------------------------------------------------- - -@pytest.fixture(scope="session") -def postgres_container(): - from testcontainers.postgres import PostgresContainer - - with PostgresContainer( - image="postgres:16", - username="logstashui", - password="logstashui", - dbname="logstashui_test", - ) as c: - yield c - - -@pytest.fixture(scope="session") -def mysql_container(): - from testcontainers.mysql import MySqlContainer - - c = MySqlContainer( - image="mysql:8.0", - root_password="logstashui", - dbname="logstashui_test", - ) - c.with_command( - "--character-set-server=utf8mb4 --collation-server=utf8mb4_bin" - ) - with c: - yield c - - -@pytest.fixture(scope="session") -def mariadb_container(): - from testcontainers.mysql import MySqlContainer - - c = MySqlContainer( - image="mariadb:11", - root_password="logstashui", - dbname="logstashui_test", - ) - c.with_command( - "--character-set-server=utf8mb4 --collation-server=utf8mb4_bin" - ) - with c: - yield c - - -# --------------------------------------------------------------------------- -# Env-dict helpers (module-level, not fixtures — importable by test files) -# --------------------------------------------------------------------------- - -def pg_env(container, *, dbname: str = "logstashui_test") -> dict[str, str]: - """Return LOGSTASHUI_DB_* env dict for a PostgreSQL container.""" - return { - "LOGSTASHUI_DB_ENGINE": "postgresql", - "LOGSTASHUI_DB_HOST": container.get_container_host_ip(), - "LOGSTASHUI_DB_PORT": str(container.get_exposed_port(5432)), - "LOGSTASHUI_DB_USER": "logstashui", - "LOGSTASHUI_DB_PASSWORD": "logstashui", - "LOGSTASHUI_DB_NAME": dbname, - } - - -def mysql_env(container, *, dbname: str = "logstashui_test") -> dict[str, str]: - """Return LOGSTASHUI_DB_* env dict for a MySQL/MariaDB container (root user).""" - return { - "LOGSTASHUI_DB_ENGINE": "mysql", - "LOGSTASHUI_DB_HOST": container.get_container_host_ip(), - "LOGSTASHUI_DB_PORT": str(container.get_exposed_port(3306)), - "LOGSTASHUI_DB_USER": "root", - "LOGSTASHUI_DB_PASSWORD": "logstashui", - "LOGSTASHUI_DB_NAME": dbname, - } - - -def new_dbname() -> str: - """Generate a unique database name for test isolation.""" - return f"logstashui_{uuid.uuid4().hex[:8]}" - - -# --------------------------------------------------------------------------- -# Fresh-database helpers (create/drop via native drivers) -# --------------------------------------------------------------------------- - -def create_pg_db(base_env: dict[str, str], dbname: str) -> None: - """Create *dbname* in the PostgreSQL container reachable via *base_env*.""" - import psycopg - - connstr = ( - f"host={base_env['LOGSTASHUI_DB_HOST']} " - f"port={base_env['LOGSTASHUI_DB_PORT']} " - f"user={base_env['LOGSTASHUI_DB_USER']} " - f"password={base_env['LOGSTASHUI_DB_PASSWORD']} " - f"dbname={base_env['LOGSTASHUI_DB_NAME']}" - ) - with psycopg.connect(connstr, autocommit=True) as conn: - conn.execute(f'CREATE DATABASE "{dbname}"') - - -def drop_pg_db(base_env: dict[str, str], dbname: str) -> None: - import psycopg - - connstr = ( - f"host={base_env['LOGSTASHUI_DB_HOST']} " - f"port={base_env['LOGSTASHUI_DB_PORT']} " - f"user={base_env['LOGSTASHUI_DB_USER']} " - f"password={base_env['LOGSTASHUI_DB_PASSWORD']} " - f"dbname={base_env['LOGSTASHUI_DB_NAME']}" - ) - with psycopg.connect(connstr, autocommit=True) as conn: - conn.execute(f'DROP DATABASE IF EXISTS "{dbname}"') - - -def create_mysql_db(base_env: dict[str, str], dbname: str) -> None: - """Create *dbname* with utf8mb4/utf8mb4_bin in the MySQL/MariaDB container.""" - import pymysql - - conn = pymysql.connect( - host=base_env["LOGSTASHUI_DB_HOST"], - port=int(base_env["LOGSTASHUI_DB_PORT"]), - user=base_env["LOGSTASHUI_DB_USER"], - password=base_env["LOGSTASHUI_DB_PASSWORD"], - autocommit=True, - ) - try: - with conn.cursor() as cur: - cur.execute( - f"CREATE DATABASE `{dbname}` " - f"CHARACTER SET utf8mb4 COLLATE utf8mb4_bin" - ) - finally: - conn.close() - - -def drop_mysql_db(base_env: dict[str, str], dbname: str) -> None: - import pymysql - - conn = pymysql.connect( - host=base_env["LOGSTASHUI_DB_HOST"], - port=int(base_env["LOGSTASHUI_DB_PORT"]), - user=base_env["LOGSTASHUI_DB_USER"], - password=base_env["LOGSTASHUI_DB_PASSWORD"], - autocommit=True, - ) - try: - with conn.cursor() as cur: - cur.execute(f"DROP DATABASE IF EXISTS `{dbname}`") - finally: - conn.close() - - -# --------------------------------------------------------------------------- -# Parametrized engine fixture (postgres + mysql) -# --------------------------------------------------------------------------- - -@pytest.fixture(params=["postgres", "mysql"]) -def engine_env(request, postgres_container, mysql_container): - """Yield (engine_name, env_dict) for each supported engine.""" - if request.param == "postgres": - yield "postgresql", pg_env(postgres_container) - else: - yield "mysql", mysql_env(mysql_container) - - -# --------------------------------------------------------------------------- -# Fresh per-test database fixture (for migration / round-trip tests) -# --------------------------------------------------------------------------- - -@pytest.fixture -def fresh_db_env(engine_env, tmp_path): - """ - Yield (engine_name, env_dict) with a unique throwaway database and - LOGSTASHUI_DATA_DIR set. The database is created before the test and - dropped afterwards. - """ - engine, base_env = engine_env - dbname = new_dbname() - - if engine == "postgresql": - create_pg_db(base_env, dbname) - full_env = { - **base_env, - "LOGSTASHUI_DB_NAME": dbname, - "LOGSTASHUI_DATA_DIR": str(tmp_path), - } - yield engine, full_env - drop_pg_db(base_env, dbname) - else: - create_mysql_db(base_env, dbname) - full_env = { - **base_env, - "LOGSTASHUI_DB_NAME": dbname, - "LOGSTASHUI_DATA_DIR": str(tmp_path), - } - yield engine, full_env - drop_mysql_db(base_env, dbname) diff --git a/tests/integration/test_db_config.py b/tests/integration/test_db_config.py deleted file mode 100644 index 0aed525..0000000 --- a/tests/integration/test_db_config.py +++ /dev/null @@ -1,166 +0,0 @@ -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. - -""" -Integration tests for database configuration and server version checking. -All tests that open real connections use subprocesses so Django settings -are configured in an isolated interpreter with the container env vars. -""" - -import os -import subprocess -import sys - -import pytest - -from LogstashUI.database import build_databases - - -# --------------------------------------------------------------------------- -# Subprocess helper -# --------------------------------------------------------------------------- - -def _run_python(code: str, extra_env: dict[str, str]) -> str: - from LogstashUI import migrate_engine as me - - env = os.environ.copy() - env.update(extra_env) - env.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") - me._with_package_pythonpath(env) - proc = subprocess.run( - [sys.executable, "-c", code], - env=env, - check=False, - capture_output=True, - text=True, - ) - if proc.returncode != 0: - raise AssertionError( - f"python -c exited {proc.returncode}\n" - f"stdout:\n{proc.stdout}\nstderr:\n{proc.stderr}" - ) - return proc.stdout - - -# --------------------------------------------------------------------------- -# Inline scripts -# --------------------------------------------------------------------------- - -_CHECK_SERVER_VERSION = """ -import os -import django -os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") -django.setup() -from django.db import connection -from LogstashUI.database import check_server_version -connection.ensure_connection() -check_server_version(connection) -print("OK") -""" - -_ENSURE_CONNECTION = """ -import os -import django -os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") -django.setup() -from django.db import connection -connection.ensure_connection() -assert connection.connection is not None, "connection is None" -print("OK") -""" - - -# --------------------------------------------------------------------------- -# Tests — build_databases() dict structure (no Docker needed) -# --------------------------------------------------------------------------- - -def test_mysql_options_include_utf8mb4(monkeypatch, tmp_path): - """build_databases() for MySQL must include utf8mb4 charset and utf8mb4_bin collation.""" - for key in ( - "LOGSTASHUI_DB_ENGINE", "LOGSTASHUI_DB_HOST", "LOGSTASHUI_DB_PORT", - "LOGSTASHUI_DB_USER", "LOGSTASHUI_DB_PASSWORD", "LOGSTASHUI_DB_NAME", - "LOGSTASHUI_DB_CONN_MAX_AGE", - ): - monkeypatch.delenv(key, raising=False) - monkeypatch.setenv("LOGSTASHUI_DB_ENGINE", "mysql") - monkeypatch.setenv("LOGSTASHUI_DB_HOST", "127.0.0.1") - monkeypatch.setenv("LOGSTASHUI_DB_USER", "root") - monkeypatch.setattr("LogstashUI.database._import_or_raise", lambda *a, **k: _fake_pymysql()) - db = build_databases(tmp_path)["default"] - assert db["OPTIONS"]["charset"] == "utf8mb4" - assert "utf8mb4_bin" in db["OPTIONS"]["init_command"] - assert db["TEST"]["CHARSET"] == "utf8mb4" - assert db["TEST"]["COLLATION"] == "utf8mb4_bin" - - -def _fake_pymysql(): - from types import SimpleNamespace - fake = SimpleNamespace( - version_info=(1, 1, 1, "final", 0), - install_as_MySQLdb=lambda: None, - ) - return fake - - -def test_conn_max_age_applied(monkeypatch, tmp_path): - """LOGSTASHUI_DB_CONN_MAX_AGE overrides the default 60s for postgres.""" - for key in ( - "LOGSTASHUI_DB_ENGINE", "LOGSTASHUI_DB_HOST", "LOGSTASHUI_DB_PORT", - "LOGSTASHUI_DB_USER", "LOGSTASHUI_DB_PASSWORD", "LOGSTASHUI_DB_NAME", - "LOGSTASHUI_DB_CONN_MAX_AGE", - ): - monkeypatch.delenv(key, raising=False) - monkeypatch.setenv("LOGSTASHUI_DB_ENGINE", "postgresql") - monkeypatch.setenv("LOGSTASHUI_DB_HOST", "db.example") - monkeypatch.setenv("LOGSTASHUI_DB_USER", "lsui") - monkeypatch.setenv("LOGSTASHUI_DB_CONN_MAX_AGE", "120") - monkeypatch.setattr("LogstashUI.database._import_or_raise", lambda *a, **k: None) - db = build_databases(tmp_path)["default"] - assert db["CONN_MAX_AGE"] == 120 - - -def test_build_databases_returns_valid_dict(engine_env, tmp_path, monkeypatch): - """build_databases() produces a valid DATABASES dict for each engine.""" - engine, env = engine_env - for k, v in env.items(): - monkeypatch.setenv(k, v) - monkeypatch.setenv("LOGSTASHUI_DATA_DIR", str(tmp_path)) - # For MySQL, stub _import_or_raise so it runs without the spoof side-effect - if engine == "mysql": - monkeypatch.setattr( - "LogstashUI.database._import_or_raise", - lambda *a, **k: _fake_pymysql(), - ) - db = build_databases(tmp_path)["default"] - assert "ENGINE" in db - assert "HOST" in db - assert "PORT" in db - assert "NAME" in db - assert isinstance(db["PORT"], str) - - -# --------------------------------------------------------------------------- -# Tests — real container connections (subprocess) -# --------------------------------------------------------------------------- - -def test_real_connection_opens(engine_env, tmp_path): - """Django can open a connection to the container database.""" - engine, env = engine_env - full_env = {**env, "LOGSTASHUI_DATA_DIR": str(tmp_path)} - _run_python(_ENSURE_CONNECTION, full_env) - - -def test_check_server_version_passes_on_real_connection(engine_env, tmp_path): - """check_server_version() passes without error on a real container connection.""" - engine, env = engine_env - full_env = {**env, "LOGSTASHUI_DATA_DIR": str(tmp_path)} - _run_python(_CHECK_SERVER_VERSION, full_env) - - -def test_check_server_version_mariadb_branch(mariadb_container, tmp_path): - """check_server_version() MariaDB detection branch passes on a real MariaDB server.""" - from tests.integration.conftest import mysql_env - env = mysql_env(mariadb_container) - full_env = {**env, "LOGSTASHUI_DATA_DIR": str(tmp_path)} - _run_python(_CHECK_SERVER_VERSION, full_env) diff --git a/tests/integration/test_migrate_engine.py b/tests/integration/test_migrate_engine.py deleted file mode 100644 index fab32f8..0000000 --- a/tests/integration/test_migrate_engine.py +++ /dev/null @@ -1,292 +0,0 @@ -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. - -""" -Integration tests for cmd_migrate_engine (SQLite → PostgreSQL / MySQL / MariaDB). - -Each test uses a unique throwaway database in the session-scoped container -to prevent cross-test contamination. -""" - -import json -import os -import subprocess -import sys -from argparse import Namespace - -import pytest - -from LogstashUI import migrate_engine as me -from tests.integration.conftest import ( - create_mysql_db, - create_pg_db, - drop_mysql_db, - drop_pg_db, - mysql_env, - new_dbname, - pg_env, -) - - -# --------------------------------------------------------------------------- -# Subprocess helper -# --------------------------------------------------------------------------- - -def _run_python(code: str, extra_env: dict[str, str]) -> str: - env = os.environ.copy() - env.update(extra_env) - env.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") - me._with_package_pythonpath(env) - proc = subprocess.run( - [sys.executable, "-c", code], - env=env, - check=False, - capture_output=True, - text=True, - ) - if proc.returncode != 0: - raise AssertionError( - f"python -c exited {proc.returncode}\n" - f"stdout:\n{proc.stdout}\nstderr:\n{proc.stderr}" - ) - return proc.stdout - - -# --------------------------------------------------------------------------- -# Inline scripts -# --------------------------------------------------------------------------- - -_SEED = """ -import os -import django -os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") -django.setup() -from django.contrib.auth import get_user_model -from PipelineManager.models import Connection, Policy -User = get_user_model() -User.objects.create_user(username="migrate-user", password="migrate-pass") -policy = Policy.objects.create( - name="Migrate Policy", - logstash_yml="http.host: 0.0.0.0", - jvm_options="-Xms1g", - log4j2_properties="status = error", -) -Connection.objects.create( - name="Migrate Conn", - connection_type=Connection.ConnectionType.AGENT, - host="127.0.0.1", - policy=policy, - status_blob={"health": "green", "n": 1}, -) -""" - -_COUNT = """ -import json -import os -import django -os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") -django.setup() -from django.contrib.auth import get_user_model -from PipelineManager.models import Connection, Policy -User = get_user_model() -conn = Connection.objects.filter(name="Migrate Conn").first() -print(json.dumps({ - "users": User.objects.count(), - "policies": Policy.objects.count(), - "migrate_user": User.objects.filter(username="migrate-user").count(), - "migrate_policy": Policy.objects.filter(name="Migrate Policy").count(), - "status_blob": conn.status_blob if conn else None, -})) -""" - -_POST_MIGRATE_INSERT = """ -import json -import os -import django -os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") -django.setup() -from django.contrib.auth import get_user_model -User = get_user_model() -u = User.objects.create_user(username="post-migrate-user", password="x") -assert u.pk is not None -print(json.dumps({"pk": u.pk})) -""" - -_TARGET_KEYS = ( - "LOGSTASHUI_DATA_DIR", - "LOGSTASHUI_DB_ENGINE", - "LOGSTASHUI_DB_HOST", - "LOGSTASHUI_DB_PORT", - "LOGSTASHUI_DB_USER", - "LOGSTASHUI_DB_PASSWORD", - "LOGSTASHUI_DB_NAME", -) - - -# --------------------------------------------------------------------------- -# Core migration helper -# --------------------------------------------------------------------------- - -def _run_to(tmp_path, target_env: dict[str, str]) -> dict: - """ - Seed a fresh SQLite database, run cmd_migrate_engine to the target, - then assert data counts. Returns the parsed count dict. - """ - data_dir = tmp_path - sqlite_path = data_dir / "db.sqlite3" - sqlite_env = { - "LOGSTASHUI_DATA_DIR": str(data_dir), - "LOGSTASHUI_DB_ENGINE": "sqlite", - "LOGSTASHUI_DB_NAME": str(sqlite_path), - } - me.run_manage(["migrate", "--noinput", "--verbosity", "0"], sqlite_env) - _run_python(_SEED, sqlite_env) - - full_target = {**target_env, "LOGSTASHUI_DATA_DIR": str(data_dir)} - previous = {key: os.environ.get(key) for key in _TARGET_KEYS} - try: - os.environ.update(full_target) - ns = Namespace( - to=full_target["LOGSTASHUI_DB_ENGINE"], - i_have_a_backup=True, - pid=None, - write_env=None, - ) - try: - rc = me.cmd_migrate_engine(ns) - except SystemExit as exc: - raise AssertionError( - f"cmd_migrate_engine SystemExit {exc.code}" - ) from exc - assert rc == 0 - raw = _run_python(_COUNT, full_target) - finally: - for key, value in previous.items(): - if value is None: - os.environ.pop(key, None) - else: - os.environ[key] = value - - counts = json.loads(raw.strip().splitlines()[-1]) - assert counts["users"] >= 1 - assert counts["policies"] >= 1 - assert counts["migrate_user"] == 1 - assert counts["migrate_policy"] == 1 - assert counts["status_blob"] == {"health": "green", "n": 1} - return counts - - -# --------------------------------------------------------------------------- -# Tests -# --------------------------------------------------------------------------- - -def test_migrate_engine_to_postgres(postgres_container, tmp_path): - dbname = new_dbname() - base = pg_env(postgres_container) - create_pg_db(base, dbname) - try: - target = pg_env(postgres_container, dbname=dbname) - _run_to(tmp_path, target) - finally: - drop_pg_db(base, dbname) - - -def test_migrate_engine_to_mysql(mysql_container, tmp_path): - dbname = new_dbname() - base = mysql_env(mysql_container) - create_mysql_db(base, dbname) - try: - target = mysql_env(mysql_container, dbname=dbname) - _run_to(tmp_path, target) - finally: - drop_mysql_db(base, dbname) - - -def test_migrate_engine_to_mariadb(mariadb_container, tmp_path): - dbname = new_dbname() - base = mysql_env(mariadb_container) - create_mysql_db(base, dbname) - try: - target = mysql_env(mariadb_container, dbname=dbname) - _run_to(tmp_path, target) - finally: - drop_mysql_db(base, dbname) - - -def test_sequence_reset_postgres(postgres_container, tmp_path): - """After SQLite→Postgres migration, inserting a new User does not fail on sequence.""" - dbname = new_dbname() - base = pg_env(postgres_container) - create_pg_db(base, dbname) - try: - target = pg_env(postgres_container, dbname=dbname) - _run_to(tmp_path, target) - full_env = {**target, "LOGSTASHUI_DATA_DIR": str(tmp_path)} - out = _run_python(_POST_MIGRATE_INSERT, full_env) - result = json.loads(out.strip()) - assert isinstance(result["pk"], int) and result["pk"] > 0 - finally: - drop_pg_db(base, dbname) - - -def test_migrate_engine_write_env(postgres_container, tmp_path): - """--write-env produces a file with engine/host keys but no PASSWORD.""" - dbname = new_dbname() - base = pg_env(postgres_container) - create_pg_db(base, dbname) - env_file = tmp_path / "logstashui.env" - try: - target = pg_env(postgres_container, dbname=dbname) - full_target = {**target, "LOGSTASHUI_DATA_DIR": str(tmp_path)} - sqlite_path = tmp_path / "db.sqlite3" - sqlite_env = { - "LOGSTASHUI_DATA_DIR": str(tmp_path), - "LOGSTASHUI_DB_ENGINE": "sqlite", - "LOGSTASHUI_DB_NAME": str(sqlite_path), - } - me.run_manage(["migrate", "--noinput", "--verbosity", "0"], sqlite_env) - _run_python(_SEED, sqlite_env) - previous = {key: os.environ.get(key) for key in _TARGET_KEYS} - try: - os.environ.update(full_target) - ns = Namespace( - to="postgresql", - i_have_a_backup=True, - pid=None, - write_env=str(env_file), - ) - rc = me.cmd_migrate_engine(ns) - finally: - for key, value in previous.items(): - if value is None: - os.environ.pop(key, None) - else: - os.environ[key] = value - assert rc == 0 - text = env_file.read_text() - assert "LOGSTASHUI_DB_ENGINE=postgresql" in text - assert "LOGSTASHUI_DB_HOST=" in text - assert "PASSWORD" not in text - finally: - drop_pg_db(base, dbname) - - -def test_migrate_engine_idempotent_env(postgres_container, tmp_path): - """Running write_env twice produces no duplicate keys in the output file.""" - dbname = new_dbname() - base = pg_env(postgres_container) - create_pg_db(base, dbname) - env_file = tmp_path / "logstashui.env" - try: - target = pg_env(postgres_container, dbname=dbname) - full_target = {**target, "LOGSTASHUI_DATA_DIR": str(tmp_path)} - # Run migration once - _run_to(tmp_path, target) - # write_env twice, pointing at the already-migrated DB - me.write_env_file(env_file, "postgresql") - me.write_env_file(env_file, "postgresql") - text = env_file.read_text() - assert text.count("LOGSTASHUI_DB_ENGINE=postgresql") == 1 - finally: - drop_pg_db(base, dbname) diff --git a/tests/integration/test_migrations.py b/tests/integration/test_migrations.py deleted file mode 100644 index b20b524..0000000 --- a/tests/integration/test_migrations.py +++ /dev/null @@ -1,101 +0,0 @@ -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. - -""" -Integration tests for Django migrations against real database containers. -Each parametrized test gets a fresh isolated database (created and dropped -per-test via the fresh_db_env fixture). -""" - -import os -import subprocess -import sys - -import pytest - -from LogstashUI import migrate_engine as me - - -# --------------------------------------------------------------------------- -# Subprocess helper -# --------------------------------------------------------------------------- - -def _run_python(code: str, extra_env: dict[str, str]) -> str: - env = os.environ.copy() - env.update(extra_env) - env.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") - me._with_package_pythonpath(env) - proc = subprocess.run( - [sys.executable, "-c", code], - env=env, - check=False, - capture_output=True, - text=True, - ) - if proc.returncode != 0: - raise AssertionError( - f"python -c exited {proc.returncode}\n" - f"stdout:\n{proc.stdout}\nstderr:\n{proc.stderr}" - ) - return proc.stdout - - -# --------------------------------------------------------------------------- -# Inline scripts -# --------------------------------------------------------------------------- - -_NO_UNAPPLIED = """ -import os -import django -os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") -django.setup() -from django.db import connection -from django.db.migrations.executor import MigrationExecutor -from django.db.migrations.loader import MigrationLoader -loader = MigrationLoader(connection) -executor = MigrationExecutor(connection) -plan = executor.migration_plan(loader.graph.leaf_nodes()) -assert plan == [], f"Unapplied migrations: {[str(m) for m, _ in plan]}" -print("OK") -""" - - -# --------------------------------------------------------------------------- -# Tests — container migrations -# --------------------------------------------------------------------------- - -def test_migrate_runs_clean(fresh_db_env): - """migrate --noinput completes without error on a fresh container database.""" - engine, env = fresh_db_env - me.run_manage(["migrate", "--noinput", "--verbosity", "0"], env) - - -def test_migrate_is_idempotent(fresh_db_env): - """Running migrate twice is safe (no errors, no unexpected state).""" - engine, env = fresh_db_env - me.run_manage(["migrate", "--noinput", "--verbosity", "0"], env) - me.run_manage(["migrate", "--noinput", "--verbosity", "0"], env) - - -def test_no_unapplied_migrations(fresh_db_env): - """After migrate, MigrationExecutor reports an empty plan.""" - engine, env = fresh_db_env - me.run_manage(["migrate", "--noinput", "--verbosity", "0"], env) - _run_python(_NO_UNAPPLIED, env) - - -# --------------------------------------------------------------------------- -# SQLite baseline (no Docker needed — fast sanity check) -# --------------------------------------------------------------------------- - -def test_sqlite_migrate_baseline(tmp_path): - """migrate --noinput works against SQLite; ensures the test runner itself is healthy.""" - sqlite_path = tmp_path / "db.sqlite3" - env = { - "LOGSTASHUI_DATA_DIR": str(tmp_path), - "LOGSTASHUI_DB_ENGINE": "sqlite", - "LOGSTASHUI_DB_NAME": str(sqlite_path), - } - me.run_manage(["migrate", "--noinput", "--verbosity", "0"], env) - assert sqlite_path.exists() diff --git a/tests/integration/test_orm.py b/tests/integration/test_orm.py deleted file mode 100644 index 5aef6ab..0000000 --- a/tests/integration/test_orm.py +++ /dev/null @@ -1,389 +0,0 @@ -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. - -""" -ORM integration tests against real database containers. - -All Django interaction happens in subprocesses so each test gets a clean -interpreter with the container env applied to settings. Each test migrates -(idempotently) then runs its CRUD script which self-cleans at the end. -""" - -import json -import os -import subprocess -import sys - -import pytest - -from LogstashUI import migrate_engine as me - - -# --------------------------------------------------------------------------- -# Subprocess helper -# --------------------------------------------------------------------------- - -def _run_python(code: str, extra_env: dict[str, str]) -> str: - env = os.environ.copy() - env.update(extra_env) - env.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") - me._with_package_pythonpath(env) - proc = subprocess.run( - [sys.executable, "-c", code], - env=env, - check=False, - capture_output=True, - text=True, - ) - if proc.returncode != 0: - raise AssertionError( - f"python -c exited {proc.returncode}\n" - f"stdout:\n{proc.stdout}\nstderr:\n{proc.stderr}" - ) - return proc.stdout - - -def _migrate(env: dict[str, str]) -> None: - me.run_manage(["migrate", "--noinput", "--verbosity", "0"], env) - - -# --------------------------------------------------------------------------- -# Inline CRUD scripts (each cleans up its own data) -# --------------------------------------------------------------------------- - -_POLICY_CRUD = """ -import json, os, django -os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") -django.setup() -from PipelineManager.models import Policy -TAG = "crud-policy-test" -Policy.objects.filter(name=TAG).delete() -p = Policy.objects.create( - name=TAG, - logstash_yml="http.host: 0.0.0.0", - jvm_options="-Xms512m", - log4j2_properties="status = error", -) -pk = p.pk -assert Policy.objects.get(pk=pk).name == TAG -Policy.objects.filter(pk=pk).update(logstash_yml="http.host: 127.0.0.1") -assert Policy.objects.get(pk=pk).logstash_yml == "http.host: 127.0.0.1" -Policy.objects.filter(pk=pk).delete() -assert Policy.objects.filter(pk=pk).count() == 0 -print(json.dumps({"ok": True})) -""" - -_CONNECTION_CRUD = """ -import json, os, django -os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") -django.setup() -from PipelineManager.models import Connection, Policy -Policy.objects.filter(name="crud-conn-policy").delete() -policy = Policy.objects.create( - name="crud-conn-policy", - logstash_yml="http.host: 0.0.0.0", - jvm_options="-Xms512m", - log4j2_properties="status = error", -) -Connection.objects.filter(name="crud-conn-test").delete() -c = Connection.objects.create( - name="crud-conn-test", - connection_type=Connection.ConnectionType.AGENT, - host="127.0.0.1", - policy=policy, - status_blob={"health": "green"}, -) -pk = c.pk -assert Connection.objects.get(pk=pk).name == "crud-conn-test" -Connection.objects.filter(pk=pk).update(status_blob={"health": "yellow"}) -assert Connection.objects.get(pk=pk).status_blob["health"] == "yellow" -Connection.objects.filter(pk=pk).delete() -policy.delete() -assert Connection.objects.filter(pk=pk).count() == 0 -print(json.dumps({"ok": True})) -""" - -_PIPELINE_CRUD = """ -import json, os, django -os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") -django.setup() -from PipelineManager.models import Pipeline, Policy -Policy.objects.filter(name="crud-pipe-policy").delete() -policy = Policy.objects.create( - name="crud-pipe-policy", - logstash_yml="http.host: 0.0.0.0", - jvm_options="-Xms512m", - log4j2_properties="status = error", -) -Pipeline.objects.filter(policy=policy, name="test-pipeline").delete() -p = Pipeline.objects.create( - policy=policy, - name="test-pipeline", - lscl="input { stdin{} } output { stdout{} }", -) -pk = p.pk -assert Pipeline.objects.get(pk=pk).name == "test-pipeline" -Pipeline.objects.filter(pk=pk).delete() -policy.delete() -assert Pipeline.objects.filter(pk=pk).count() == 0 -print(json.dumps({"ok": True})) -""" - -_REVISION_JSON = """ -import json, os, django -os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") -django.setup() -from PipelineManager.models import Policy, Revision -Policy.objects.filter(name="rev-json-policy").delete() -policy = Policy.objects.create( - name="rev-json-policy", - logstash_yml="http.host: 0.0.0.0", - jvm_options="-Xms512m", - log4j2_properties="status = error", -) -snapshot = { - "pipelines": [{"name": "main", "lscl": "input{} output{}"}], - "meta": {"tags": ["a", "b"], "nested": {"k": 1}}, -} -r = Revision.objects.create( - policy=policy, revision_number=1, snapshot_json=snapshot, created_by="testrunner" -) -pk = r.pk -fetched = Revision.objects.get(pk=pk) -assert fetched.snapshot_json == snapshot, f"mismatch: {fetched.snapshot_json!r}" -Revision.objects.filter(pk=pk).delete() -policy.delete() -print(json.dumps({"ok": True})) -""" - -_STATUS_BLOB_JSON = """ -import json, os, django -os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") -django.setup() -from PipelineManager.models import Connection, Policy -Policy.objects.filter(name="blob-policy").delete() -policy = Policy.objects.create( - name="blob-policy", - logstash_yml="http.host: 0.0.0.0", - jvm_options="-Xms512m", - log4j2_properties="status = error", -) -blob = {"health": "green", "n": 42, "nested": {"k": [1, 2, 3]}} -Connection.objects.filter(name="blob-conn").delete() -c = Connection.objects.create( - name="blob-conn", - connection_type=Connection.ConnectionType.AGENT, - host="127.0.0.1", - policy=policy, - status_blob=blob, -) -pk = c.pk -fetched = Connection.objects.get(pk=pk) -assert fetched.status_blob == blob, f"mismatch: {fetched.status_blob!r}" -Connection.objects.filter(pk=pk).delete() -policy.delete() -print(json.dumps({"ok": True})) -""" - -_STATUS_BLOB_NULL = """ -import json, os, django -os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") -django.setup() -from PipelineManager.models import Connection, Policy -Policy.objects.filter(name="null-blob-policy").delete() -policy = Policy.objects.create( - name="null-blob-policy", - logstash_yml="http.host: 0.0.0.0", - jvm_options="-Xms512m", - log4j2_properties="status = error", -) -Connection.objects.filter(name="null-blob-conn").delete() -c = Connection.objects.create( - name="null-blob-conn", - connection_type=Connection.ConnectionType.AGENT, - host="127.0.0.1", - policy=policy, - status_blob=None, -) -pk = c.pk -assert Connection.objects.get(pk=pk).status_blob is None -Connection.objects.filter(pk=pk).delete() -policy.delete() -print(json.dumps({"ok": True})) -""" - -_SNMP_NETWORK_CRUD = """ -import json, os, django -os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") -django.setup() -from SNMP.models import Network -Network.objects.filter(name="crud-network-test").delete() -n = Network.objects.create(name="crud-network-test", network_range="10.0.0.0/24") -pk = n.pk -assert Network.objects.get(pk=pk).name == "crud-network-test" -Network.objects.filter(pk=pk).delete() -assert Network.objects.filter(pk=pk).count() == 0 -print(json.dumps({"ok": True})) -""" - -_UNIQUE_POLICY_NAME = """ -import json, os, django -os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") -django.setup() -from django.db import IntegrityError -from PipelineManager.models import Policy -Policy.objects.filter(name="dupe-policy").delete() -p1 = Policy.objects.create( - name="dupe-policy", - logstash_yml="http.host: 0.0.0.0", - jvm_options="-Xms512m", - log4j2_properties="status = error", -) -try: - Policy.objects.create( - name="dupe-policy", - logstash_yml="different", - jvm_options="-Xmx512m", - log4j2_properties="status = warn", - ) - raise AssertionError("Expected IntegrityError for duplicate Policy.name") -except IntegrityError: - pass -finally: - Policy.objects.filter(name="dupe-policy").delete() -print(json.dumps({"ok": True})) -""" - -_UNIQUE_PIPELINE_PER_POLICY = """ -import json, os, django -os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") -django.setup() -from django.db import IntegrityError -from PipelineManager.models import Pipeline, Policy -Policy.objects.filter(name__in=["uq-pol-a", "uq-pol-b"]).delete() -pol_a = Policy.objects.create( - name="uq-pol-a", - logstash_yml="http.host: 0.0.0.0", - jvm_options="-Xms512m", - log4j2_properties="status = error", -) -pol_b = Policy.objects.create( - name="uq-pol-b", - logstash_yml="http.host: 0.0.0.0", - jvm_options="-Xms512m", - log4j2_properties="status = error", -) -# Same name under different policies is allowed -Pipeline.objects.create(policy=pol_a, name="shared-pipe", lscl="input{} output{}") -Pipeline.objects.create(policy=pol_b, name="shared-pipe", lscl="input{} output{}") -# Same name under same policy must raise -try: - Pipeline.objects.create(policy=pol_a, name="shared-pipe", lscl="input{} output{}") - raise AssertionError("Expected IntegrityError for duplicate pipeline name per policy") -except IntegrityError: - pass -pol_a.delete() -pol_b.delete() -print(json.dumps({"ok": True})) -""" - -# Validates utf8mb4_bin on MySQL and native case-sensitivity on PostgreSQL. -# "CaseNet" and "casenet" must be distinct; second "CaseNet" must fail. -_CASE_SENSITIVE_UNIQUE = """ -import json, os, django -os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") -django.setup() -from django.db import IntegrityError -from SNMP.models import Network -Network.objects.filter(name__in=["CaseNet", "casenet"]).delete() -Network.objects.create(name="CaseNet", network_range="10.1.0.0/24") -# Different case must succeed -Network.objects.create(name="casenet", network_range="10.2.0.0/24") -# Exact duplicate must fail -try: - Network.objects.create(name="CaseNet", network_range="10.3.0.0/24") - raise AssertionError("Expected IntegrityError for duplicate Network.name") -except IntegrityError: - pass -finally: - Network.objects.filter(name__in=["CaseNet", "casenet"]).delete() -print(json.dumps({"ok": True})) -""" - - -# --------------------------------------------------------------------------- -# Tests (all parametrized over postgres + mysql via engine_env) -# --------------------------------------------------------------------------- - -def test_policy_crud(engine_env, tmp_path): - engine, env = engine_env - full_env = {**env, "LOGSTASHUI_DATA_DIR": str(tmp_path)} - _migrate(full_env) - assert json.loads(_run_python(_POLICY_CRUD, full_env).strip())["ok"] is True - - -def test_connection_crud(engine_env, tmp_path): - engine, env = engine_env - full_env = {**env, "LOGSTASHUI_DATA_DIR": str(tmp_path)} - _migrate(full_env) - assert json.loads(_run_python(_CONNECTION_CRUD, full_env).strip())["ok"] is True - - -def test_pipeline_crud(engine_env, tmp_path): - engine, env = engine_env - full_env = {**env, "LOGSTASHUI_DATA_DIR": str(tmp_path)} - _migrate(full_env) - assert json.loads(_run_python(_PIPELINE_CRUD, full_env).strip())["ok"] is True - - -def test_revision_json_roundtrip(engine_env, tmp_path): - engine, env = engine_env - full_env = {**env, "LOGSTASHUI_DATA_DIR": str(tmp_path)} - _migrate(full_env) - assert json.loads(_run_python(_REVISION_JSON, full_env).strip())["ok"] is True - - -def test_status_blob_roundtrip(engine_env, tmp_path): - engine, env = engine_env - full_env = {**env, "LOGSTASHUI_DATA_DIR": str(tmp_path)} - _migrate(full_env) - assert json.loads(_run_python(_STATUS_BLOB_JSON, full_env).strip())["ok"] is True - - -def test_status_blob_null_roundtrip(engine_env, tmp_path): - engine, env = engine_env - full_env = {**env, "LOGSTASHUI_DATA_DIR": str(tmp_path)} - _migrate(full_env) - assert json.loads(_run_python(_STATUS_BLOB_NULL, full_env).strip())["ok"] is True - - -def test_snmp_network_crud(engine_env, tmp_path): - engine, env = engine_env - full_env = {**env, "LOGSTASHUI_DATA_DIR": str(tmp_path)} - _migrate(full_env) - assert json.loads(_run_python(_SNMP_NETWORK_CRUD, full_env).strip())["ok"] is True - - -def test_unique_policy_name(engine_env, tmp_path): - engine, env = engine_env - full_env = {**env, "LOGSTASHUI_DATA_DIR": str(tmp_path)} - _migrate(full_env) - assert json.loads(_run_python(_UNIQUE_POLICY_NAME, full_env).strip())["ok"] is True - - -def test_unique_pipeline_per_policy(engine_env, tmp_path): - engine, env = engine_env - full_env = {**env, "LOGSTASHUI_DATA_DIR": str(tmp_path)} - _migrate(full_env) - assert json.loads(_run_python(_UNIQUE_PIPELINE_PER_POLICY, full_env).strip())["ok"] is True - - -def test_case_sensitive_unique(engine_env, tmp_path): - """Both engines treat unique names as case-sensitive. - On MySQL this validates utf8mb4_bin is active; on PostgreSQL it's the default. - """ - engine, env = engine_env - full_env = {**env, "LOGSTASHUI_DATA_DIR": str(tmp_path)} - _migrate(full_env) - assert json.loads(_run_python(_CASE_SENSITIVE_UNIQUE, full_env).strip())["ok"] is True diff --git a/tests/unit/__init__.py b/tests/unit/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/tests/unit/test_database.py b/tests/unit/test_database.py deleted file mode 100644 index 5afb0de..0000000 --- a/tests/unit/test_database.py +++ /dev/null @@ -1,245 +0,0 @@ -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. - -from pathlib import Path -from types import SimpleNamespace - -import pytest - -from LogstashUI.database import ( - build_databases, - canonical_engine, - check_server_version, -) - - -def _clear_db_env(monkeypatch): - for name in ( - "LOGSTASHUI_DB_ENGINE", - "LOGSTASHUI_DB_NAME", - "LOGSTASHUI_DB_HOST", - "LOGSTASHUI_DB_PORT", - "LOGSTASHUI_DB_USER", - "LOGSTASHUI_DB_PASSWORD", - "LOGSTASHUI_DB_SSLMODE", - "LOGSTASHUI_DB_SSL_CA", - "LOGSTASHUI_DB_CONN_MAX_AGE", - "LOGSTASHUI_DB_CONN_HEALTH_CHECKS", - ): - monkeypatch.delenv(name, raising=False) - - -def test_build_databases_sqlite_default(tmp_path, monkeypatch): - _clear_db_env(monkeypatch) - db = build_databases(tmp_path) - assert db["default"]["ENGINE"] == "django.db.backends.sqlite3" - assert db["default"]["NAME"] == tmp_path / "db.sqlite3" - assert db["default"]["OPTIONS"]["timeout"] == 20 - assert "PRAGMA journal_mode=WAL" in db["default"]["OPTIONS"]["init_command"] - - -@pytest.mark.parametrize( - "raw,expected", - [ - ("", "sqlite"), - ("sqlite", "sqlite"), - ("sqlite3", "sqlite"), - ("postgres", "postgresql"), - ("postgresql", "postgresql"), - ("mysql", "mysql"), - ("mariadb", "mysql"), - ("my", "mysql"), - ("POSTGRESQL", "postgresql"), - ], -) -def test_canonical_engine_aliases(raw, expected): - assert canonical_engine(raw) == expected - - -def test_unknown_engine_fails(tmp_path, monkeypatch): - _clear_db_env(monkeypatch) - monkeypatch.setenv("LOGSTASHUI_DB_ENGINE", "oracle") - with pytest.raises(RuntimeError, match="Unknown LOGSTASHUI_DB_ENGINE"): - build_databases(tmp_path) - - -def test_postgresql_requires_host_user(tmp_path, monkeypatch): - _clear_db_env(monkeypatch) - monkeypatch.setenv("LOGSTASHUI_DB_ENGINE", "postgresql") - monkeypatch.setattr("LogstashUI.database._import_or_raise", lambda *a, **k: None) - with pytest.raises(RuntimeError, match="LOGSTASHUI_DB_HOST"): - build_databases(tmp_path) - monkeypatch.setenv("LOGSTASHUI_DB_HOST", "db.example") - with pytest.raises(RuntimeError, match="LOGSTASHUI_DB_USER"): - build_databases(tmp_path) - - -def test_build_databases_postgresql(tmp_path, monkeypatch): - _clear_db_env(monkeypatch) - monkeypatch.setenv("LOGSTASHUI_DB_ENGINE", "postgres") - monkeypatch.setenv("LOGSTASHUI_DB_HOST", "db.example") - monkeypatch.setenv("LOGSTASHUI_DB_USER", "lsui") - monkeypatch.setenv("LOGSTASHUI_DB_PASSWORD", "s3cret") - monkeypatch.setenv("LOGSTASHUI_DB_SSLMODE", "require") - monkeypatch.setenv("LOGSTASHUI_DB_SSL_CA", "/etc/ssl/db-ca.pem") - monkeypatch.setattr("LogstashUI.database._import_or_raise", lambda *a, **k: None) - db = build_databases(tmp_path)["default"] - assert db["ENGINE"] == "django.db.backends.postgresql" - assert db["NAME"] == "logstashui" - assert db["HOST"] == "db.example" - assert db["PORT"] == "5432" - assert db["USER"] == "lsui" - assert db["PASSWORD"] == "s3cret" - assert db["CONN_MAX_AGE"] == 60 - assert db["CONN_HEALTH_CHECKS"] is True - assert db["OPTIONS"]["sslmode"] == "require" - assert db["OPTIONS"]["sslrootcert"] == "/etc/ssl/db-ca.pem" - - -def test_build_databases_mysql_mariadb_alias(tmp_path, monkeypatch): - _clear_db_env(monkeypatch) - monkeypatch.setenv("LOGSTASHUI_DB_ENGINE", "mariadb") - monkeypatch.setenv("LOGSTASHUI_DB_HOST", "127.0.0.1") - monkeypatch.setenv("LOGSTASHUI_DB_USER", "lsui") - monkeypatch.setenv("LOGSTASHUI_DB_PORT", "3307") - monkeypatch.setenv("LOGSTASHUI_DB_CONN_MAX_AGE", "0") - monkeypatch.setenv("LOGSTASHUI_DB_CONN_HEALTH_CHECKS", "false") - monkeypatch.setattr("LogstashUI.database._import_or_raise", lambda *a, **k: None) - db = build_databases(tmp_path)["default"] - assert db["ENGINE"] == "django.db.backends.mysql" - assert db["PORT"] == "3307" - assert db["CONN_MAX_AGE"] == 0 - assert db["CONN_HEALTH_CHECKS"] is False - assert db["OPTIONS"]["charset"] == "utf8mb4" - assert "utf8mb4_bin" in db["OPTIONS"]["init_command"] - assert db["TEST"]["CHARSET"] == "utf8mb4" - assert db["TEST"]["COLLATION"] == "utf8mb4_bin" - - -def test_mysql_spoofs_pymysql_version_info(tmp_path, monkeypatch): - _clear_db_env(monkeypatch) - monkeypatch.setenv("LOGSTASHUI_DB_ENGINE", "mysql") - monkeypatch.setenv("LOGSTASHUI_DB_HOST", "127.0.0.1") - monkeypatch.setenv("LOGSTASHUI_DB_USER", "lsui") - installed = [] - fake = SimpleNamespace( - version_info=(1, 1, 1, "final", 0), - install_as_MySQLdb=lambda: installed.append(True), - ) - monkeypatch.setattr("LogstashUI.database._import_or_raise", lambda *a, **k: fake) - build_databases(tmp_path) - assert fake.version_info == (2, 2, 1, "final", 0) - assert installed - - -def test_postgresql_missing_driver(tmp_path, monkeypatch): - _clear_db_env(monkeypatch) - monkeypatch.setenv("LOGSTASHUI_DB_ENGINE", "postgresql") - monkeypatch.setenv("LOGSTASHUI_DB_HOST", "localhost") - monkeypatch.setenv("LOGSTASHUI_DB_USER", "lsui") - - def boom(module, extra): - raise RuntimeError( - f"{module} is not installed. Install with: uv pip install 'LogstashUI[{extra}]'" - ) - - monkeypatch.setattr("LogstashUI.database._import_or_raise", boom) - with pytest.raises(RuntimeError, match=r"LogstashUI\[postgres\]"): - build_databases(tmp_path) - - -def test_check_server_version_sqlite_noop(): - class Conn: - vendor = "sqlite" - - check_server_version(Conn()) - - -def test_check_server_version_postgres_too_old(): - class Conn: - vendor = "postgresql" - pg_version = 130000 - - with pytest.raises(RuntimeError, match="PostgreSQL 14"): - check_server_version(Conn()) - - -def test_check_server_version_postgres_zero_is_too_old(): - class Conn: - vendor = "postgresql" - pg_version = 0 - - with pytest.raises(RuntimeError, match="PostgreSQL 14"): - check_server_version(Conn()) - - -def test_conn_max_age_invalid_raises_runtimeerror(tmp_path, monkeypatch): - _clear_db_env(monkeypatch) - monkeypatch.setenv("LOGSTASHUI_DB_CONN_MAX_AGE", "nope") - with pytest.raises(RuntimeError, match="LOGSTASHUI_DB_CONN_MAX_AGE"): - build_databases(tmp_path) - - -def test_password_is_stripped(tmp_path, monkeypatch): - _clear_db_env(monkeypatch) - monkeypatch.setenv("LOGSTASHUI_DB_ENGINE", "postgresql") - monkeypatch.setenv("LOGSTASHUI_DB_HOST", "db.example") - monkeypatch.setenv("LOGSTASHUI_DB_USER", "lsui") - monkeypatch.setenv("LOGSTASHUI_DB_PASSWORD", " secret\n") - monkeypatch.setattr("LogstashUI.database._import_or_raise", lambda *a, **k: None) - db = build_databases(tmp_path)["default"] - assert db["PASSWORD"] == "secret" - - -def test_ensure_psycopg_gevent_assigns_wait_select(): - from types import SimpleNamespace - - from LogstashUI.database import ensure_psycopg_gevent - - def wait_select(*args, **kwargs): - return "select" - - waiting = SimpleNamespace(wait_select=wait_select, wait=None) - ensure_psycopg_gevent(waiting) - assert waiting.wait is wait_select - - -def test_ensure_psycopg_gevent_does_not_raise(): - from LogstashUI.database import ensure_psycopg_gevent - - ensure_psycopg_gevent() - - -def test_check_server_version_mysql_and_mariadb(): - class Mysql: - vendor = "mysql" - mysql_is_mariadb = False - mysql_server_info = "8.0.36" - - def get_database_version(self): - return (8, 0, 36) - - check_server_version(Mysql()) - - class OldMysql: - vendor = "mysql" - mysql_is_mariadb = False - mysql_server_info = "5.7.44" - - def get_database_version(self): - return (5, 7, 44) - - with pytest.raises(RuntimeError, match="MySQL 8.0"): - check_server_version(OldMysql()) - - class Maria: - vendor = "mysql" - mysql_is_mariadb = True - mysql_server_info = "10.5.22-MariaDB" - - def get_database_version(self): - return (10, 5, 22) - - with pytest.raises(RuntimeError, match="MariaDB 10.6"): - check_server_version(Maria()) diff --git a/tests/unit/test_migrate_engine.py b/tests/unit/test_migrate_engine.py deleted file mode 100644 index c0008f7..0000000 --- a/tests/unit/test_migrate_engine.py +++ /dev/null @@ -1,111 +0,0 @@ -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. - -from argparse import Namespace -from pathlib import Path - -import pytest - -from LogstashUI import migrate_engine as me - - -def test_refuses_without_backup_flag(capsys): - ns = Namespace(to="postgresql", i_have_a_backup=False, pid=None, write_env=None) - with pytest.raises(SystemExit) as exc: - me.cmd_migrate_engine(ns) - assert exc.value.code == 2 - assert "back up" in capsys.readouterr().err.lower() - - -def test_refuses_sqlite_target(tmp_path, monkeypatch, capsys): - monkeypatch.setenv("LOGSTASHUI_DATA_DIR", str(tmp_path)) - ns = Namespace(to="sqlite", i_have_a_backup=True, pid=None, write_env=None) - with pytest.raises(SystemExit): - me.cmd_migrate_engine(ns) - - -def test_refuses_missing_sqlite_file(tmp_path, monkeypatch, capsys): - monkeypatch.setenv("LOGSTASHUI_DATA_DIR", str(tmp_path)) - ns = Namespace(to="postgresql", i_have_a_backup=True, pid=None, write_env=None) - with pytest.raises(SystemExit) as exc: - me.cmd_migrate_engine(ns) - assert exc.value.code == 1 - assert "db.sqlite3" in capsys.readouterr().err - - -def test_stop_pid_sends_sigterm(tmp_path, monkeypatch): - pidfile = tmp_path / "gunicorn.pid" - pidfile.write_text("4242\n") - sent = {} - calls = {"n": 0} - - def kill_then_gone(pid, sig): - calls["n"] += 1 - if calls["n"] == 1: - sent["pid"] = pid - sent["sig"] = sig - return - raise ProcessLookupError() - - monkeypatch.setattr(me.os, "kill", kill_then_gone) - monkeypatch.setattr(me.time, "sleep", lambda s: None) - me.stop_gunicorn(pidfile) - assert sent["pid"] == 4242 - assert sent["sig"] == me.signal.SIGTERM - assert not pidfile.exists() - - -def test_write_env_appends(tmp_path, monkeypatch): - envf = tmp_path / "logstashui.default" - envf.write_text("LOGSTASHUI_DATA_DIR=/var/lib/logstashui\n") - monkeypatch.setenv("LOGSTASHUI_DB_HOST", "db.example") - monkeypatch.setenv("LOGSTASHUI_DB_USER", "lsui") - monkeypatch.setenv("LOGSTASHUI_DB_NAME", "logstashui") - me.write_env_file(envf, "postgresql") - me.write_env_file(envf, "postgresql") - text = envf.read_text() - assert text.count("LOGSTASHUI_DB_ENGINE=postgresql") == 1 - assert "LOGSTASHUI_DB_HOST=db.example" in text - assert "PASSWORD" not in text - - -def test_run_manage_sets_package_pythonpath(monkeypatch): - captured = {} - - def fake_run(cmd, env=None, check=False): - captured["env"] = env - class Result: - returncode = 0 - return Result() - - monkeypatch.setattr(me.subprocess, "run", fake_run) - me.run_manage(["migrate", "--noinput"], {"LOGSTASHUI_DB_ENGINE": "sqlite"}) - pythonpath = captured["env"]["PYTHONPATH"] - pkg_root = str(Path(me.__file__).resolve().parent.parent) - assert pythonpath.split(me.os.pathsep)[0] == pkg_root - - -def test_reset_postgres_sequences_does_not_require_psql(monkeypatch): - captured = {} - - def fake_run(cmd, env=None, check=False, capture_output=False, text=False): - captured["cmd"] = cmd - captured["env"] = env - class Result: - returncode = 0 - stdout = "" - stderr = "" - return Result() - - monkeypatch.setattr(me.subprocess, "run", fake_run) - me._reset_postgres_sequences({"LOGSTASHUI_DB_ENGINE": "postgresql"}) - assert captured["cmd"][0] == me.sys.executable - assert captured["cmd"][1] == "-c" - code = captured["cmd"][2] - assert "dbshell" not in code - assert "psql" not in code - assert "sequence_reset_sql" in code - assert "cursor.execute" in code - pkg_root = str(Path(me.__file__).resolve().parent.parent) - assert captured["env"]["PYTHONPATH"].split(me.os.pathsep)[0] == pkg_root From 1fe7ef1da83750cccf6b063f4ab48e38bfdec897 Mon Sep 17 00:00:00 2001 From: Aaron Mildenstein Date: Thu, 3 Sep 2026 12:22:24 -0600 Subject: [PATCH 30/62] The tests that were relocated are now added --- tests/Common/__init__.py | 0 tests/Common/unit/__init__.py | 0 .../components/ls-repo-apache2.json | 162 ++ .../components/test-asa-new.json | 1846 ++++++++++++ .../conversion_data/components/test-asa.json | 1846 ++++++++++++ .../components/test-boolean-numeric.json | 134 + .../test-comments-brace-in-comment.json | 72 + .../components/test-comments-mixed.json | 154 + .../test-comments-plugin-inline.json | 111 + .../test-comments-section-opener.json | 100 + .../test-comments-standalone-in-plugin.json | 61 + .../components/test-complex2.json | 952 +++++++ .../components/test-complex3.json | 1497 ++++++++++ .../components/test-data-types.json | 52 + .../components/test-datatypes.json | 52 + .../components/test-devopsschool-1.json | 51 + .../components/test-devopsschool-2.json | 39 + .../components/test-devopsschool-4.json | 62 + .../components/test-devopsschool-5.json | 48 + .../components/test-elasticdocs-apache.json | 86 + .../test-elasticdocs-configuring_filters.json | 60 + .../components/test-elasticdocs-syslog.json | 92 + .../components/test-es-input.json | 48 + .../components/test-ls-repo-mysql.json | 156 ++ .../components/test-ls-repo-nginx.json | 156 ++ .../components/test-ls-repo-system.json | 135 + .../test-multiline-ruby-with-hash.json | 60 + .../test-nested-conditionals-comments.json | 266 ++ .../components/test-regex-conditions.json | 228 ++ .../components/test-sample-nginx.json | 183 ++ .../components/test-snmp-v0.2.json | 212 ++ .../components/test-string-escaping.json | 60 + .../components/test-twitter.json | 89 + .../components/test_complex1.json | 952 +++++++ .../test_elasticdocs-conditional.json | 116 + .../components/text-complex4.json | 386 +++ .../components/text-complex5.json | 94 + .../components/text-complex6.json | 469 ++++ .../components/text-complex7.json | 558 ++++ .../components/text-ls-repo-nginx-error.json | 183 ++ .../pipelines/ls-repo-apache2.conf | 69 + .../pipelines/test-asa-new.conf | 897 ++++++ .../conversion_data/pipelines/test-asa.conf | 897 ++++++ .../pipelines/test-boolean-numeric.conf | 58 + .../test-comments-brace-in-comment.conf | 39 + .../pipelines/test-comments-mixed.conf | 57 + .../test-comments-plugin-inline.conf | 46 + .../test-comments-section-opener.conf | 29 + .../test-comments-standalone-in-plugin.conf | 29 + .../pipelines/test-complex2.conf | 329 +++ .../pipelines/test-complex3.conf | 652 +++++ .../pipelines/test-data-types.conf | 33 + .../pipelines/test-datatypes.conf | 33 + .../pipelines/test-devopsschool-1.conf | 21 + .../pipelines/test-devopsschool-2.conf | 17 + .../pipelines/test-devopsschool-4.conf | 25 + .../pipelines/test-devopsschool-5.conf | 20 + .../pipelines/test-elasticdocs-apache.conf | 31 + .../test-elasticdocs-configuring_filters.conf | 22 + .../pipelines/test-elasticdocs-syslog.conf | 31 + .../pipelines/test-es-input.conf | 26 + .../pipelines/test-ls-repo-mysql.conf | 69 + .../pipelines/test-ls-repo-nginx.conf | 64 + .../pipelines/test-ls-repo-system.conf | 61 + .../test-multiline-ruby-with-hash.conf | 49 + .../test-nested-conditionals-comments.conf | 71 + .../pipelines/test-regex-conditions.conf | 62 + .../pipelines/test-sample-nginx.conf | 67 + .../pipelines/test-snmp-v0.2.conf | 213 ++ .../pipelines/test-string-escaping.conf | 33 + .../pipelines/test-twitter.conf | 41 + .../pipelines/test_complex1.conf | 329 +++ .../test_elasticdocs-conditional.conf | 44 + .../pipelines/text-complex4.conf | 123 + .../pipelines/text-complex5.conf | 40 + .../pipelines/text-complex6.conf | 192 ++ .../pipelines/text-complex7.conf | 222 ++ .../pipelines/text-ls-repo-nginx-error.conf | 67 + .../unit/test_components_to_pipeline.py | 57 + tests/Common/unit/test_context_processors.py | 224 ++ tests/Common/unit/test_decorators.py | 269 ++ tests/Common/unit/test_elastic_utils.py | 654 +++++ tests/Common/unit/test_encryption.py | 267 ++ tests/Common/unit/test_error_handlers.py | 263 ++ tests/Common/unit/test_formatters.py | 407 +++ .../Common/unit/test_logstash_config_parse.py | 542 ++++ tests/Common/unit/test_logstash_utils.py | 163 ++ tests/Common/unit/test_middleware.py | 242 ++ .../unit/test_pipeline_to_components.py | 51 + tests/Common/unit/test_product_ca.py | 334 +++ tests/Common/unit/test_validators.py | 175 ++ tests/Database/integration/__init__.py | 0 tests/Database/integration/conftest.py | 244 ++ tests/Database/integration/test_db_config.py | 166 ++ .../integration/test_migrate_engine.py | 292 ++ tests/Database/integration/test_migrations.py | 101 + tests/Database/integration/test_orm.py | 389 +++ tests/Database/unit/__init__.py | 0 tests/Database/unit/test_database.py | 245 ++ tests/Database/unit/test_migrate_engine.py | 111 + tests/LogstashUI/__init__.py | 0 tests/LogstashUI/unit/__init__.py | 0 tests/LogstashUI/unit/test_cli.py | 230 ++ tests/LogstashUI/unit/test_config.py | 61 + tests/LogstashUI/unit/test_logging_config.py | 50 + tests/LogstashUI/unit/test_paths.py | 91 + tests/Management/__init__.py | 0 tests/Management/unit/__init__.py | 0 tests/Management/unit/test_views.py | 1066 +++++++ tests/Monitoring/__init__.py | 0 tests/Monitoring/unit/__init__.py | 0 tests/Monitoring/unit/test_views.py | 882 ++++++ tests/PipelineManager/__init__.py | 0 tests/PipelineManager/unit/__init__.py | 0 tests/PipelineManager/unit/test_agent_api.py | 1108 ++++++++ .../PipelineManager/unit/test_agent_modes.py | 731 +++++ .../unit/test_agent_policies.py | 1003 +++++++ .../unit/test_agent_versions.py | 128 + .../unit/test_connections_crud.py | 602 ++++ .../PipelineManager/unit/test_editor_views.py | 750 +++++ .../unit/test_elasticsearch_queries.py | 490 ++++ .../unit/test_manager_views.py | 791 ++++++ .../unit/test_pipeline_editor.py | 547 ++++ .../unit/test_pipelines_crud.py | 774 +++++ .../unit/test_policies_crud.py | 1095 ++++++++ .../PipelineManager/unit/test_sim_keystore.py | 157 ++ tests/PipelineManager/unit/test_simulation.py | 817 ++++++ tests/SNMP/__init__.py | 0 tests/SNMP/data | 1 + tests/SNMP/unit/__init__.py | 0 tests/SNMP/unit/test_commands.py | 776 +++++ tests/SNMP/unit/test_inline_grounding.py | 66 + tests/SNMP/unit/test_models.py | 498 ++++ tests/SNMP/unit/test_network_map.py | 618 ++++ tests/SNMP/unit/test_overview.py | 482 ++++ tests/SNMP/unit/test_snmp_crud.py | 2485 +++++++++++++++++ tests/SNMP/unit/test_snmp_grounding.py | 263 ++ tests/SNMP/unit/test_snmp_normalizers.py | 545 ++++ .../SNMP/unit/test_snmp_pipeline_generator.py | 789 ++++++ tests/SNMP/unit/test_snmp_test.py | 920 ++++++ tests/SNMP/unit/test_views.py | 1098 ++++++++ tests/Site/__init__.py | 0 tests/Site/unit/__init__.py | 0 tests/Site/unit/test_views.py | 381 +++ tests/Utilities/__init__.py | 0 tests/Utilities/data | 1 + tests/Utilities/unit/__init__.py | 0 tests/Utilities/unit/test_grok_patterns.py | 138 + tests/Utilities/unit/test_views.py | 671 +++++ tests/conftest.py | 58 + 150 files changed, 43295 insertions(+) create mode 100644 tests/Common/__init__.py create mode 100644 tests/Common/unit/__init__.py create mode 100644 tests/Common/unit/conversion_data/components/ls-repo-apache2.json create mode 100644 tests/Common/unit/conversion_data/components/test-asa-new.json create mode 100644 tests/Common/unit/conversion_data/components/test-asa.json create mode 100644 tests/Common/unit/conversion_data/components/test-boolean-numeric.json create mode 100644 tests/Common/unit/conversion_data/components/test-comments-brace-in-comment.json create mode 100644 tests/Common/unit/conversion_data/components/test-comments-mixed.json create mode 100644 tests/Common/unit/conversion_data/components/test-comments-plugin-inline.json create mode 100644 tests/Common/unit/conversion_data/components/test-comments-section-opener.json create mode 100644 tests/Common/unit/conversion_data/components/test-comments-standalone-in-plugin.json create mode 100644 tests/Common/unit/conversion_data/components/test-complex2.json create mode 100644 tests/Common/unit/conversion_data/components/test-complex3.json create mode 100644 tests/Common/unit/conversion_data/components/test-data-types.json create mode 100644 tests/Common/unit/conversion_data/components/test-datatypes.json create mode 100644 tests/Common/unit/conversion_data/components/test-devopsschool-1.json create mode 100644 tests/Common/unit/conversion_data/components/test-devopsschool-2.json create mode 100644 tests/Common/unit/conversion_data/components/test-devopsschool-4.json create mode 100644 tests/Common/unit/conversion_data/components/test-devopsschool-5.json create mode 100644 tests/Common/unit/conversion_data/components/test-elasticdocs-apache.json create mode 100644 tests/Common/unit/conversion_data/components/test-elasticdocs-configuring_filters.json create mode 100644 tests/Common/unit/conversion_data/components/test-elasticdocs-syslog.json create mode 100644 tests/Common/unit/conversion_data/components/test-es-input.json create mode 100644 tests/Common/unit/conversion_data/components/test-ls-repo-mysql.json create mode 100644 tests/Common/unit/conversion_data/components/test-ls-repo-nginx.json create mode 100644 tests/Common/unit/conversion_data/components/test-ls-repo-system.json create mode 100644 tests/Common/unit/conversion_data/components/test-multiline-ruby-with-hash.json create mode 100644 tests/Common/unit/conversion_data/components/test-nested-conditionals-comments.json create mode 100644 tests/Common/unit/conversion_data/components/test-regex-conditions.json create mode 100644 tests/Common/unit/conversion_data/components/test-sample-nginx.json create mode 100644 tests/Common/unit/conversion_data/components/test-snmp-v0.2.json create mode 100644 tests/Common/unit/conversion_data/components/test-string-escaping.json create mode 100644 tests/Common/unit/conversion_data/components/test-twitter.json create mode 100644 tests/Common/unit/conversion_data/components/test_complex1.json create mode 100644 tests/Common/unit/conversion_data/components/test_elasticdocs-conditional.json create mode 100644 tests/Common/unit/conversion_data/components/text-complex4.json create mode 100644 tests/Common/unit/conversion_data/components/text-complex5.json create mode 100644 tests/Common/unit/conversion_data/components/text-complex6.json create mode 100644 tests/Common/unit/conversion_data/components/text-complex7.json create mode 100644 tests/Common/unit/conversion_data/components/text-ls-repo-nginx-error.json create mode 100644 tests/Common/unit/conversion_data/pipelines/ls-repo-apache2.conf create mode 100644 tests/Common/unit/conversion_data/pipelines/test-asa-new.conf create mode 100644 tests/Common/unit/conversion_data/pipelines/test-asa.conf create mode 100644 tests/Common/unit/conversion_data/pipelines/test-boolean-numeric.conf create mode 100644 tests/Common/unit/conversion_data/pipelines/test-comments-brace-in-comment.conf create mode 100644 tests/Common/unit/conversion_data/pipelines/test-comments-mixed.conf create mode 100644 tests/Common/unit/conversion_data/pipelines/test-comments-plugin-inline.conf create mode 100644 tests/Common/unit/conversion_data/pipelines/test-comments-section-opener.conf create mode 100644 tests/Common/unit/conversion_data/pipelines/test-comments-standalone-in-plugin.conf create mode 100644 tests/Common/unit/conversion_data/pipelines/test-complex2.conf create mode 100644 tests/Common/unit/conversion_data/pipelines/test-complex3.conf create mode 100644 tests/Common/unit/conversion_data/pipelines/test-data-types.conf create mode 100644 tests/Common/unit/conversion_data/pipelines/test-datatypes.conf create mode 100644 tests/Common/unit/conversion_data/pipelines/test-devopsschool-1.conf create mode 100644 tests/Common/unit/conversion_data/pipelines/test-devopsschool-2.conf create mode 100644 tests/Common/unit/conversion_data/pipelines/test-devopsschool-4.conf create mode 100644 tests/Common/unit/conversion_data/pipelines/test-devopsschool-5.conf create mode 100644 tests/Common/unit/conversion_data/pipelines/test-elasticdocs-apache.conf create mode 100644 tests/Common/unit/conversion_data/pipelines/test-elasticdocs-configuring_filters.conf create mode 100644 tests/Common/unit/conversion_data/pipelines/test-elasticdocs-syslog.conf create mode 100644 tests/Common/unit/conversion_data/pipelines/test-es-input.conf create mode 100644 tests/Common/unit/conversion_data/pipelines/test-ls-repo-mysql.conf create mode 100644 tests/Common/unit/conversion_data/pipelines/test-ls-repo-nginx.conf create mode 100644 tests/Common/unit/conversion_data/pipelines/test-ls-repo-system.conf create mode 100644 tests/Common/unit/conversion_data/pipelines/test-multiline-ruby-with-hash.conf create mode 100644 tests/Common/unit/conversion_data/pipelines/test-nested-conditionals-comments.conf create mode 100644 tests/Common/unit/conversion_data/pipelines/test-regex-conditions.conf create mode 100644 tests/Common/unit/conversion_data/pipelines/test-sample-nginx.conf create mode 100644 tests/Common/unit/conversion_data/pipelines/test-snmp-v0.2.conf create mode 100644 tests/Common/unit/conversion_data/pipelines/test-string-escaping.conf create mode 100644 tests/Common/unit/conversion_data/pipelines/test-twitter.conf create mode 100644 tests/Common/unit/conversion_data/pipelines/test_complex1.conf create mode 100644 tests/Common/unit/conversion_data/pipelines/test_elasticdocs-conditional.conf create mode 100644 tests/Common/unit/conversion_data/pipelines/text-complex4.conf create mode 100644 tests/Common/unit/conversion_data/pipelines/text-complex5.conf create mode 100644 tests/Common/unit/conversion_data/pipelines/text-complex6.conf create mode 100644 tests/Common/unit/conversion_data/pipelines/text-complex7.conf create mode 100644 tests/Common/unit/conversion_data/pipelines/text-ls-repo-nginx-error.conf create mode 100644 tests/Common/unit/test_components_to_pipeline.py create mode 100644 tests/Common/unit/test_context_processors.py create mode 100644 tests/Common/unit/test_decorators.py create mode 100644 tests/Common/unit/test_elastic_utils.py create mode 100644 tests/Common/unit/test_encryption.py create mode 100644 tests/Common/unit/test_error_handlers.py create mode 100644 tests/Common/unit/test_formatters.py create mode 100644 tests/Common/unit/test_logstash_config_parse.py create mode 100644 tests/Common/unit/test_logstash_utils.py create mode 100644 tests/Common/unit/test_middleware.py create mode 100644 tests/Common/unit/test_pipeline_to_components.py create mode 100644 tests/Common/unit/test_product_ca.py create mode 100644 tests/Common/unit/test_validators.py create mode 100644 tests/Database/integration/__init__.py create mode 100644 tests/Database/integration/conftest.py create mode 100644 tests/Database/integration/test_db_config.py create mode 100644 tests/Database/integration/test_migrate_engine.py create mode 100644 tests/Database/integration/test_migrations.py create mode 100644 tests/Database/integration/test_orm.py create mode 100644 tests/Database/unit/__init__.py create mode 100644 tests/Database/unit/test_database.py create mode 100644 tests/Database/unit/test_migrate_engine.py create mode 100644 tests/LogstashUI/__init__.py create mode 100644 tests/LogstashUI/unit/__init__.py create mode 100644 tests/LogstashUI/unit/test_cli.py create mode 100644 tests/LogstashUI/unit/test_config.py create mode 100644 tests/LogstashUI/unit/test_logging_config.py create mode 100644 tests/LogstashUI/unit/test_paths.py create mode 100644 tests/Management/__init__.py create mode 100644 tests/Management/unit/__init__.py create mode 100644 tests/Management/unit/test_views.py create mode 100644 tests/Monitoring/__init__.py create mode 100644 tests/Monitoring/unit/__init__.py create mode 100644 tests/Monitoring/unit/test_views.py create mode 100644 tests/PipelineManager/__init__.py create mode 100644 tests/PipelineManager/unit/__init__.py create mode 100644 tests/PipelineManager/unit/test_agent_api.py create mode 100644 tests/PipelineManager/unit/test_agent_modes.py create mode 100644 tests/PipelineManager/unit/test_agent_policies.py create mode 100644 tests/PipelineManager/unit/test_agent_versions.py create mode 100644 tests/PipelineManager/unit/test_connections_crud.py create mode 100644 tests/PipelineManager/unit/test_editor_views.py create mode 100644 tests/PipelineManager/unit/test_elasticsearch_queries.py create mode 100644 tests/PipelineManager/unit/test_manager_views.py create mode 100644 tests/PipelineManager/unit/test_pipeline_editor.py create mode 100644 tests/PipelineManager/unit/test_pipelines_crud.py create mode 100644 tests/PipelineManager/unit/test_policies_crud.py create mode 100644 tests/PipelineManager/unit/test_sim_keystore.py create mode 100644 tests/PipelineManager/unit/test_simulation.py create mode 100644 tests/SNMP/__init__.py create mode 120000 tests/SNMP/data create mode 100644 tests/SNMP/unit/__init__.py create mode 100644 tests/SNMP/unit/test_commands.py create mode 100644 tests/SNMP/unit/test_inline_grounding.py create mode 100644 tests/SNMP/unit/test_models.py create mode 100644 tests/SNMP/unit/test_network_map.py create mode 100644 tests/SNMP/unit/test_overview.py create mode 100644 tests/SNMP/unit/test_snmp_crud.py create mode 100644 tests/SNMP/unit/test_snmp_grounding.py create mode 100644 tests/SNMP/unit/test_snmp_normalizers.py create mode 100644 tests/SNMP/unit/test_snmp_pipeline_generator.py create mode 100644 tests/SNMP/unit/test_snmp_test.py create mode 100644 tests/SNMP/unit/test_views.py create mode 100644 tests/Site/__init__.py create mode 100644 tests/Site/unit/__init__.py create mode 100644 tests/Site/unit/test_views.py create mode 100644 tests/Utilities/__init__.py create mode 120000 tests/Utilities/data create mode 100644 tests/Utilities/unit/__init__.py create mode 100644 tests/Utilities/unit/test_grok_patterns.py create mode 100644 tests/Utilities/unit/test_views.py create mode 100644 tests/conftest.py diff --git a/tests/Common/__init__.py b/tests/Common/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/Common/unit/__init__.py b/tests/Common/unit/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/Common/unit/conversion_data/components/ls-repo-apache2.json b/tests/Common/unit/conversion_data/components/ls-repo-apache2.json new file mode 100644 index 0000000..439a46f --- /dev/null +++ b/tests/Common/unit/conversion_data/components/ls-repo-apache2.json @@ -0,0 +1,162 @@ +{ + "input": [ + { + "id": "input_beats_0", + "type": "input", + "plugin": "beats", + "config": { + "port": 5044, + "host": "0.0.0.0" + }, + "comments": [] + } + ], + "filter": [ + { + "id": "filter_if_1", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[fileset][module] == \"apache2\"", + "plugins": [ + { + "id": "filter_if_2", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[fileset][name] == \"access\"", + "plugins": [ + { + "id": "filter_grok_3", + "type": "filter", + "plugin": "grok", + "config": { + "match": { + "message": [ + "%{IPORHOST:[apache2][access][remote_ip]} - %{DATA:[apache2][access][user_name]} \\[%{HTTPDATE:[apache2][access][time]}\\] \"%{WORD:[apache2][access][method]} %{DATA:[apache2][access][url]} HTTP/%{NUMBER:[apache2][access][http_version]}\" %{NUMBER:[apache2][access][response_code]} %{NUMBER:[apache2][access][body_sent][bytes]}( \"%{DATA:[apache2][access][referrer]}\")?( \"%{DATA:[apache2][access][agent]}\")?", + "%{IPORHOST:[apache2][access][remote_ip]} - %{DATA:[apache2][access][user_name]} \\[%{HTTPDATE:[apache2][access][time]}\\] \"-\" %{NUMBER:[apache2][access][response_code]} -" + ] + }, + "remove_field": "message" + }, + "comments": [] + }, + { + "id": "filter_mutate_4", + "type": "filter", + "plugin": "mutate", + "config": { + "add_field": { + "read_timestamp": "%{@timestamp}" + } + }, + "comments": [] + }, + { + "id": "filter_date_5", + "type": "filter", + "plugin": "date", + "config": { + "match": [ + "[apache2][access][time]", + "dd/MMM/YYYY:H:m:s Z" + ], + "remove_field": "[apache2][access][time]" + }, + "comments": [] + }, + { + "id": "filter_useragent_6", + "type": "filter", + "plugin": "useragent", + "config": { + "source": "[apache2][access][agent]", + "target": "[apache2][access][user_agent]", + "remove_field": "[apache2][access][agent]" + }, + "comments": [] + }, + { + "id": "filter_geoip_7", + "type": "filter", + "plugin": "geoip", + "config": { + "source": "[apache2][access][remote_ip]", + "target": "[apache2][access][geoip]" + }, + "comments": [] + } + ], + "else_ifs": [ + { + "condition": "[fileset][name] == \"error\"", + "plugins": [ + { + "id": "filter_grok_8", + "type": "filter", + "plugin": "grok", + "config": { + "match": { + "message": [ + "\\[%{APACHE_TIME:[apache2][error][timestamp]}\\] \\[%{LOGLEVEL:[apache2][error][level]}\\]( \\[client %{IPORHOST:[apache2][error][client]}\\])? %{GREEDYDATA:[apache2][error][message]}", + "\\[%{APACHE_TIME:[apache2][error][timestamp]}\\] \\[%{DATA:[apache2][error][module]}:%{LOGLEVEL:[apache2][error][level]}\\] \\[pid %{NUMBER:[apache2][error][pid]}(:tid %{NUMBER:[apache2][error][tid]})?\\]( \\[client %{IPORHOST:[apache2][error][client]}\\])? %{GREEDYDATA:[apache2][error][message1]}" + ] + }, + "pattern_definitions": { + "APACHE_TIME": "%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR}" + }, + "remove_field": "message" + }, + "comments": [] + }, + { + "id": "filter_mutate_9", + "type": "filter", + "plugin": "mutate", + "config": { + "rename": { + "[apache2][error][message1]": "[apache2][error][message]" + } + }, + "comments": [] + }, + { + "id": "filter_date_10", + "type": "filter", + "plugin": "date", + "config": { + "match": [ + "[apache2][error][timestamp]", + "EEE MMM dd H:m:s YYYY", + "EEE MMM dd H:m:s.SSSSSS YYYY" + ], + "remove_field": "[apache2][error][timestamp]" + }, + "comments": [] + } + ] + } + ], + "else": null + } + } + ], + "else_ifs": [], + "else": null + } + } + ], + "output": [ + { + "id": "output_elasticsearch_11", + "type": "output", + "plugin": "elasticsearch", + "config": { + "hosts": "localhost", + "manage_template": "false", + "index": "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" + }, + "comments": [] + } + ] +} diff --git a/tests/Common/unit/conversion_data/components/test-asa-new.json b/tests/Common/unit/conversion_data/components/test-asa-new.json new file mode 100644 index 0000000..3945c20 --- /dev/null +++ b/tests/Common/unit/conversion_data/components/test-asa-new.json @@ -0,0 +1,1846 @@ +{ + "input": [ + { + "id": "input_udp_0", + "type": "input", + "plugin": "udp", + "config": { + "id": "input_udp_1", + "port": "5119" + }, + "comments": [] + }, + { + "id": "input_cloudwatch_1", + "type": "input", + "plugin": "cloudwatch", + "config": {}, + "comments": [] + } + ], + "filter": [ + { + "id": "filter_mutate_2", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_1", + "rename": { + "message": "log.original", + "host": "observer.ip" + }, + "copy": { + "host": "sysloghost" + } + }, + "comments": [] + }, + { + "id": "filter_grok_3", + "type": "filter", + "plugin": "grok", + "config": { + "id": "filter_grok_1", + "match": { + "log.original": [ + "%{CISCO_TAGGED_SYSLOG} %{GREEDYDATA:message}", + "^<%{POSINT:syslog_pri}>%{DATA}: %%{DATA:ciscotag}: %{GREEDYDATA:message}", + "^<%{POSINT:syslog_pri}>%%{DATA:ciscotag}: %{GREEDYDATA:message}" + ] + } + }, + "comments": [] + }, + { + "id": "filter_grok_4", + "type": "filter", + "plugin": "grok", + "config": { + "id": "filter_grok_2", + "match": { + "ciscotag": [ + "%{WORD}-%{INT:event.severity}-%{INT:event.code}", + "%{WORD}-%{WORD}-%{INT:event.severity}-%{INT:event.code}" + ] + } + }, + "comments": [] + }, + { + "id": "filter_mutate_5", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_2", + "add_field": { + "event.action": "firewall-rule" + } + }, + "comments": [] + }, + { + "id": "filter_if_6", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[event.code] == \"105012\"", + "plugins": [ + { + "id": "filter_grok_7", + "type": "filter", + "plugin": "grok", + "config": { + "id": "filter_grok_3", + "match": { + "message": [ + "Teardown dynamic %{WORD:network.transport} translation from %{DATA:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} to %{DATA:cisco.destination_interface}:%{IP:destination.address}/%{INT:destination.port} duration %{DATA:cisco.duration_hms}$" + ] + } + }, + "comments": [] + } + ], + "else_ifs": [ + { + "condition": "[event.code] == \"106001\"", + "plugins": [ + { + "id": "filter_dissect_8", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_1", + "mapping": { + "message": "%{network.direction} %{network.transport} connection %{event.outcome} from %{source.address}/%{source.port} to %{destination.address}/%{destination.port} flags %{} on interface %{source_interface}" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_9", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_3", + "add_field": { + "event.category": "network_traffic" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"106002\"", + "plugins": [ + { + "id": "filter_dissect_10", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_2", + "mapping": { + "message": "%{network.transport} Connection %{event.outcome} by %{network.direction} list %{cisco.list_id} src %{source.address} dest %{destination.address}" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_11", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_4", + "add_field": { + "event.category": "network_traffic" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"106006\"", + "plugins": [ + { + "id": "filter_dissect_12", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_3", + "mapping": { + "message": "%{event.outcome} %{network.direction} %{network.transport} from %{source.address}/%{source.port} to %{destination.address}/%{destination.port} on interface %{cisco.source_interface}" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_13", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_5", + "add_field": { + "event.category": "network_traffic" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"106007\"", + "plugins": [ + { + "id": "filter_dissect_14", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_4", + "mapping": { + "message": "%{event.outcome} %{network.direction} %{network.transport} from %{source.address}/%{source.port} to %{destination.address}/%{destination.port} due to %{network.protocol} %{}" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_15", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_6", + "add_field": { + "event.category": "network_traffic" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"106010\"", + "plugins": [ + { + "id": "filter_dissect_16", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_5", + "mapping": { + "message": "%{event.outcome} %{network.direction} %{network.transport} src %{cisco.source_interface}:%{source.address}/%{source.port} %{} dst %{cisco.destination_interface}:%{destination.address}/%{destination.port} %{}" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_17", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_7", + "add_field": { + "event.category": "network_traffic" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"106013\"", + "plugins": [ + { + "id": "filter_dissect_18", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_6", + "mapping": { + "message": "Dropping echo request from %{source.address} to PAT address %{destination.address}" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_19", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_8", + "add_field": { + "network.transport": "icmp", + "network.direction": "inbound" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"106014\"", + "plugins": [ + { + "id": "filter_dissect_20", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_7", + "mapping": { + "message": "%{event.outcome} %{network.direction} %{network.transport} src %{cisco.source_interface}:%{source.address} %{}dst %{cisco.destination_interface}:%{destination.address} %{}" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_21", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_9", + "add_field": { + "event.category": "network_traffic" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"106015\"", + "plugins": [ + { + "id": "filter_dissect_22", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_8", + "mapping": { + "message": "%{event.outcome} %{network.transport} (no connection) from %{source.address}/%{source.port} to %{destination.address}/%{destination.port} flags %{} on interface %{cisco.source_interface}" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_23", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_10", + "add_field": { + "event.category": "nat_translation" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"106016\"", + "plugins": [ + { + "id": "filter_dissect_24", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_9", + "mapping": { + "message": "%{event.outcome} IP spoof from (%{source.address}) to %{destination.address} on interface %{cisco.source_interface}" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"106017\"", + "plugins": [ + { + "id": "filter_dissect_25", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_10", + "mapping": { + "message": "%{event.outcome} IP due to Land Attack from %{source.address} to %{destination.address}" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"106018\"", + "plugins": [ + { + "id": "filter_dissect_26", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_11", + "mapping": { + "message": "%{network.transport} packet type %{cisco.icmp_type} %{event.outcome} by %{network.direction} list %{cisco.list_id} src %{source.address} dest %{destination.address}" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"106020\"", + "plugins": [ + { + "id": "filter_dissect_27", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_12", + "mapping": { + "message": "%{event.outcome} IP teardrop fragment (size = %{}, offset = %{}) from %{source.address} to %{destination.address}" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"106021\"", + "plugins": [ + { + "id": "filter_dissect_28", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_13", + "mapping": { + "message": "%{event.outcome} %{network.transport} reverse path check from %{source.address} to %{destination.address} on interface %{cisco.source_interface}" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"106022\"", + "plugins": [ + { + "id": "filter_dissect_29", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_14", + "mapping": { + "message": "%{event.outcome} %{network.transport} connection spoof from %{source.address} to %{destination.address} on interface %{cisco.source_interface}" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"106023\"", + "plugins": [ + { + "id": "filter_grok_30", + "type": "filter", + "plugin": "grok", + "config": { + "id": "filter_grok_4", + "match": { + "message": [ + "%{WORD:event.outcome} %{WORD:network.transport} src %{WORD:cisco.source_interface}:%{IPORHOST:source.address}?(/%{INT:source.port}) dst %{WORD:cisco.destination_interface}:%{IPORHOST:destination.address}?(/%{INT:destination.port}) by access-group \"%{DATA:cisco.list_id}\"", + "%{WORD:event.outcome} %{WORD:network.transport} src %{WORD:cisco.source_interface}:%{IPORHOST:source.address} dst %{WORD:destination.direction}:%{IPORHOST:destination.address} \\(%{DATA}\\) by access-group \"%{DATA:cisco.list_id}\"", + "%{WORD:event.outcome} %{WORD:network.transport} src %{WORD:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} dst %{WORD:cisco.destination.interface}:%{IPORHOST:destination.address}/%{INT:destination.port} by access-group \"%{DATA:cisco.list_id}\"" + ] + } + }, + "comments": [] + }, + { + "id": "filter_mutate_31", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_11", + "add_field": { + "event.category": "network_traffic" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"106027\"", + "plugins": [ + { + "id": "filter_dissect_32", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_15", + "mapping": { + "message": "%{} %{event.outcome} src %{source.address} dst %{destination.address} by access-group \"%{cisco.list_id}\"%{}" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"106100\"", + "plugins": [ + { + "id": "filter_dissect_33", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_16", + "mapping": { + "message": "access-list %{cisco.list_id} %{event.outcome} %{network.transport} %{cisco.source_interface}/%{source.address}(%{source.port}) -> %{cisco.destination_interface}/%{destination.address}(%{destination.port}) %{}" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"106102\"", + "plugins": [ + { + "id": "filter_dissect_34", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_17", + "mapping": { + "message": "access-list %{cisco.list_id} %{event.outcome} %{network.transport} for user %{cisco.username} %{cisco.source_interface}/%{source.address} %{source.port} %{cisco.destination_interface}/%{destination.address} %{destination.port} %{}" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"106103\"", + "plugins": [ + { + "id": "filter_dissect_35", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_18", + "mapping": { + "message": "access-list %{cisco.list_id} %{event.outcome} %{network.transport} for user %{cisco.username} %{cisco.source_interface}/%{source.address} %{source.port} %{cisco.destination_interface}/%{destination.address} %{destination.port} %{}" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"113004\"", + "plugins": [ + { + "id": "filter_grok_36", + "type": "filter", + "plugin": "grok", + "config": { + "id": "filter_grok_5", + "match": { + "message": [ + "AAA user accounting %{WORD:cisco.auth_outcome} : server =%{SPACE}%{IP:source.address} : user =%{SPACE}%{DATA:source.user.name}$" + ] + } + }, + "comments": [] + }, + { + "id": "filter_mutate_37", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_12", + "add_field": { + "event.category": "authentication" + } + }, + "comments": [] + }, + { + "id": "filter_if_38", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[cisco.auth_outcome] == \"Successful\"", + "plugins": [ + { + "id": "filter_mutate_39", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_13", + "add_field": { + "event.action": "authentication_success" + } + }, + "comments": [] + } + ], + "else_ifs": [], + "else": { + "plugins": [ + { + "id": "filter_mutate_40", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_14", + "add_field": { + "event.action": "authentication_failure" + } + }, + "comments": [] + } + ] + } + } + } + ] + }, + { + "condition": "[event.code] == \"302015\" or [event.code] == \"302013\"", + "plugins": [ + { + "id": "filter_grok_41", + "type": "filter", + "plugin": "grok", + "config": { + "id": "filter_grok_6", + "match": { + "message": [ + "Built %{WORD:network.direction} %{WORD:network.transport} connection %{INT} for %{DATA:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} \\(%{IP}|\\) to %{DATA:cisco.destination_interface}:%{IPORHOST:destination.address}/%{INT:destination.port} \\(%{DATA}\\)", + "Built %{WORD:network.direction} %{WORD:network.transport} connection %{INT} for %{DATA:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} \\(%{DATA}\\)?(\\(%{DATA:cisco.source_username}\\)) to %{DATA:cisco.destination_interface}:%{IPORHOST:destination.address}/%{INT:destination.port} \\(%{DATA}\\) ?(\\(%{DATA:cisco.username}\\))", + "Built %{WORD:network.direction} %{WORD:network.transport} connection %{INT:cisco.connection_id} for %{WORD:cisco.source_interface}:%{IPORHOST:source.address}\\/%{INT:source.port} \\(%{DATA}\\) to %{WORD:cisco.destination_interface}:%{IPORHOST:destination.address}/%{INT:destination.port}" + ] + } + }, + "comments": [] + }, + { + "id": "filter_mutate_42", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_15", + "add_field": { + "event.category": "nat_translation" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"110003\"", + "plugins": [ + { + "id": "filter_grok_43", + "type": "filter", + "plugin": "grok", + "config": { + "id": "filter_grok_7", + "match": { + "message": [ + "%{DATA:cisco.event_error} for %{WORD:network.transport} from %{DATA:cisco.source_interface}:%{IP:source.address}\\/%{INT:source.port} to %{DATA:cisco.destination_interface}:%{IP:destination.address}\\/%{INT:destination.port}" + ] + } + }, + "comments": [] + }, + { + "id": "filter_mutate_44", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_16", + "add_field": { + "event.category": "error" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"113019\"", + "plugins": [ + { + "id": "filter_grok_45", + "type": "filter", + "plugin": "grok", + "config": { + "id": "filter_grok_8", + "match": { + "message": [ + "Group = %{DATA:cisco.group}, Username = %{DATA:user.name}, IP = %{IP:cisco.client_vpn_ip}, %{DATA:cisco.client_vpn_action}\\. Session Type: %{DATA:cisco.session_type}, Duration: %{DATA:cisco.duration}, Bytes xmt: %{INT:cisco.vpn_transmit_byte_summary}, Bytes rcv: %{INT:cisco.vpn_receive_byte_summary}, Reason: %{DATA:cisco.client_vpn_outcome}$" + ] + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"304001\"", + "plugins": [ + { + "id": "filter_dissect_46", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_19", + "mapping": { + "message": "%{source.address} %{}ccessed URL %{destination.address}:%{url.original}" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_47", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_17", + "add_field": { + "event.outcome": "allow" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"304002\"", + "plugins": [ + { + "id": "filter_dissect_48", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_20", + "mapping": { + "message": "Access %{event.outcome} URL %{url.original} SRC %{source.address} %{}EST %{destination.address} on interface %{cisco.source_interface}" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"305011\"", + "plugins": [ + { + "id": "filter_grok_49", + "type": "filter", + "plugin": "grok", + "config": { + "id": "filter_grok_9", + "match": { + "message": [ + "Built dynamic %{WORD:network.transport} translation from %{DATA:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} to %{DATA:cisco.destination_interface}:%{IP:destination.address}/%{INT:destination.port}" + ] + } + }, + "comments": [] + }, + { + "id": "filter_mutate_50", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_18", + "add_field": { + "event.category": "nat_translation" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"305012\"", + "plugins": [ + { + "id": "filter_grok_51", + "type": "filter", + "plugin": "grok", + "config": { + "id": "filter_grok_10", + "match": { + "message": [ + "Teardown dynamic %{WORD:network.transport} translation from %{DATA:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} to %{DATA:cisco.destination_interface}:%{IP:destination.address}/%{INT:destination.port}" + ] + } + }, + "comments": [] + }, + { + "id": "filter_mutate_52", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_19", + "add_field": { + "event.category": "nat_translation" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"313001\"", + "plugins": [ + { + "id": "filter_dissect_53", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_21", + "mapping": { + "message": "%{event.outcome} %{network.transport} type=%{cisco.icmp_type}, code=%{cisco.icmp_code} from %{source.address} on interface %{cisco.source_interface}" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"313004\"", + "plugins": [ + { + "id": "filter_dissect_54", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_22", + "mapping": { + "message": "%{event.outcome} %{network.transport} type=%{cisco.icmp_type}, from%{}addr %{source.address} on interface %{cisco.source_interface} to %{destination.address}: no matching session" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"313005\"", + "plugins": [ + { + "id": "filter_dissect_55", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_23", + "mapping": { + "message": "No matching connection for %{network.transport} error message: %{} on %{cisco.source_interface} interface.%{}riginal IP payload: %{}" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"313008\"", + "plugins": [ + { + "id": "filter_dissect_56", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_24", + "mapping": { + "message": "%{event.outcome} %{network.transport} type=%{cisco.icmp_type} , code=%{cisco.icmp_code} from %{source.address} on interface %{cisco.source_interface}" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"313009\"", + "plugins": [ + { + "id": "filter_dissect_57", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_25", + "mapping": { + "message": "%{event.outcome} invalid %{network.transport} code %{cisco.icmp_code} , for %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"322001\"", + "plugins": [ + { + "id": "filter_dissect_58", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_26", + "mapping": { + "message": "%{event.outcome} MAC address %{source.mac}, possible spoof attempt on interface %{cisco.source_interface}" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"338001\"", + "plugins": [ + { + "id": "filter_dissect_59", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_27", + "mapping": { + "message": "Dynamic filter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{source.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"338002\"", + "plugins": [ + { + "id": "filter_dissect_60", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_28", + "mapping": { + "message": "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{destination.domain}" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_61", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_20", + "add_field": { + "server.domain": "[destination.domain]" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"338003\"", + "plugins": [ + { + "id": "filter_dissect_62", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_29", + "mapping": { + "message": "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"338004\"", + "plugins": [ + { + "id": "filter_dissect_63", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_30", + "mapping": { + "message": "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"338005\"", + "plugins": [ + { + "id": "filter_dissect_64", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_31", + "mapping": { + "message": "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{source.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_65", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_21", + "add_field": { + "server.domain": "[source.domain]" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"338006\"", + "plugins": [ + { + "id": "filter_dissect_66", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_32", + "mapping": { + "message": "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{destination.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_67", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_22", + "add_field": { + "server.domain": "server.domain" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"338007\"", + "plugins": [ + { + "id": "filter_dissect_68", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_33", + "mapping": { + "message": "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"338008\"", + "plugins": [ + { + "id": "filter_dissect_69", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_34", + "mapping": { + "message": "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"338101\"", + "plugins": [ + { + "id": "filter_dissect_70", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_35", + "mapping": { + "message": "Dynamic %{}ilter %{event.outcome} white%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{source.domain}" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_71", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_23", + "add_field": { + "server.domain": "server.domain" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"338102\"", + "plugins": [ + { + "id": "filter_dissect_72", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_36", + "mapping": { + "message": "Dynamic %{}ilter %{event.outcome} white%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{destination.domain}" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_73", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_24", + "add_field": { + "server.domain": "server.domain" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"338103\"", + "plugins": [ + { + "id": "filter_dissect_74", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_37", + "mapping": { + "message": "Dynamic %{}ilter %{event.outcome} white%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{}" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"338104\"", + "plugins": [ + { + "id": "filter_dissect_75", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_38", + "mapping": { + "message": "Dynamic %{}ilter %{event.outcome} white%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{}" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"338201\"", + "plugins": [ + { + "id": "filter_dissect_76", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_39", + "mapping": { + "message": "Dynamic %{}ilter %{event.outcome} grey%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{source.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_77", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_25", + "add_field": { + "server.domain": "server.domain" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"338202\"", + "plugins": [ + { + "id": "filter_dissect_78", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_40", + "mapping": { + "message": "Dynamic %{}ilter %{event.outcome} grey%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{destination.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_79", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_26", + "add_field": { + "server.domain": "server.domain" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"338203\"", + "plugins": [ + { + "id": "filter_dissect_80", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_41", + "mapping": { + "message": "Dynamic %{}ilter %{event.outcome} grey%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{source.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_81", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_27", + "add_field": { + "server.domain": "server.domain" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"338204\"", + "plugins": [ + { + "id": "filter_dissect_82", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_42", + "mapping": { + "message": "Dynamic %{}ilter %{event.outcome} grey%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{destination.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_83", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_28", + "add_field": { + "server.domain": "server.domain" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"338301\"", + "plugins": [ + { + "id": "filter_dissect_84", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_43", + "mapping": { + "message": "Intercepted DNS reply for domain %{source.domain} from %{cisco.source_interface}:%{source.address}/%{source.port} to %{cisco.destination_interface}:%{destination.address}/%{destination.port}, matched %{cisco.list_id}" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_85", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_29", + "add_field": { + "client.address": "client.address" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_86", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_30", + "add_field": { + "client.port": "client.port" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_87", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_31", + "add_field": { + "server.address": "server.address" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_88", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_32", + "add_field": { + "server.port": "server.port" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] in [\"302014\", \"302016\", \"302018\", \"302021\", \"302036\", \"302304\", \"302306\", \"302020\"]", + "plugins": [ + { + "id": "filter_grok_89", + "type": "filter", + "plugin": "grok", + "config": { + "id": "filter_grok_11", + "pattern_definitions": { + "NOTCOLON": "[^:]*", + "ECSSOURCEIPORHOST": "(?:%{IP:source.address}|%{HOSTNAME:source.domain})", + "ECSDESTIPORHOST": "(?:%{IP:destination.address}|%{HOSTNAME:destination.domain})", + "MAPPEDSRC": "(?:%{DATA:cisco.mapped_source_ip}|%{HOSTNAME})" + }, + "match": { + "message": [ + "Teardown %{NOTSPACE:network.transport} (?:state-bypass )?connection %{NOTSPACE:cisco.connection_id} (?:for|from) %{NOTCOLON}:%{DATA:source.address}/%{NUMBER:source.port:int}?(\\(%{DATA:cisco.source_username}\\)|) ?to %{NOTCOLON:cisco.destination_interface}:%{DATA:destination.address}/%{NUMBER:destination.port:int}?(\\(%{DATA:cisco.source_username}\\)|) ?(?:duration %{TIME:cisco.duration_hms} bytes %{NUMBER:network.bytes:int})%{GREEDYDATA}", + "Teardown %{NOTSPACE:network.transport} (?:state-bypass )?connection %{NOTSPACE:cisco.connection_id} (?:for|from) %{NOTCOLON}:%{DATA:source.address}/%{NUMBER:source.port:int} (?:%{NOTSPACE:cisco.source_username} )?to %{NOTCOLON:cisco.destination_interface}:%{DATA:destination.address}/%{NUMBER:destination.port:int} (?:%{NOTSPACE:cisco.destination_username} )?(?:duration %{TIME:cisco.duration_hms} bytes %{NUMBER:network.bytes:int})%{GREEDYDATA}", + "Teardown %{NOTSPACE:network.transport} connection for faddr (?:%{NOTCOLON:cisco.source_interface}:)?%{ECSDESTIPORHOST}/%{NUMBER} (?:%{NOTSPACE:cisco.destination_username} )?gaddr (?:%{NOTCOLON}:)?%{MAPPEDSRC}/%{NUMBER} laddr (?:%{NOTCOLON:cisco.source_interface}:)?%{ECSSOURCEIPORHOST}/%{NUMBER}(?: %{NOTSPACE:cisco.source_username})?%{GREEDYDATA}", + "Built %{WORD:network.direction} %{NOTSPACE:network.transport} connection for faddr (?:%{NOTCOLON:cisco.source_interface}:)?%{ECSDESTIPORHOST}/%{NUMBER} (?:%{NOTSPACE:cisco.destination_username} )?gaddr (?:%{NOTCOLON}:)?%{MAPPEDSRC}/%{NUMBER} laddr (?:%{NOTCOLON:cisco.source_interface}:)?%{ECSSOURCEIPORHOST}/%{NUMBER}(?: %{NOTSPACE:cisco.source_username})?%{GREEDYDATA}" + ] + } + }, + "comments": [] + }, + { + "id": "filter_mutate_90", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_33", + "add_field": { + "event.category": "nat_translation" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"419002\"", + "plugins": [ + { + "id": "filter_grok_91", + "type": "filter", + "plugin": "grok", + "config": { + "id": "filter_grok_12", + "match": { + "message": [ + "%{DATA:cisco.event_error} from %{WORD:cisco.source_interface}:%{IPORHOST:source.address}\\/%{INT:source.port} to %{WORD:cisco.destination_interface}:%{IPORHOST:destination.address}\\/%{INT:destination.port}" + ] + } + }, + "comments": [] + }, + { + "id": "filter_mutate_92", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_34", + "add_field": { + "event.category": "error" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] in [\"733100\", \"752015\", \"752012\"]", + "plugins": [ + { + "id": "filter_grok_93", + "type": "filter", + "plugin": "grok", + "config": { + "id": "filter_grok_13", + "match": { + "message": [ + "%{GREEDYDATA:cisco.event_error}" + ] + } + }, + "comments": [] + }, + { + "id": "filter_mutate_94", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_35", + "add_field": { + "event.category": "error" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"716002\"", + "plugins": [ + { + "id": "filter_grok_95", + "type": "filter", + "plugin": "grok", + "config": { + "id": "filter_grok_14", + "match": { + "message": [ + "Group \\<%{DATA:cisco.group} User \\<%{DATA:user.name}\\> IP \\<%{IP:cisco.client_vpn_ip}\\> WebVPN session %{WORD:cisco.client_vpn_session_outcome}\\: %{DATA:cisco.web_vpn_action}\\." + ] + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] in ['722022', '722033', '722055', '722051', '113039', '722023', '722037']", + "plugins": [ + { + "id": "filter_grok_96", + "type": "filter", + "plugin": "grok", + "config": { + "id": "filter_grok_15", + "match": { + "message": [ + "Group \\<%{DATA:cisco.group} User \\<%{DATA:user.name}\\> IP \\<%{IP:cisco.client_vpn_ip}\\> %{GREEDYDATA:cisco.message}" + ] + } + }, + "comments": [] + } + ] + } + ], + "else": { + "plugins": [ + { + "id": "filter_grok_97", + "type": "filter", + "plugin": "grok", + "config": { + "id": "filter_grok_16", + "match": { + "message": [ + "forced_failure" + ] + } + }, + "comments": [] + } + ] + } + } + }, + { + "id": "filter_if_98", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[event.category] == \"nat_translation\"", + "plugins": [ + { + "id": "filter_drop_99", + "type": "filter", + "plugin": "drop", + "config": { + "id": "filter_drop_1" + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_if_100", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[source.address]", + "plugins": [ + { + "id": "filter_grok_101", + "type": "filter", + "plugin": "grok", + "config": { + "id": "filter_grok_17", + "match": { + "source.address": [ + "(?:%{IP:source.ip}|%{GREEDYDATA:source.domain})" + ] + } + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_if_102", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[destination.address]", + "plugins": [ + { + "id": "filter_grok_103", + "type": "filter", + "plugin": "grok", + "config": { + "id": "filter_grok_18", + "match": { + "destination.address": [ + "(?:%{IP:destination.ip}|%{GREEDYDATA:destination.domain})" + ] + } + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_if_104", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[client.address]", + "plugins": [ + { + "id": "filter_grok_105", + "type": "filter", + "plugin": "grok", + "config": { + "id": "filter_grok_19", + "match": { + "client.address": [ + "(?:%{IP:client.ip}|%{GREEDYDATA:client.domain})" + ] + } + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_if_106", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[server.address]", + "plugins": [ + { + "id": "filter_grok_107", + "type": "filter", + "plugin": "grok", + "config": { + "id": "filter_grok_20", + "match": { + "server.address": [ + "(?:%{IP:server.ip}|%{GREEDYDATA:server.domain})" + ] + } + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_mutate_108", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_36", + "lowercase": [ + "network.transport", + "network.protocol", + "network.direction", + "event.outcome" + ] + }, + "comments": [] + }, + { + "id": "filter_if_109", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[event.outcome] == \"est-allowed\"", + "plugins": [ + { + "id": "filter_mutate_110", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_37", + "update": { + "event.outcome": "allow" + } + }, + "comments": [] + } + ], + "else_ifs": [ + { + "condition": "[event.outcome] == \"permitted\"", + "plugins": [ + { + "id": "filter_mutate_111", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_38", + "update": { + "event.outcome": "allow" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.outcome] == \"denied\"", + "plugins": [ + { + "id": "filter_mutate_112", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_39", + "update": { + "event.outcome": "deny" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.outcome] == \"dropped\"", + "plugins": [ + { + "id": "filter_mutate_113", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_40", + "update": { + "event.outcome": "deny" + } + }, + "comments": [] + } + ] + } + ], + "else": null + } + }, + { + "id": "filter_if_114", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[network.transport] == \"icmpv6\"", + "plugins": [ + { + "id": "filter_mutate_115", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_41", + "update": { + "network.transport": "ipv6-icmp" + } + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_translate_116", + "type": "filter", + "plugin": "translate", + "config": { + "id": "filter_translate_1", + "field": "network.transport", + "destination": "network.iana_number", + "dictionary": { + "icmp": "1", + "igmp": "2", + "ipv4": "4", + "tcp": "6", + "egp": "8", + "igp": "9", + "pup": "12", + "udp": "17", + "rdp": "27", + "irtp": "28", + "dccp": "33", + "idpr": "35", + "ipv6": "41", + "ipv6-route": "43", + "ipv6-frag": "44", + "rsvp": "46", + "gre": "47", + "esp": "50", + "ipv6-icmp": "58", + "ipv6-nonxt": "59", + "ipv6-opts": "60" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_117", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_42", + "remove_field": [ + "ciscotag", + "timestamp" + ] + }, + "comments": [] + }, + { + "id": "filter_mutate_118", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_43", + "add_field": { + "event.module": "cisco", + "event.dataset": "asa" + } + }, + "comments": [] + }, + { + "id": "filter_translate_119", + "type": "filter", + "plugin": "translate", + "config": { + "id": "filter_translate_2", + "field": "[event.severity]", + "destination": "[log.level]", + "dictionary": { + "0": "emergency", + "1": "alert", + "2": "critical", + "3": "error", + "4": "warning", + "5": "notification", + "6": "informational", + "7": "debug" + } + }, + "comments": [] + } + ], + "output": [ + { + "id": "output_elasticsearch_120", + "type": "output", + "plugin": "elasticsearch", + "config": { + "id": "output_elasticsearch_1", + "api_key": "${es_api_key}", + "hosts": "https://homedc-90e54c.es.us-east-2.aws.elastic-cloud.com:443", + "index": "asa-1.2", + "pipeline": "asa" + }, + "comments": [] + } + ] +} diff --git a/tests/Common/unit/conversion_data/components/test-asa.json b/tests/Common/unit/conversion_data/components/test-asa.json new file mode 100644 index 0000000..3945c20 --- /dev/null +++ b/tests/Common/unit/conversion_data/components/test-asa.json @@ -0,0 +1,1846 @@ +{ + "input": [ + { + "id": "input_udp_0", + "type": "input", + "plugin": "udp", + "config": { + "id": "input_udp_1", + "port": "5119" + }, + "comments": [] + }, + { + "id": "input_cloudwatch_1", + "type": "input", + "plugin": "cloudwatch", + "config": {}, + "comments": [] + } + ], + "filter": [ + { + "id": "filter_mutate_2", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_1", + "rename": { + "message": "log.original", + "host": "observer.ip" + }, + "copy": { + "host": "sysloghost" + } + }, + "comments": [] + }, + { + "id": "filter_grok_3", + "type": "filter", + "plugin": "grok", + "config": { + "id": "filter_grok_1", + "match": { + "log.original": [ + "%{CISCO_TAGGED_SYSLOG} %{GREEDYDATA:message}", + "^<%{POSINT:syslog_pri}>%{DATA}: %%{DATA:ciscotag}: %{GREEDYDATA:message}", + "^<%{POSINT:syslog_pri}>%%{DATA:ciscotag}: %{GREEDYDATA:message}" + ] + } + }, + "comments": [] + }, + { + "id": "filter_grok_4", + "type": "filter", + "plugin": "grok", + "config": { + "id": "filter_grok_2", + "match": { + "ciscotag": [ + "%{WORD}-%{INT:event.severity}-%{INT:event.code}", + "%{WORD}-%{WORD}-%{INT:event.severity}-%{INT:event.code}" + ] + } + }, + "comments": [] + }, + { + "id": "filter_mutate_5", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_2", + "add_field": { + "event.action": "firewall-rule" + } + }, + "comments": [] + }, + { + "id": "filter_if_6", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[event.code] == \"105012\"", + "plugins": [ + { + "id": "filter_grok_7", + "type": "filter", + "plugin": "grok", + "config": { + "id": "filter_grok_3", + "match": { + "message": [ + "Teardown dynamic %{WORD:network.transport} translation from %{DATA:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} to %{DATA:cisco.destination_interface}:%{IP:destination.address}/%{INT:destination.port} duration %{DATA:cisco.duration_hms}$" + ] + } + }, + "comments": [] + } + ], + "else_ifs": [ + { + "condition": "[event.code] == \"106001\"", + "plugins": [ + { + "id": "filter_dissect_8", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_1", + "mapping": { + "message": "%{network.direction} %{network.transport} connection %{event.outcome} from %{source.address}/%{source.port} to %{destination.address}/%{destination.port} flags %{} on interface %{source_interface}" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_9", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_3", + "add_field": { + "event.category": "network_traffic" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"106002\"", + "plugins": [ + { + "id": "filter_dissect_10", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_2", + "mapping": { + "message": "%{network.transport} Connection %{event.outcome} by %{network.direction} list %{cisco.list_id} src %{source.address} dest %{destination.address}" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_11", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_4", + "add_field": { + "event.category": "network_traffic" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"106006\"", + "plugins": [ + { + "id": "filter_dissect_12", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_3", + "mapping": { + "message": "%{event.outcome} %{network.direction} %{network.transport} from %{source.address}/%{source.port} to %{destination.address}/%{destination.port} on interface %{cisco.source_interface}" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_13", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_5", + "add_field": { + "event.category": "network_traffic" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"106007\"", + "plugins": [ + { + "id": "filter_dissect_14", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_4", + "mapping": { + "message": "%{event.outcome} %{network.direction} %{network.transport} from %{source.address}/%{source.port} to %{destination.address}/%{destination.port} due to %{network.protocol} %{}" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_15", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_6", + "add_field": { + "event.category": "network_traffic" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"106010\"", + "plugins": [ + { + "id": "filter_dissect_16", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_5", + "mapping": { + "message": "%{event.outcome} %{network.direction} %{network.transport} src %{cisco.source_interface}:%{source.address}/%{source.port} %{} dst %{cisco.destination_interface}:%{destination.address}/%{destination.port} %{}" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_17", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_7", + "add_field": { + "event.category": "network_traffic" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"106013\"", + "plugins": [ + { + "id": "filter_dissect_18", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_6", + "mapping": { + "message": "Dropping echo request from %{source.address} to PAT address %{destination.address}" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_19", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_8", + "add_field": { + "network.transport": "icmp", + "network.direction": "inbound" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"106014\"", + "plugins": [ + { + "id": "filter_dissect_20", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_7", + "mapping": { + "message": "%{event.outcome} %{network.direction} %{network.transport} src %{cisco.source_interface}:%{source.address} %{}dst %{cisco.destination_interface}:%{destination.address} %{}" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_21", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_9", + "add_field": { + "event.category": "network_traffic" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"106015\"", + "plugins": [ + { + "id": "filter_dissect_22", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_8", + "mapping": { + "message": "%{event.outcome} %{network.transport} (no connection) from %{source.address}/%{source.port} to %{destination.address}/%{destination.port} flags %{} on interface %{cisco.source_interface}" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_23", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_10", + "add_field": { + "event.category": "nat_translation" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"106016\"", + "plugins": [ + { + "id": "filter_dissect_24", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_9", + "mapping": { + "message": "%{event.outcome} IP spoof from (%{source.address}) to %{destination.address} on interface %{cisco.source_interface}" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"106017\"", + "plugins": [ + { + "id": "filter_dissect_25", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_10", + "mapping": { + "message": "%{event.outcome} IP due to Land Attack from %{source.address} to %{destination.address}" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"106018\"", + "plugins": [ + { + "id": "filter_dissect_26", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_11", + "mapping": { + "message": "%{network.transport} packet type %{cisco.icmp_type} %{event.outcome} by %{network.direction} list %{cisco.list_id} src %{source.address} dest %{destination.address}" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"106020\"", + "plugins": [ + { + "id": "filter_dissect_27", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_12", + "mapping": { + "message": "%{event.outcome} IP teardrop fragment (size = %{}, offset = %{}) from %{source.address} to %{destination.address}" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"106021\"", + "plugins": [ + { + "id": "filter_dissect_28", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_13", + "mapping": { + "message": "%{event.outcome} %{network.transport} reverse path check from %{source.address} to %{destination.address} on interface %{cisco.source_interface}" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"106022\"", + "plugins": [ + { + "id": "filter_dissect_29", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_14", + "mapping": { + "message": "%{event.outcome} %{network.transport} connection spoof from %{source.address} to %{destination.address} on interface %{cisco.source_interface}" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"106023\"", + "plugins": [ + { + "id": "filter_grok_30", + "type": "filter", + "plugin": "grok", + "config": { + "id": "filter_grok_4", + "match": { + "message": [ + "%{WORD:event.outcome} %{WORD:network.transport} src %{WORD:cisco.source_interface}:%{IPORHOST:source.address}?(/%{INT:source.port}) dst %{WORD:cisco.destination_interface}:%{IPORHOST:destination.address}?(/%{INT:destination.port}) by access-group \"%{DATA:cisco.list_id}\"", + "%{WORD:event.outcome} %{WORD:network.transport} src %{WORD:cisco.source_interface}:%{IPORHOST:source.address} dst %{WORD:destination.direction}:%{IPORHOST:destination.address} \\(%{DATA}\\) by access-group \"%{DATA:cisco.list_id}\"", + "%{WORD:event.outcome} %{WORD:network.transport} src %{WORD:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} dst %{WORD:cisco.destination.interface}:%{IPORHOST:destination.address}/%{INT:destination.port} by access-group \"%{DATA:cisco.list_id}\"" + ] + } + }, + "comments": [] + }, + { + "id": "filter_mutate_31", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_11", + "add_field": { + "event.category": "network_traffic" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"106027\"", + "plugins": [ + { + "id": "filter_dissect_32", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_15", + "mapping": { + "message": "%{} %{event.outcome} src %{source.address} dst %{destination.address} by access-group \"%{cisco.list_id}\"%{}" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"106100\"", + "plugins": [ + { + "id": "filter_dissect_33", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_16", + "mapping": { + "message": "access-list %{cisco.list_id} %{event.outcome} %{network.transport} %{cisco.source_interface}/%{source.address}(%{source.port}) -> %{cisco.destination_interface}/%{destination.address}(%{destination.port}) %{}" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"106102\"", + "plugins": [ + { + "id": "filter_dissect_34", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_17", + "mapping": { + "message": "access-list %{cisco.list_id} %{event.outcome} %{network.transport} for user %{cisco.username} %{cisco.source_interface}/%{source.address} %{source.port} %{cisco.destination_interface}/%{destination.address} %{destination.port} %{}" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"106103\"", + "plugins": [ + { + "id": "filter_dissect_35", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_18", + "mapping": { + "message": "access-list %{cisco.list_id} %{event.outcome} %{network.transport} for user %{cisco.username} %{cisco.source_interface}/%{source.address} %{source.port} %{cisco.destination_interface}/%{destination.address} %{destination.port} %{}" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"113004\"", + "plugins": [ + { + "id": "filter_grok_36", + "type": "filter", + "plugin": "grok", + "config": { + "id": "filter_grok_5", + "match": { + "message": [ + "AAA user accounting %{WORD:cisco.auth_outcome} : server =%{SPACE}%{IP:source.address} : user =%{SPACE}%{DATA:source.user.name}$" + ] + } + }, + "comments": [] + }, + { + "id": "filter_mutate_37", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_12", + "add_field": { + "event.category": "authentication" + } + }, + "comments": [] + }, + { + "id": "filter_if_38", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[cisco.auth_outcome] == \"Successful\"", + "plugins": [ + { + "id": "filter_mutate_39", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_13", + "add_field": { + "event.action": "authentication_success" + } + }, + "comments": [] + } + ], + "else_ifs": [], + "else": { + "plugins": [ + { + "id": "filter_mutate_40", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_14", + "add_field": { + "event.action": "authentication_failure" + } + }, + "comments": [] + } + ] + } + } + } + ] + }, + { + "condition": "[event.code] == \"302015\" or [event.code] == \"302013\"", + "plugins": [ + { + "id": "filter_grok_41", + "type": "filter", + "plugin": "grok", + "config": { + "id": "filter_grok_6", + "match": { + "message": [ + "Built %{WORD:network.direction} %{WORD:network.transport} connection %{INT} for %{DATA:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} \\(%{IP}|\\) to %{DATA:cisco.destination_interface}:%{IPORHOST:destination.address}/%{INT:destination.port} \\(%{DATA}\\)", + "Built %{WORD:network.direction} %{WORD:network.transport} connection %{INT} for %{DATA:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} \\(%{DATA}\\)?(\\(%{DATA:cisco.source_username}\\)) to %{DATA:cisco.destination_interface}:%{IPORHOST:destination.address}/%{INT:destination.port} \\(%{DATA}\\) ?(\\(%{DATA:cisco.username}\\))", + "Built %{WORD:network.direction} %{WORD:network.transport} connection %{INT:cisco.connection_id} for %{WORD:cisco.source_interface}:%{IPORHOST:source.address}\\/%{INT:source.port} \\(%{DATA}\\) to %{WORD:cisco.destination_interface}:%{IPORHOST:destination.address}/%{INT:destination.port}" + ] + } + }, + "comments": [] + }, + { + "id": "filter_mutate_42", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_15", + "add_field": { + "event.category": "nat_translation" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"110003\"", + "plugins": [ + { + "id": "filter_grok_43", + "type": "filter", + "plugin": "grok", + "config": { + "id": "filter_grok_7", + "match": { + "message": [ + "%{DATA:cisco.event_error} for %{WORD:network.transport} from %{DATA:cisco.source_interface}:%{IP:source.address}\\/%{INT:source.port} to %{DATA:cisco.destination_interface}:%{IP:destination.address}\\/%{INT:destination.port}" + ] + } + }, + "comments": [] + }, + { + "id": "filter_mutate_44", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_16", + "add_field": { + "event.category": "error" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"113019\"", + "plugins": [ + { + "id": "filter_grok_45", + "type": "filter", + "plugin": "grok", + "config": { + "id": "filter_grok_8", + "match": { + "message": [ + "Group = %{DATA:cisco.group}, Username = %{DATA:user.name}, IP = %{IP:cisco.client_vpn_ip}, %{DATA:cisco.client_vpn_action}\\. Session Type: %{DATA:cisco.session_type}, Duration: %{DATA:cisco.duration}, Bytes xmt: %{INT:cisco.vpn_transmit_byte_summary}, Bytes rcv: %{INT:cisco.vpn_receive_byte_summary}, Reason: %{DATA:cisco.client_vpn_outcome}$" + ] + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"304001\"", + "plugins": [ + { + "id": "filter_dissect_46", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_19", + "mapping": { + "message": "%{source.address} %{}ccessed URL %{destination.address}:%{url.original}" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_47", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_17", + "add_field": { + "event.outcome": "allow" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"304002\"", + "plugins": [ + { + "id": "filter_dissect_48", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_20", + "mapping": { + "message": "Access %{event.outcome} URL %{url.original} SRC %{source.address} %{}EST %{destination.address} on interface %{cisco.source_interface}" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"305011\"", + "plugins": [ + { + "id": "filter_grok_49", + "type": "filter", + "plugin": "grok", + "config": { + "id": "filter_grok_9", + "match": { + "message": [ + "Built dynamic %{WORD:network.transport} translation from %{DATA:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} to %{DATA:cisco.destination_interface}:%{IP:destination.address}/%{INT:destination.port}" + ] + } + }, + "comments": [] + }, + { + "id": "filter_mutate_50", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_18", + "add_field": { + "event.category": "nat_translation" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"305012\"", + "plugins": [ + { + "id": "filter_grok_51", + "type": "filter", + "plugin": "grok", + "config": { + "id": "filter_grok_10", + "match": { + "message": [ + "Teardown dynamic %{WORD:network.transport} translation from %{DATA:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} to %{DATA:cisco.destination_interface}:%{IP:destination.address}/%{INT:destination.port}" + ] + } + }, + "comments": [] + }, + { + "id": "filter_mutate_52", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_19", + "add_field": { + "event.category": "nat_translation" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"313001\"", + "plugins": [ + { + "id": "filter_dissect_53", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_21", + "mapping": { + "message": "%{event.outcome} %{network.transport} type=%{cisco.icmp_type}, code=%{cisco.icmp_code} from %{source.address} on interface %{cisco.source_interface}" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"313004\"", + "plugins": [ + { + "id": "filter_dissect_54", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_22", + "mapping": { + "message": "%{event.outcome} %{network.transport} type=%{cisco.icmp_type}, from%{}addr %{source.address} on interface %{cisco.source_interface} to %{destination.address}: no matching session" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"313005\"", + "plugins": [ + { + "id": "filter_dissect_55", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_23", + "mapping": { + "message": "No matching connection for %{network.transport} error message: %{} on %{cisco.source_interface} interface.%{}riginal IP payload: %{}" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"313008\"", + "plugins": [ + { + "id": "filter_dissect_56", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_24", + "mapping": { + "message": "%{event.outcome} %{network.transport} type=%{cisco.icmp_type} , code=%{cisco.icmp_code} from %{source.address} on interface %{cisco.source_interface}" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"313009\"", + "plugins": [ + { + "id": "filter_dissect_57", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_25", + "mapping": { + "message": "%{event.outcome} invalid %{network.transport} code %{cisco.icmp_code} , for %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"322001\"", + "plugins": [ + { + "id": "filter_dissect_58", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_26", + "mapping": { + "message": "%{event.outcome} MAC address %{source.mac}, possible spoof attempt on interface %{cisco.source_interface}" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"338001\"", + "plugins": [ + { + "id": "filter_dissect_59", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_27", + "mapping": { + "message": "Dynamic filter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{source.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"338002\"", + "plugins": [ + { + "id": "filter_dissect_60", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_28", + "mapping": { + "message": "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{destination.domain}" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_61", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_20", + "add_field": { + "server.domain": "[destination.domain]" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"338003\"", + "plugins": [ + { + "id": "filter_dissect_62", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_29", + "mapping": { + "message": "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"338004\"", + "plugins": [ + { + "id": "filter_dissect_63", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_30", + "mapping": { + "message": "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"338005\"", + "plugins": [ + { + "id": "filter_dissect_64", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_31", + "mapping": { + "message": "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{source.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_65", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_21", + "add_field": { + "server.domain": "[source.domain]" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"338006\"", + "plugins": [ + { + "id": "filter_dissect_66", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_32", + "mapping": { + "message": "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{destination.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_67", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_22", + "add_field": { + "server.domain": "server.domain" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"338007\"", + "plugins": [ + { + "id": "filter_dissect_68", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_33", + "mapping": { + "message": "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"338008\"", + "plugins": [ + { + "id": "filter_dissect_69", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_34", + "mapping": { + "message": "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"338101\"", + "plugins": [ + { + "id": "filter_dissect_70", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_35", + "mapping": { + "message": "Dynamic %{}ilter %{event.outcome} white%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{source.domain}" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_71", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_23", + "add_field": { + "server.domain": "server.domain" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"338102\"", + "plugins": [ + { + "id": "filter_dissect_72", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_36", + "mapping": { + "message": "Dynamic %{}ilter %{event.outcome} white%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{destination.domain}" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_73", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_24", + "add_field": { + "server.domain": "server.domain" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"338103\"", + "plugins": [ + { + "id": "filter_dissect_74", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_37", + "mapping": { + "message": "Dynamic %{}ilter %{event.outcome} white%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{}" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"338104\"", + "plugins": [ + { + "id": "filter_dissect_75", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_38", + "mapping": { + "message": "Dynamic %{}ilter %{event.outcome} white%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{}" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"338201\"", + "plugins": [ + { + "id": "filter_dissect_76", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_39", + "mapping": { + "message": "Dynamic %{}ilter %{event.outcome} grey%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{source.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_77", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_25", + "add_field": { + "server.domain": "server.domain" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"338202\"", + "plugins": [ + { + "id": "filter_dissect_78", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_40", + "mapping": { + "message": "Dynamic %{}ilter %{event.outcome} grey%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{destination.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_79", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_26", + "add_field": { + "server.domain": "server.domain" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"338203\"", + "plugins": [ + { + "id": "filter_dissect_80", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_41", + "mapping": { + "message": "Dynamic %{}ilter %{event.outcome} grey%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{source.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_81", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_27", + "add_field": { + "server.domain": "server.domain" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"338204\"", + "plugins": [ + { + "id": "filter_dissect_82", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_42", + "mapping": { + "message": "Dynamic %{}ilter %{event.outcome} grey%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{destination.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_83", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_28", + "add_field": { + "server.domain": "server.domain" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"338301\"", + "plugins": [ + { + "id": "filter_dissect_84", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "filter_dissect_43", + "mapping": { + "message": "Intercepted DNS reply for domain %{source.domain} from %{cisco.source_interface}:%{source.address}/%{source.port} to %{cisco.destination_interface}:%{destination.address}/%{destination.port}, matched %{cisco.list_id}" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_85", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_29", + "add_field": { + "client.address": "client.address" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_86", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_30", + "add_field": { + "client.port": "client.port" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_87", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_31", + "add_field": { + "server.address": "server.address" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_88", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_32", + "add_field": { + "server.port": "server.port" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] in [\"302014\", \"302016\", \"302018\", \"302021\", \"302036\", \"302304\", \"302306\", \"302020\"]", + "plugins": [ + { + "id": "filter_grok_89", + "type": "filter", + "plugin": "grok", + "config": { + "id": "filter_grok_11", + "pattern_definitions": { + "NOTCOLON": "[^:]*", + "ECSSOURCEIPORHOST": "(?:%{IP:source.address}|%{HOSTNAME:source.domain})", + "ECSDESTIPORHOST": "(?:%{IP:destination.address}|%{HOSTNAME:destination.domain})", + "MAPPEDSRC": "(?:%{DATA:cisco.mapped_source_ip}|%{HOSTNAME})" + }, + "match": { + "message": [ + "Teardown %{NOTSPACE:network.transport} (?:state-bypass )?connection %{NOTSPACE:cisco.connection_id} (?:for|from) %{NOTCOLON}:%{DATA:source.address}/%{NUMBER:source.port:int}?(\\(%{DATA:cisco.source_username}\\)|) ?to %{NOTCOLON:cisco.destination_interface}:%{DATA:destination.address}/%{NUMBER:destination.port:int}?(\\(%{DATA:cisco.source_username}\\)|) ?(?:duration %{TIME:cisco.duration_hms} bytes %{NUMBER:network.bytes:int})%{GREEDYDATA}", + "Teardown %{NOTSPACE:network.transport} (?:state-bypass )?connection %{NOTSPACE:cisco.connection_id} (?:for|from) %{NOTCOLON}:%{DATA:source.address}/%{NUMBER:source.port:int} (?:%{NOTSPACE:cisco.source_username} )?to %{NOTCOLON:cisco.destination_interface}:%{DATA:destination.address}/%{NUMBER:destination.port:int} (?:%{NOTSPACE:cisco.destination_username} )?(?:duration %{TIME:cisco.duration_hms} bytes %{NUMBER:network.bytes:int})%{GREEDYDATA}", + "Teardown %{NOTSPACE:network.transport} connection for faddr (?:%{NOTCOLON:cisco.source_interface}:)?%{ECSDESTIPORHOST}/%{NUMBER} (?:%{NOTSPACE:cisco.destination_username} )?gaddr (?:%{NOTCOLON}:)?%{MAPPEDSRC}/%{NUMBER} laddr (?:%{NOTCOLON:cisco.source_interface}:)?%{ECSSOURCEIPORHOST}/%{NUMBER}(?: %{NOTSPACE:cisco.source_username})?%{GREEDYDATA}", + "Built %{WORD:network.direction} %{NOTSPACE:network.transport} connection for faddr (?:%{NOTCOLON:cisco.source_interface}:)?%{ECSDESTIPORHOST}/%{NUMBER} (?:%{NOTSPACE:cisco.destination_username} )?gaddr (?:%{NOTCOLON}:)?%{MAPPEDSRC}/%{NUMBER} laddr (?:%{NOTCOLON:cisco.source_interface}:)?%{ECSSOURCEIPORHOST}/%{NUMBER}(?: %{NOTSPACE:cisco.source_username})?%{GREEDYDATA}" + ] + } + }, + "comments": [] + }, + { + "id": "filter_mutate_90", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_33", + "add_field": { + "event.category": "nat_translation" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"419002\"", + "plugins": [ + { + "id": "filter_grok_91", + "type": "filter", + "plugin": "grok", + "config": { + "id": "filter_grok_12", + "match": { + "message": [ + "%{DATA:cisco.event_error} from %{WORD:cisco.source_interface}:%{IPORHOST:source.address}\\/%{INT:source.port} to %{WORD:cisco.destination_interface}:%{IPORHOST:destination.address}\\/%{INT:destination.port}" + ] + } + }, + "comments": [] + }, + { + "id": "filter_mutate_92", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_34", + "add_field": { + "event.category": "error" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] in [\"733100\", \"752015\", \"752012\"]", + "plugins": [ + { + "id": "filter_grok_93", + "type": "filter", + "plugin": "grok", + "config": { + "id": "filter_grok_13", + "match": { + "message": [ + "%{GREEDYDATA:cisco.event_error}" + ] + } + }, + "comments": [] + }, + { + "id": "filter_mutate_94", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_35", + "add_field": { + "event.category": "error" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] == \"716002\"", + "plugins": [ + { + "id": "filter_grok_95", + "type": "filter", + "plugin": "grok", + "config": { + "id": "filter_grok_14", + "match": { + "message": [ + "Group \\<%{DATA:cisco.group} User \\<%{DATA:user.name}\\> IP \\<%{IP:cisco.client_vpn_ip}\\> WebVPN session %{WORD:cisco.client_vpn_session_outcome}\\: %{DATA:cisco.web_vpn_action}\\." + ] + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.code] in ['722022', '722033', '722055', '722051', '113039', '722023', '722037']", + "plugins": [ + { + "id": "filter_grok_96", + "type": "filter", + "plugin": "grok", + "config": { + "id": "filter_grok_15", + "match": { + "message": [ + "Group \\<%{DATA:cisco.group} User \\<%{DATA:user.name}\\> IP \\<%{IP:cisco.client_vpn_ip}\\> %{GREEDYDATA:cisco.message}" + ] + } + }, + "comments": [] + } + ] + } + ], + "else": { + "plugins": [ + { + "id": "filter_grok_97", + "type": "filter", + "plugin": "grok", + "config": { + "id": "filter_grok_16", + "match": { + "message": [ + "forced_failure" + ] + } + }, + "comments": [] + } + ] + } + } + }, + { + "id": "filter_if_98", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[event.category] == \"nat_translation\"", + "plugins": [ + { + "id": "filter_drop_99", + "type": "filter", + "plugin": "drop", + "config": { + "id": "filter_drop_1" + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_if_100", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[source.address]", + "plugins": [ + { + "id": "filter_grok_101", + "type": "filter", + "plugin": "grok", + "config": { + "id": "filter_grok_17", + "match": { + "source.address": [ + "(?:%{IP:source.ip}|%{GREEDYDATA:source.domain})" + ] + } + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_if_102", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[destination.address]", + "plugins": [ + { + "id": "filter_grok_103", + "type": "filter", + "plugin": "grok", + "config": { + "id": "filter_grok_18", + "match": { + "destination.address": [ + "(?:%{IP:destination.ip}|%{GREEDYDATA:destination.domain})" + ] + } + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_if_104", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[client.address]", + "plugins": [ + { + "id": "filter_grok_105", + "type": "filter", + "plugin": "grok", + "config": { + "id": "filter_grok_19", + "match": { + "client.address": [ + "(?:%{IP:client.ip}|%{GREEDYDATA:client.domain})" + ] + } + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_if_106", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[server.address]", + "plugins": [ + { + "id": "filter_grok_107", + "type": "filter", + "plugin": "grok", + "config": { + "id": "filter_grok_20", + "match": { + "server.address": [ + "(?:%{IP:server.ip}|%{GREEDYDATA:server.domain})" + ] + } + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_mutate_108", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_36", + "lowercase": [ + "network.transport", + "network.protocol", + "network.direction", + "event.outcome" + ] + }, + "comments": [] + }, + { + "id": "filter_if_109", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[event.outcome] == \"est-allowed\"", + "plugins": [ + { + "id": "filter_mutate_110", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_37", + "update": { + "event.outcome": "allow" + } + }, + "comments": [] + } + ], + "else_ifs": [ + { + "condition": "[event.outcome] == \"permitted\"", + "plugins": [ + { + "id": "filter_mutate_111", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_38", + "update": { + "event.outcome": "allow" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.outcome] == \"denied\"", + "plugins": [ + { + "id": "filter_mutate_112", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_39", + "update": { + "event.outcome": "deny" + } + }, + "comments": [] + } + ] + }, + { + "condition": "[event.outcome] == \"dropped\"", + "plugins": [ + { + "id": "filter_mutate_113", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_40", + "update": { + "event.outcome": "deny" + } + }, + "comments": [] + } + ] + } + ], + "else": null + } + }, + { + "id": "filter_if_114", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[network.transport] == \"icmpv6\"", + "plugins": [ + { + "id": "filter_mutate_115", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_41", + "update": { + "network.transport": "ipv6-icmp" + } + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_translate_116", + "type": "filter", + "plugin": "translate", + "config": { + "id": "filter_translate_1", + "field": "network.transport", + "destination": "network.iana_number", + "dictionary": { + "icmp": "1", + "igmp": "2", + "ipv4": "4", + "tcp": "6", + "egp": "8", + "igp": "9", + "pup": "12", + "udp": "17", + "rdp": "27", + "irtp": "28", + "dccp": "33", + "idpr": "35", + "ipv6": "41", + "ipv6-route": "43", + "ipv6-frag": "44", + "rsvp": "46", + "gre": "47", + "esp": "50", + "ipv6-icmp": "58", + "ipv6-nonxt": "59", + "ipv6-opts": "60" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_117", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_42", + "remove_field": [ + "ciscotag", + "timestamp" + ] + }, + "comments": [] + }, + { + "id": "filter_mutate_118", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_43", + "add_field": { + "event.module": "cisco", + "event.dataset": "asa" + } + }, + "comments": [] + }, + { + "id": "filter_translate_119", + "type": "filter", + "plugin": "translate", + "config": { + "id": "filter_translate_2", + "field": "[event.severity]", + "destination": "[log.level]", + "dictionary": { + "0": "emergency", + "1": "alert", + "2": "critical", + "3": "error", + "4": "warning", + "5": "notification", + "6": "informational", + "7": "debug" + } + }, + "comments": [] + } + ], + "output": [ + { + "id": "output_elasticsearch_120", + "type": "output", + "plugin": "elasticsearch", + "config": { + "id": "output_elasticsearch_1", + "api_key": "${es_api_key}", + "hosts": "https://homedc-90e54c.es.us-east-2.aws.elastic-cloud.com:443", + "index": "asa-1.2", + "pipeline": "asa" + }, + "comments": [] + } + ] +} diff --git a/tests/Common/unit/conversion_data/components/test-boolean-numeric.json b/tests/Common/unit/conversion_data/components/test-boolean-numeric.json new file mode 100644 index 0000000..d708725 --- /dev/null +++ b/tests/Common/unit/conversion_data/components/test-boolean-numeric.json @@ -0,0 +1,134 @@ +{ + "input": [ + { + "id": "input_tcp_0", + "type": "input", + "plugin": "tcp", + "config": { + "port": 5000, + "ssl_enable": "false", + "buffer_size": 65536 + }, + "comments": [] + }, + { + "id": "input_udp_1", + "type": "input", + "plugin": "udp", + "config": { + "port": 514, + "queue_size": 2000, + "workers": 4 + }, + "comments": [] + } + ], + "filter": [ + { + "id": "filter_grok_2", + "type": "filter", + "plugin": "grok", + "config": { + "match": { + "message": "%{NUMBER:duration:float} %{NUMBER:status:int}" + }, + "keep_empty_captures": "false", + "tag_on_failure": [ + "_grokfailure" + ], + "timeout_millis": 30000, + "break_on_match": "true" + }, + "comments": [] + }, + { + "id": "filter_mutate_3", + "type": "filter", + "plugin": "mutate", + "config": { + "convert": { + "duration": "float", + "status": "integer", + "bytes": "integer" + } + }, + "comments": [] + }, + { + "id": "filter_if_4", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[duration] > 1.5", + "plugins": [ + { + "id": "filter_mutate_5", + "type": "filter", + "plugin": "mutate", + "config": { + "add_field": { + "slow_request": "true" + } + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_if_6", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[status] >= 500", + "plugins": [ + { + "id": "filter_mutate_7", + "type": "filter", + "plugin": "mutate", + "config": { + "add_field": { + "is_error": "true", + "error_code": 500, + "threshold_pct": 0.99 + } + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_throttle_8", + "type": "filter", + "plugin": "throttle", + "config": { + "before_count": 3, + "after_count": 1, + "period": 60, + "key": "%{host}", + "add_tag": [ + "throttled" + ] + }, + "comments": [] + } + ], + "output": [ + { + "id": "output_stdout_9", + "type": "output", + "plugin": "stdout", + "config": { + "codec": { + "rubydebug": {} + } + }, + "comments": [] + } + ] +} diff --git a/tests/Common/unit/conversion_data/components/test-comments-brace-in-comment.json b/tests/Common/unit/conversion_data/components/test-comments-brace-in-comment.json new file mode 100644 index 0000000..c136b37 --- /dev/null +++ b/tests/Common/unit/conversion_data/components/test-comments-brace-in-comment.json @@ -0,0 +1,72 @@ +{ + "input": [], + "filter": [ + { + "id": "filter_mutate_0", + "type": "filter", + "plugin": "mutate", + "config": { + "add_field": { + "real_field": "real_value" + }, + "remove_field": [ + "unwanted" + ] + }, + "comments": [ + "add_field => {\"this_key\" => \"this_value\"}", + "rename => {\"old_field\" => \"new_field\"}", + "remove_field => [\"field1\", \"field2\"]", + "replace => {\"message\" => \"override: %{message}\"}" + ] + }, + { + "id": "filter_grok_1", + "type": "filter", + "plugin": "grok", + "config": { + "match": { + "message": "%{GREEDYDATA:raw_message}" + } + }, + "comments": [ + "match => { \"message\" => \"%{COMBINEDAPACHELOG}\" }", + "pattern_definitions => { \"MY_PATTERN\" => \"\\\\w+\" }" + ] + }, + { + "id": "filter_date_2", + "type": "filter", + "plugin": "date", + "config": { + "match": [ + "timestamp", + "ISO8601" + ], + "target": "@timestamp" + }, + "comments": [ + "match => [\"timestamp\", \"dd/MMM/yyyy:HH:mm:ss Z\", \"ISO8601\"]", + "target => \"@timestamp\"" + ] + }, + { + "id": "filter_translate_3", + "type": "filter", + "plugin": "translate", + "config": { + "field": "status_code", + "destination": "status_label", + "dictionary": { + "200": "OK", + "404": "Not Found" + }, + "fallback": "Unknown" + }, + "comments": [ + "dictionary => { \"200\" => \"OK\", \"404\" => \"Not Found\", \"500\" => \"Error\" }" + ] + } + ], + "output": [] +} diff --git a/tests/Common/unit/conversion_data/components/test-comments-mixed.json b/tests/Common/unit/conversion_data/components/test-comments-mixed.json new file mode 100644 index 0000000..651f5f2 --- /dev/null +++ b/tests/Common/unit/conversion_data/components/test-comments-mixed.json @@ -0,0 +1,154 @@ +{ + "input": [ + { + "id": "input_comment_0", + "type": "input", + "plugin": "comment", + "config": { + "text": "inline on section opener" + } + }, + { + "id": "input_udp_1", + "type": "input", + "plugin": "udp", + "config": { + "port": 5140, + "buffer_size": 65536, + "tags": [ + "syslog" + ] + }, + "comments": [ + "standalone inside plugin", + "add_field => {\"commented_out\" => \"value\"} standalone with braces", + "inline on plugin opener", + "inline on scalar value", + "inline on array" + ] + }, + { + "id": "input_comment_2", + "type": "input", + "plugin": "comment", + "config": { + "text": "inline on plugin closer -> section comment" + } + } + ], + "filter": [ + { + "id": "filter_comment_3", + "type": "filter", + "plugin": "comment", + "config": { + "text": "inline on filter opener\nstandalone at section level" + } + }, + { + "id": "filter_mutate_4", + "type": "filter", + "plugin": "mutate", + "config": { + "add_field": { + "key1": "value1", + "key2": "value2" + }, + "remove_field": [ + "old" + ] + }, + "comments": [ + "standalone before first pair", + "standalone between pairs", + "standalone at end of plugin", + "inline on mutate opener", + "inline on hash opener", + "inline on hash pair", + "inline on hash closer", + "inline on array pair" + ] + }, + { + "id": "filter_comment_5", + "type": "filter", + "plugin": "comment", + "config": { + "text": "inline on plugin closer -> section comment\nstandalone between plugins" + } + }, + { + "id": "filter_if_6", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[type] == \"web\"", + "plugins": [ + { + "id": "filter_comment_7", + "type": "filter", + "plugin": "comment", + "config": { + "text": "standalone inside conditional" + } + }, + { + "id": "filter_grok_8", + "type": "filter", + "plugin": "grok", + "config": { + "match": { + "message": "%{COMBINEDAPACHELOG}" + } + }, + "comments": [ + "standalone inside nested plugin", + "match => {\"message\" => \"%{GREEDYDATA}\"} standalone with braces in conditional", + "inline on nested plugin opener", + "inline inside nested hash", + "inline on nested hash closer" + ] + }, + { + "id": "filter_comment_9", + "type": "filter", + "plugin": "comment", + "config": { + "text": "inline on nested plugin closer" + } + }, + { + "id": "filter_comment_10", + "type": "filter", + "plugin": "comment", + "config": { + "text": "standalone at end of conditional block" + } + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_drop_11", + "type": "filter", + "plugin": "drop", + "config": {}, + "comments": [] + } + ], + "output": [ + { + "id": "output_stdout_12", + "type": "output", + "plugin": "stdout", + "config": { + "codec": { + "rubydebug": {} + } + }, + "comments": [] + } + ] +} diff --git a/tests/Common/unit/conversion_data/components/test-comments-plugin-inline.json b/tests/Common/unit/conversion_data/components/test-comments-plugin-inline.json new file mode 100644 index 0000000..7d04adf --- /dev/null +++ b/tests/Common/unit/conversion_data/components/test-comments-plugin-inline.json @@ -0,0 +1,111 @@ +{ + "input": [ + { + "id": "input_udp_0", + "type": "input", + "plugin": "udp", + "config": { + "port": 5140, + "buffer_size": 65536, + "tags": [ + "udp", + "syslog" + ] + }, + "comments": [ + "inline comment on plugin opener", + "inline comment on a scalar value", + "another scalar inline comment", + "inline comment after an array" + ] + }, + { + "id": "input_comment_1", + "type": "input", + "plugin": "comment", + "config": { + "text": "inline comment on plugin closer \u2014 becomes section-level comment" + } + } + ], + "filter": [ + { + "id": "filter_mutate_2", + "type": "filter", + "plugin": "mutate", + "config": { + "add_field": { + "first": "value1", + "second": "value2" + }, + "remove_field": [ + "unwanted", + "junk" + ], + "rename": { + "old_name": "new_name" + } + }, + "comments": [ + "opener comment", + "inline comment on hash opener", + "inline comment on hash pair", + "another hash pair comment", + "inline comment on hash closer", + "inline on array", + "another hash opener with inline", + "pair comment", + "hash closer comment" + ] + }, + { + "id": "filter_comment_3", + "type": "filter", + "plugin": "comment", + "config": { + "text": "plugin closer \u2014 section-level" + } + }, + { + "id": "filter_drop_4", + "type": "filter", + "plugin": "drop", + "config": {}, + "comments": [ + "opener comment on empty plugin" + ] + }, + { + "id": "filter_comment_5", + "type": "filter", + "plugin": "comment", + "config": { + "text": "closer comment on empty plugin" + } + } + ], + "output": [ + { + "id": "output_stdout_6", + "type": "output", + "plugin": "stdout", + "config": { + "codec": { + "rubydebug": {} + } + }, + "comments": [ + "output plugin with inline", + "inline on codec line" + ] + }, + { + "id": "output_comment_7", + "type": "output", + "plugin": "comment", + "config": { + "text": "closer" + } + } + ] +} diff --git a/tests/Common/unit/conversion_data/components/test-comments-section-opener.json b/tests/Common/unit/conversion_data/components/test-comments-section-opener.json new file mode 100644 index 0000000..b580988 --- /dev/null +++ b/tests/Common/unit/conversion_data/components/test-comments-section-opener.json @@ -0,0 +1,100 @@ +{ + "input": [ + { + "id": "input_comment_0", + "type": "input", + "plugin": "comment", + "config": { + "text": "inline comment on input opener" + } + }, + { + "id": "input_beats_1", + "type": "input", + "plugin": "beats", + "config": { + "port": 5044 + }, + "comments": [ + "inline on beats opener" + ] + } + ], + "filter": [ + { + "id": "filter_comment_2", + "type": "filter", + "plugin": "comment", + "config": { + "text": "inline comment on filter opener" + } + }, + { + "id": "filter_mutate_3", + "type": "filter", + "plugin": "mutate", + "config": { + "add_field": { + "processed": "true" + } + }, + "comments": [] + }, + { + "id": "filter_comment_4", + "type": "filter", + "plugin": "comment", + "config": { + "text": "standalone comment between plugins at section level" + } + }, + { + "id": "filter_drop_5", + "type": "filter", + "plugin": "drop", + "config": {}, + "comments": [] + } + ], + "output": [ + { + "id": "output_comment_6", + "type": "output", + "plugin": "comment", + "config": { + "text": "inline comment on output opener" + } + }, + { + "id": "output_elasticsearch_7", + "type": "output", + "plugin": "elasticsearch", + "config": { + "hosts": [ + "localhost:9200" + ], + "index": "logs-%{+YYYY.MM.dd}" + }, + "comments": [] + }, + { + "id": "output_comment_8", + "type": "output", + "plugin": "comment", + "config": { + "text": "standalone at section level before second output plugin" + } + }, + { + "id": "output_stdout_9", + "type": "output", + "plugin": "stdout", + "config": { + "codec": { + "rubydebug": {} + } + }, + "comments": [] + } + ] +} diff --git a/tests/Common/unit/conversion_data/components/test-comments-standalone-in-plugin.json b/tests/Common/unit/conversion_data/components/test-comments-standalone-in-plugin.json new file mode 100644 index 0000000..d7d2a80 --- /dev/null +++ b/tests/Common/unit/conversion_data/components/test-comments-standalone-in-plugin.json @@ -0,0 +1,61 @@ +{ + "input": [], + "filter": [ + { + "id": "filter_mutate_0", + "type": "filter", + "plugin": "mutate", + "config": { + "add_field": { + "field1": "value1" + }, + "remove_field": [ + "junk" + ] + }, + "comments": [ + "This is a standalone comment at the top of a plugin block", + "Standalone comment in the middle of a plugin block", + "Another standalone at the bottom" + ] + }, + { + "id": "filter_grok_1", + "type": "filter", + "plugin": "grok", + "config": { + "match": { + "message": "%{COMBINEDAPACHELOG}" + } + }, + "comments": [ + "Standalone comment before the only config key", + "Standalone at the end of plugin" + ] + }, + { + "id": "filter_comment_2", + "type": "filter", + "plugin": "comment", + "config": { + "text": "Section-level comment between plugins" + } + }, + { + "id": "filter_mutate_3", + "type": "filter", + "plugin": "mutate", + "config": { + "uppercase": [ + "log_level" + ] + }, + "comments": [ + "Leading standalone", + "Second leading standalone", + "Trailing standalone" + ] + } + ], + "output": [] +} diff --git a/tests/Common/unit/conversion_data/components/test-complex2.json b/tests/Common/unit/conversion_data/components/test-complex2.json new file mode 100644 index 0000000..5756543 --- /dev/null +++ b/tests/Common/unit/conversion_data/components/test-complex2.json @@ -0,0 +1,952 @@ +{ + "input": [ + { + "id": "input_comment_0", + "type": "input", + "plugin": "comment", + "config": { + "text": "\n\"LogstashUI kitchen sink\" pipeline\nGoal: be extremely feature-rich while staying within known-valid plugin options.\n\nBeats / Elastic Agent style shippers" + } + }, + { + "id": "input_beats_1", + "type": "input", + "plugin": "beats", + "config": { + "id": "in_beats_5044", + "port": "5044", + "add_field": { + "ingest_transport": "beats" + }, + "tags": [ + "from_beats" + ] + }, + "comments": [] + }, + { + "id": "input_comment_2", + "type": "input", + "plugin": "comment", + "config": { + "text": "JSON-over-TCP (common for app logs)" + } + }, + { + "id": "input_tcp_3", + "type": "input", + "plugin": "tcp", + "config": { + "id": "in_tcp_json_5514", + "port": 5514, + "mode": "server", + "codec": { + "json": {} + }, + "add_field": { + "ingest_transport": "tcp" + }, + "tags": [ + "from_tcp" + ] + }, + "comments": [] + }, + { + "id": "input_comment_4", + "type": "input", + "plugin": "comment", + "config": { + "text": "Syslog-ish UDP" + } + }, + { + "id": "input_udp_5", + "type": "input", + "plugin": "udp", + "config": { + "id": "in_udp_5515", + "port": 5515, + "codec": { + "plain": {} + }, + "add_field": { + "ingest_transport": "udp" + }, + "tags": [ + "from_udp" + ] + }, + "comments": [] + }, + { + "id": "input_comment_6", + "type": "input", + "plugin": "comment", + "config": { + "text": "HTTP event intake (webhooks, apps posting JSON, etc.)" + } + }, + { + "id": "input_http_7", + "type": "input", + "plugin": "http", + "config": { + "id": "in_http_8080", + "port": 8080, + "codec": { + "json": {} + }, + "add_field": { + "ingest_transport": "http" + }, + "tags": [ + "from_http" + ] + }, + "comments": [] + }, + { + "id": "input_comment_8", + "type": "input", + "plugin": "comment", + "config": { + "text": "Local dev/testing input" + } + }, + { + "id": "input_stdin_9", + "type": "input", + "plugin": "stdin", + "config": { + "id": "in_stdin", + "codec": { + "line": {} + }, + "add_field": { + "ingest_transport": "stdin" + }, + "tags": [ + "from_stdin" + ] + }, + "comments": [] + }, + { + "id": "input_comment_10", + "type": "input", + "plugin": "comment", + "config": { + "text": "Synthetic test data (makes it easy to validate end-to-end quickly)" + } + }, + { + "id": "input_generator_11", + "type": "input", + "plugin": "generator", + "config": { + "id": "in_generator", + "lines": [ + "Feb 21 09:12:01 host1 sshd[123]: Failed password for invalid user admin from 10.1.2.3 port 51234 ssh2", + "{\"@timestamp\":\"2026-02-21T14:12:02Z\",\"message\":\"GET /health 200\",\"source_ip\":\"8.8.8.8\",\"user_agent\":\"Mozilla/5.0\"}", + "level=info service=api latency_ms=42 source_ip=192.168.1.50 msg=\"request completed\"" + ], + "count": 1, + "add_field": { + "ingest_transport": "generator" + }, + "tags": [ + "from_generator" + ] + }, + "comments": [] + } + ], + "filter": [ + { + "id": "filter_comment_12", + "type": "filter", + "plugin": "comment", + "config": { + "text": "\nNormalize a few shared fields\n" + } + }, + { + "id": "filter_mutate_13", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "f_mutate_bootstrap", + "add_field": { + "[@metadata][pipeline]": "logstashui_kitchen_sink", + "event.module": "logstashui" + } + }, + "comments": [] + }, + { + "id": "filter_comment_14", + "type": "filter", + "plugin": "comment", + "config": { + "text": "Keep a canonical message field" + } + }, + { + "id": "filter_if_15", + "type": "filter", + "plugin": "if", + "config": { + "condition": "![message] and [event][original]", + "plugins": [ + { + "id": "filter_mutate_16", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "f_mutate_event_original_to_message", + "copy": { + "[event][original]": "message" + } + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_comment_17", + "type": "filter", + "plugin": "comment", + "config": { + "text": "\nTry to parse JSON *if* message looks like JSON (common when tcp/udp/plain feed JSON strings)\n" + } + }, + { + "id": "filter_if_18", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[message] =~ \"^[[:space:]]*\\\\{\"", + "plugins": [ + { + "id": "filter_json_19", + "type": "filter", + "plugin": "json", + "config": { + "id": "f_json_from_message", + "source": "message", + "target": "json", + "tag_on_failure": [ + "_jsonparsefailure_message" + ] + }, + "comments": [] + }, + { + "id": "filter_comment_20", + "type": "filter", + "plugin": "comment", + "config": { + "text": "If json parsed, promote a few expected keys (only if present)" + } + }, + { + "id": "filter_if_21", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[json][@timestamp]", + "plugins": [ + { + "id": "filter_mutate_22", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "f_promote_json_ts", + "copy": { + "[json][@timestamp]": "@timestamp" + } + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_if_23", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[json][source_ip]", + "plugins": [ + { + "id": "filter_mutate_24", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "f_promote_json_source_ip", + "copy": { + "[json][source_ip]": "source_ip" + } + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_if_25", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[json][user_agent]", + "plugins": [ + { + "id": "filter_mutate_26", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "f_promote_json_ua", + "copy": { + "[json][user_agent]": "user_agent" + } + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_comment_27", + "type": "filter", + "plugin": "comment", + "config": { + "text": "\nSyslog-ish parsing (UDP and some TCP)\n" + } + }, + { + "id": "filter_if_28", + "type": "filter", + "plugin": "if", + "config": { + "condition": "\"from_udp\" in [tags] or \"from_tcp\" in [tags]", + "plugins": [ + { + "id": "filter_comment_29", + "type": "filter", + "plugin": "comment", + "config": { + "text": "Try dissect first (fast) and fall back to grok" + } + }, + { + "id": "filter_dissect_30", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "f_dissect_syslogish", + "mapping": { + "message": "%{syslog_timestamp} %{syslog_host} %{syslog_program}[%{syslog_pid}]: %{syslog_message}" + }, + "tag_on_failure": [ + "_dissectfailure_syslogish" + ] + }, + "comments": [] + }, + { + "id": "filter_if_31", + "type": "filter", + "plugin": "if", + "config": { + "condition": "\"_dissectfailure_syslogish\" in [tags]", + "plugins": [ + { + "id": "filter_grok_32", + "type": "filter", + "plugin": "grok", + "config": { + "id": "f_grok_syslogish", + "match": { + "message": [ + "%{SYSLOGTIMESTAMP:syslog_timestamp} %{HOSTNAME:syslog_host} %{DATA:syslog_program}(?:\\[%{POSINT:syslog_pid}\\])?: %{GREEDYDATA:syslog_message}" + ] + }, + "tag_on_failure": [ + "_grokparsefailure_syslogish" + ] + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_comment_33", + "type": "filter", + "plugin": "comment", + "config": { + "text": "If we extracted a syslog timestamp, use it" + } + }, + { + "id": "filter_if_34", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[syslog_timestamp]", + "plugins": [ + { + "id": "filter_date_35", + "type": "filter", + "plugin": "date", + "config": { + "id": "f_date_syslog", + "match": [ + "syslog_timestamp", + "MMM d HH:mm:ss", + "MMM dd HH:mm:ss" + ], + "tag_on_failure": [ + "_dateparsefailure_syslog" + ] + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_comment_36", + "type": "filter", + "plugin": "comment", + "config": { + "text": "\nkey=value parsing for \u201cflat\u201d log lines\n" + } + }, + { + "id": "filter_if_37", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[message] =~ \"([A-Za-z0-9_.-]+)=([^\\\"]\\\\S+|\\\"[^\\\"]*\\\")\"", + "plugins": [ + { + "id": "filter_kv_38", + "type": "filter", + "plugin": "kv", + "config": { + "id": "f_kv_message", + "source": "message", + "trim_key": " ", + "trim_value": " ", + "value_split": "=", + "field_split_pattern": "\\s+", + "tag_on_failure": [ + "_kvfailure_message" + ] + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_comment_39", + "type": "filter", + "plugin": "comment", + "config": { + "text": "\nBasic typing / normalization\n" + } + }, + { + "id": "filter_mutate_40", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "f_mutate_normalize", + "rename": { + "msg": "message_short" + }, + "convert": { + "latency_ms": "integer" + }, + "lowercase": [ + "level" + ] + }, + "comments": [] + }, + { + "id": "filter_comment_41", + "type": "filter", + "plugin": "comment", + "config": { + "text": "\nEnrichments: useragent, geoip, cidr, dns\n" + } + }, + { + "id": "filter_if_42", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[user_agent]", + "plugins": [ + { + "id": "filter_useragent_43", + "type": "filter", + "plugin": "useragent", + "config": { + "id": "f_useragent", + "source": "user_agent", + "target": "user_agent_parsed" + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_comment_44", + "type": "filter", + "plugin": "comment", + "config": { + "text": "Canonicalize IP into source_ip if it exists elsewhere" + } + }, + { + "id": "filter_if_45", + "type": "filter", + "plugin": "if", + "config": { + "condition": "![source_ip] and [source][ip]", + "plugins": [ + { + "id": "filter_mutate_46", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "f_copy_source_ip", + "copy": { + "[source][ip]": "source_ip" + } + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_if_47", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[source_ip]", + "plugins": [ + { + "id": "filter_comment_48", + "type": "filter", + "plugin": "comment", + "config": { + "text": "Tag private vs public" + } + }, + { + "id": "filter_cidr_49", + "type": "filter", + "plugin": "cidr", + "config": { + "id": "f_cidr_private", + "address": [ + "%{source_ip}" + ], + "network": [ + "10.0.0.0/8", + "172.16.0.0/12", + "192.168.0.0/16" + ], + "add_tag": [ + "src_private" + ] + }, + "comments": [] + }, + { + "id": "filter_comment_50", + "type": "filter", + "plugin": "comment", + "config": { + "text": "GeoIP typically only makes sense for public IPs, so do it only if not private-tagged" + } + }, + { + "id": "filter_if_51", + "type": "filter", + "plugin": "if", + "config": { + "condition": "\"src_private\" not in [tags]", + "plugins": [ + { + "id": "filter_geoip_52", + "type": "filter", + "plugin": "geoip", + "config": { + "id": "f_geoip", + "source": "source_ip", + "target": "source_geo" + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_comment_53", + "type": "filter", + "plugin": "comment", + "config": { + "text": "Reverse DNS lookup; replace source_ip with hostname when possible (or leave as-is)" + } + }, + { + "id": "filter_dns_54", + "type": "filter", + "plugin": "dns", + "config": { + "id": "f_dns_reverse", + "reverse": [ + "source_ip" + ], + "action": "replace" + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_comment_55", + "type": "filter", + "plugin": "comment", + "config": { + "text": "\nTranslate severity/level into a normalized numeric\n" + } + }, + { + "id": "filter_translate_56", + "type": "filter", + "plugin": "translate", + "config": { + "id": "f_translate_level_to_severity", + "source": "level", + "target": "severity", + "dictionary": { + "trace": "0", + "debug": "1", + "info": "2", + "warn": "3", + "error": "4", + "fatal": "5" + }, + "fallback": "2" + }, + "comments": [] + }, + { + "id": "filter_mutate_57", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "f_convert_severity_int", + "convert": { + "severity": "integer" + } + }, + "comments": [] + }, + { + "id": "filter_comment_58", + "type": "filter", + "plugin": "comment", + "config": { + "text": "\nStable fingerprint for dedup / correlation\n" + } + }, + { + "id": "filter_fingerprint_59", + "type": "filter", + "plugin": "fingerprint", + "config": { + "id": "f_fingerprint_message", + "source": [ + "message" + ], + "method": "MURMUR3", + "target": "[@metadata][fingerprint]" + }, + "comments": [] + }, + { + "id": "filter_comment_60", + "type": "filter", + "plugin": "comment", + "config": { + "text": "\nExample branching: treat auth-ish messages specially\n" + } + }, + { + "id": "filter_if_61", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[syslog_program] == \"sshd\" or [message] =~ \"(?i)failed password|authentication failure|invalid user\"", + "plugins": [ + { + "id": "filter_mutate_62", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "f_tag_auth", + "add_tag": [ + "category_auth" + ], + "add_field": { + "event.category": "authentication" + } + }, + "comments": [] + } + ], + "else_ifs": [ + { + "condition": "[message] =~ \"(?i)GET\\\\s+/health|/ready|/live\"", + "plugins": [ + { + "id": "filter_mutate_63", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "f_tag_health", + "add_tag": [ + "category_healthcheck" + ], + "add_field": { + "event.category": "availability" + } + }, + "comments": [] + } + ] + } + ], + "else": { + "plugins": [ + { + "id": "filter_mutate_64", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "f_tag_generic", + "add_tag": [ + "category_generic" + ] + }, + "comments": [] + } + ] + } + } + }, + { + "id": "filter_comment_65", + "type": "filter", + "plugin": "comment", + "config": { + "text": "\nPrune down noisy fields (keeps top-level essentials)\n" + } + }, + { + "id": "filter_prune_66", + "type": "filter", + "plugin": "prune", + "config": { + "id": "f_prune", + "whitelist_names": [ + "^@timestamp$", + "^message$", + "^message_short$", + "^host$", + "^source_ip$", + "^source_geo$", + "^severity$", + "^level$", + "^tags$", + "^event\\..*$", + "^user_agent.*$", + "^syslog_.*$", + "^ingest_transport$" + ] + }, + "comments": [] + } + ], + "output": [ + { + "id": "output_comment_67", + "type": "output", + "plugin": "comment", + "config": { + "text": "Always see something in console during dev" + } + }, + { + "id": "output_stdout_68", + "type": "output", + "plugin": "stdout", + "config": { + "id": "out_stdout_rubydebug", + "codec": { + "rubydebug": { + "metadata": "true" + } + } + }, + "comments": [] + }, + { + "id": "output_comment_69", + "type": "output", + "plugin": "comment", + "config": { + "text": "Write to disk (great for debugging replay)" + } + }, + { + "id": "output_file_70", + "type": "output", + "plugin": "file", + "config": { + "id": "out_file_jsonl", + "path": "/tmp/logstashui-%{+YYYY.MM.dd}.jsonl", + "codec": { + "json_lines": {} + } + }, + "comments": [] + }, + { + "id": "output_comment_71", + "type": "output", + "plugin": "comment", + "config": { + "text": "Elasticsearch (local default)" + } + }, + { + "id": "output_elasticsearch_72", + "type": "output", + "plugin": "elasticsearch", + "config": { + "id": "out_es_local", + "hosts": [ + "http://localhost:9200" + ], + "index": "logstashui-%{+YYYY.MM.dd}", + "ilm_enabled": "false" + }, + "comments": [] + }, + { + "id": "output_comment_73", + "type": "output", + "plugin": "comment", + "config": { + "text": "Webhook back to your UI/API (example)" + } + }, + { + "id": "output_http_74", + "type": "output", + "plugin": "http", + "config": { + "id": "out_http_callback", + "url": "http://localhost:9000/logstash/callback", + "http_method": "post", + "format": "json" + }, + "comments": [] + }, + { + "id": "output_comment_75", + "type": "output", + "plugin": "comment", + "config": { + "text": "Kafka (example)" + } + }, + { + "id": "output_kafka_76", + "type": "output", + "plugin": "kafka", + "config": { + "id": "out_kafka", + "bootstrap_servers": "localhost:9092", + "topic_id": "logstashui-events" + }, + "comments": [] + }, + { + "id": "output_comment_77", + "type": "output", + "plugin": "comment", + "config": { + "text": "Pipeline-to-pipeline (requires another pipeline with pipeline input address => \"downstream\")" + } + }, + { + "id": "output_pipeline_78", + "type": "output", + "plugin": "pipeline", + "config": { + "id": "out_pipeline_downstream", + "send_to": [ + "downstream" + ] + }, + "comments": [] + } + ] +} diff --git a/tests/Common/unit/conversion_data/components/test-complex3.json b/tests/Common/unit/conversion_data/components/test-complex3.json new file mode 100644 index 0000000..1771e5a --- /dev/null +++ b/tests/Common/unit/conversion_data/components/test-complex3.json @@ -0,0 +1,1497 @@ +{ + "input": [ + { + "id": "input_beats_0", + "type": "input", + "plugin": "beats", + "config": { + "port": "5044", + "ssl": "true", + "ssl_certificate": "/etc/logstash/certs/server.crt", + "ssl_key": "/etc/logstash/certs/server.key", + "ssl_verify_mode": "force_peer", + "ssl_certificate_authorities": [ + "/etc/logstash/certs/ca.crt" + ], + "codec": { + "json": {} + }, + "type": "beats", + "tags": [ + "beats_ssl" + ] + }, + "comments": [] + }, + { + "id": "input_http_1", + "type": "input", + "plugin": "http", + "config": { + "port": "8080", + "codec": { + "json": {} + }, + "ssl": "true", + "ssl_certificate": "/etc/logstash/certs/http.crt", + "ssl_key": "/etc/logstash/certs/http.key", + "threads": "4", + "max_pending_requests": "100", + "response_headers": { + "Content-Type": "application/json" + }, + "type": "webhook", + "tags": [ + "http_api" + ] + }, + "comments": [] + }, + { + "id": "input_kafka_2", + "type": "input", + "plugin": "kafka", + "config": { + "bootstrap_servers": "kafka1:9092,kafka2:9092,kafka3:9092", + "topics": [ + "app-logs", + "security-events", + "metrics" + ], + "group_id": "logstash-consumer", + "consumer_threads": "3", + "codec": { + "avro": { + "schema_uri": "http://schema-registry:8081/schemas/ids/1" + } + }, + "decorate_events": "true", + "security_protocol": "SASL_SSL", + "sasl_mechanism": "SCRAM-SHA-512", + "sasl_jaas_config": "org.apache.kafka.common.security.scram.ScramLoginModule required username='logstash' password='${KAFKA_PASS}';", + "type": "kafka", + "tags": [ + "kafka_stream" + ] + }, + "comments": [] + }, + { + "id": "input_jdbc_3", + "type": "input", + "plugin": "jdbc", + "config": { + "jdbc_driver_library": "/usr/share/logstash/vendor/jar/jdbc/postgresql.jar", + "jdbc_driver_class": "org.postgresql.Driver", + "jdbc_connection_string": "jdbc:postgresql://db:5432/prod", + "jdbc_user": "${DB_USER}", + "jdbc_password": "${DB_PASS}", + "schedule": "*/5 * * * *", + "statement": "SELECT * FROM events WHERE created_at > :sql_last_value", + "use_column_value": "true", + "tracking_column": "created_at", + "tracking_column_type": "timestamp", + "type": "database", + "tags": [ + "jdbc_poll" + ] + }, + "comments": [] + }, + { + "id": "input_file_4", + "type": "input", + "plugin": "file", + "config": { + "path": [ + "/var/log/nginx/*.log", + "/var/log/app/**/*.log" + ], + "start_position": "beginning", + "sincedb_path": "/var/lib/logstash/sincedb", + "codec": { + "multiline": { + "pattern": "^%{TIMESTAMP_ISO8601}", + "negate": "true", + "what": "previous", + "max_lines": 500 + } + }, + "type": "file", + "tags": [ + "file_input" + ] + }, + "comments": [] + }, + { + "id": "input_tcp_5", + "type": "input", + "plugin": "tcp", + "config": { + "port": "5000", + "codec": { + "json_lines": {} + }, + "ssl_enable": "true", + "ssl_cert": "/etc/logstash/certs/tcp.crt", + "ssl_key": "/etc/logstash/certs/tcp.key", + "type": "tcp_json", + "tags": [ + "tcp_secure" + ] + }, + "comments": [] + }, + { + "id": "input_udp_6", + "type": "input", + "plugin": "udp", + "config": { + "port": "514", + "codec": { + "cef": {} + }, + "type": "syslog", + "tags": [ + "syslog_udp" + ] + }, + "comments": [] + }, + { + "id": "input_rabbitmq_7", + "type": "input", + "plugin": "rabbitmq", + "config": { + "host": "rabbitmq", + "port": "5672", + "user": "${RABBIT_USER}", + "password": "${RABBIT_PASS}", + "queue": "logs", + "exchange": "logs-exchange", + "exchange_type": "topic", + "key": "logs.#", + "durable": "true", + "codec": { + "json": {} + }, + "type": "rabbitmq", + "tags": [ + "amqp" + ] + }, + "comments": [] + }, + { + "id": "input_redis_8", + "type": "input", + "plugin": "redis", + "config": { + "host": "redis", + "port": "6379", + "password": "${REDIS_PASS}", + "data_type": "list", + "key": "logstash:queue", + "codec": { + "json": {} + }, + "type": "redis", + "tags": [ + "redis_queue" + ] + }, + "comments": [] + }, + { + "id": "input_s3_9", + "type": "input", + "plugin": "s3", + "config": { + "bucket": "logs-archive", + "region": "us-east-1", + "access_key_id": "${AWS_KEY}", + "secret_access_key": "${AWS_SECRET}", + "interval": "300", + "codec": { + "json_lines": {} + }, + "type": "s3", + "tags": [ + "s3_archive" + ] + }, + "comments": [] + }, + { + "id": "input_kinesis_10", + "type": "input", + "plugin": "kinesis", + "config": { + "kinesis_stream_name": "app-stream", + "region": "us-west-2", + "codec": { + "json": {} + }, + "type": "kinesis", + "tags": [ + "aws_kinesis" + ] + }, + "comments": [] + } + ], + "filter": [ + { + "id": "filter_if_11", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[type] == \"beats\"", + "plugins": [ + { + "id": "filter_if_12", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[agent][type] == \"filebeat\"", + "plugins": [ + { + "id": "filter_if_13", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[log][file][path] =~ /nginx/", + "plugins": [ + { + "id": "filter_grok_14", + "type": "filter", + "plugin": "grok", + "config": { + "match": { + "message": "%{IPORHOST:client_ip} - %{DATA:user} \\[%{HTTPDATE:timestamp}\\] \"%{WORD:method} %{DATA:path} HTTP/%{NUMBER:version}\" %{NUMBER:status:int} %{NUMBER:bytes:int} \"%{DATA:referrer}\" \"%{DATA:agent}\"" + } + }, + "comments": [] + }, + { + "id": "filter_date_15", + "type": "filter", + "plugin": "date", + "config": { + "match": [ + "timestamp", + "dd/MMM/yyyy:HH:mm:ss Z" + ], + "target": "@timestamp" + }, + "comments": [] + }, + { + "id": "filter_useragent_16", + "type": "filter", + "plugin": "useragent", + "config": { + "source": "agent", + "target": "ua" + }, + "comments": [] + }, + { + "id": "filter_geoip_17", + "type": "filter", + "plugin": "geoip", + "config": { + "source": "client_ip", + "target": "geo", + "database": "/usr/share/GeoIP/GeoLite2-City.mmdb" + }, + "comments": [] + }, + { + "id": "filter_if_18", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[status] >= 500", + "plugins": [ + { + "id": "filter_mutate_19", + "type": "filter", + "plugin": "mutate", + "config": { + "add_tag": [ + "error", + "server_error" + ], + "add_field": { + "severity": "critical" + } + }, + "comments": [] + } + ], + "else_ifs": [ + { + "condition": "[status] >= 400", + "plugins": [ + { + "id": "filter_mutate_20", + "type": "filter", + "plugin": "mutate", + "config": { + "add_tag": [ + "error", + "client_error" + ], + "add_field": { + "severity": "warning" + } + }, + "comments": [] + } + ] + } + ], + "else": null + } + }, + { + "id": "filter_ruby_21", + "type": "filter", + "plugin": "ruby", + "config": { + "code": "\n bytes = event.get(\"bytes\").to_i\n if bytes > 10485760\n event.set(\"size_class\", \"large\")\n elsif bytes > 1048576\n event.set(\"size_class\", \"medium\")\n else\n event.set(\"size_class\", \"small\")\n end\n " + }, + "comments": [] + }, + { + "id": "filter_fingerprint_22", + "type": "filter", + "plugin": "fingerprint", + "config": { + "source": [ + "client_ip", + "path", + "timestamp" + ], + "target": "[@metadata][fingerprint]", + "method": "SHA256" + }, + "comments": [] + } + ], + "else_ifs": [ + { + "condition": "[log][file][path] =~ /application/", + "plugins": [ + { + "id": "filter_json_23", + "type": "filter", + "plugin": "json", + "config": { + "source": "message", + "target": "app" + }, + "comments": [] + }, + { + "id": "filter_if_24", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[app][level]", + "plugins": [ + { + "id": "filter_translate_25", + "type": "filter", + "plugin": "translate", + "config": { + "field": "[app][level]", + "destination": "severity_num", + "dictionary": { + "DEBUG": "1", + "INFO": "2", + "WARN": "3", + "ERROR": "4", + "FATAL": "5" + }, + "fallback": "2" + }, + "comments": [] + }, + { + "id": "filter_mutate_26", + "type": "filter", + "plugin": "mutate", + "config": { + "convert": { + "severity_num": "integer" + } + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_if_27", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[app][exception]", + "plugins": [ + { + "id": "filter_mutate_28", + "type": "filter", + "plugin": "mutate", + "config": { + "add_tag": [ + "exception" + ] + }, + "comments": [] + }, + { + "id": "filter_ruby_29", + "type": "filter", + "plugin": "ruby", + "config": { + "code": "\n exc = event.get(\"[app][exception]\")\n if exc.is_a?(Hash)\n event.set(\"exception_class\", exc[\"class\"])\n event.set(\"exception_msg\", exc[\"message\"])\n end\n " + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + } + ] + } + ], + "else": null + } + } + ], + "else_ifs": [ + { + "condition": "[agent][type] == \"metricbeat\"", + "plugins": [ + { + "id": "filter_if_30", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[system][cpu]", + "plugins": [ + { + "id": "filter_ruby_31", + "type": "filter", + "plugin": "ruby", + "config": { + "code": "\n cpu = event.get(\"[system][cpu]\")\n if cpu && cpu[\"cores\"]\n total = 0.0\n cpu[\"cores\"].each { |c| total += c[\"user\"][\"pct\"].to_f if c[\"user\"] }\n avg = total / cpu[\"cores\"].length\n event.set(\"[system][cpu][avg_pct]\", avg.round(2))\n event.tag(\"cpu_warning\") if avg > 75\n event.tag(\"cpu_critical\") if avg > 90\n end\n " + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_if_32", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[system][memory][actual][used][pct]", + "plugins": [ + { + "id": "filter_ruby_33", + "type": "filter", + "plugin": "ruby", + "config": { + "code": "\n pct = event.get(\"[system][memory][actual][used][pct]\").to_f * 100\n event.set(\"mem_used_pct\", pct.round(2))\n event.tag(\"memory_warning\") if pct > 85\n event.tag(\"memory_critical\") if pct > 95\n " + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + } + ] + } + ], + "else": null + } + } + ], + "else_ifs": [ + { + "condition": "[type] == \"kafka\"", + "plugins": [ + { + "id": "filter_if_34", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[kafka][topic] == \"security-events\"", + "plugins": [ + { + "id": "filter_json_35", + "type": "filter", + "plugin": "json", + "config": { + "source": "message", + "target": "security" + }, + "comments": [] + }, + { + "id": "filter_if_36", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[security][ip]", + "plugins": [ + { + "id": "filter_cidr_37", + "type": "filter", + "plugin": "cidr", + "config": { + "address": [ + "%{[security][ip]}" + ], + "network": [ + "10.0.0.0/8", + "172.16.0.0/12", + "192.168.0.0/16" + ], + "add_tag": [ + "internal_ip" + ] + }, + "comments": [] + }, + { + "id": "filter_if_38", + "type": "filter", + "plugin": "if", + "config": { + "condition": "\"internal_ip\" not in [tags]", + "plugins": [ + { + "id": "filter_geoip_39", + "type": "filter", + "plugin": "geoip", + "config": { + "source": "[security][ip]", + "target": "threat_geo", + "database": "/usr/share/GeoIP/GeoLite2-City.mmdb" + }, + "comments": [] + }, + { + "id": "filter_geoip_40", + "type": "filter", + "plugin": "geoip", + "config": { + "source": "[security][ip]", + "target": "threat_asn", + "database": "/usr/share/GeoIP/GeoLite2-ASN.mmdb", + "default_database_type": "ASN" + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_if_41", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[security][event_type]", + "plugins": [ + { + "id": "filter_translate_42", + "type": "filter", + "plugin": "translate", + "config": { + "field": "[security][event_type]", + "destination": "threat_score", + "dictionary": { + "brute_force": "75", + "sql_injection": "90", + "xss": "85", + "unauthorized": "80", + "privilege_escalation": "95", + "malware": "100" + }, + "fallback": "50" + }, + "comments": [] + }, + { + "id": "filter_mutate_43", + "type": "filter", + "plugin": "mutate", + "config": { + "convert": { + "threat_score": "integer" + } + }, + "comments": [] + }, + { + "id": "filter_if_44", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[threat_score] >= 90", + "plugins": [ + { + "id": "filter_mutate_45", + "type": "filter", + "plugin": "mutate", + "config": { + "add_tag": [ + "critical_threat" + ] + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_ruby_46", + "type": "filter", + "plugin": "ruby", + "config": { + "code": "\n score = event.get(\"threat_score\").to_i\n is_ext = event.get(\"tags\").include?(\"internal_ip\") ? 0 : 20\n composite = score + is_ext\n event.set(\"composite_risk\", [composite, 100].min)\n\t\t\t\t\t\t\n if composite >= 100\n event.set(\"risk\", \"critical\")\n elsif composite >= 80\n event.set(\"risk\", \"high\")\n elsif composite >= 60\n event.set(\"risk\", \"medium\")\n else\n event.set(\"risk\", \"low\")\n end\n " + }, + "comments": [] + } + ], + "else_ifs": [ + { + "condition": "[kafka][topic] == \"metrics\"", + "plugins": [ + { + "id": "filter_dissect_47", + "type": "filter", + "plugin": "dissect", + "config": { + "mapping": { + "metric": "%{env}.%{dc}.%{host}.%{service}.%{type}.%{name}" + } + }, + "comments": [] + }, + { + "id": "filter_if_48", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[type] == \"response_time\"", + "plugins": [ + { + "id": "filter_ruby_49", + "type": "filter", + "plugin": "ruby", + "config": { + "code": "\n val = event.get(\"value\").to_f\n if val > 5000\n event.set(\"perf_status\", \"critical\")\n elsif val > 2000\n event.set(\"perf_status\", \"slow\")\n else\n event.set(\"perf_status\", \"normal\")\n end\n " + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_aggregate_50", + "type": "filter", + "plugin": "aggregate", + "config": { + "task_id": "%{service}_%{name}", + "code": "\n map[\"count\"] ||= 0\n map[\"sum\"] ||= 0.0\n map[\"min\"] ||= Float::INFINITY\n map[\"max\"] ||= -Float::INFINITY\n\t\t\t\t\t\t\n val = event.get(\"value\").to_f\n map[\"count\"] += 1\n map[\"sum\"] += val\n map[\"min\"] = [map[\"min\"], val].min\n map[\"max\"] = [map[\"max\"], val].max\n\t\t\t\t\t\t\n avg = map[\"sum\"] / map[\"count\"]\n event.set(\"rolling_avg\", avg.round(2))\n event.set(\"rolling_min\", map[\"min\"])\n event.set(\"rolling_max\", map[\"max\"])\n ", + "timeout": "300" + }, + "comments": [] + } + ] + } + ], + "else": null + } + } + ] + }, + { + "condition": "[type] == \"database\"", + "plugins": [ + { + "id": "filter_if_51", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[event_data]", + "plugins": [ + { + "id": "filter_json_52", + "type": "filter", + "plugin": "json", + "config": { + "source": "event_data", + "target": "evt" + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_date_53", + "type": "filter", + "plugin": "date", + "config": { + "match": [ + "created_at", + "ISO8601", + "yyyy-MM-dd HH:mm:ss" + ], + "target": "@timestamp" + }, + "comments": [] + }, + { + "id": "filter_elasticsearch_54", + "type": "filter", + "plugin": "elasticsearch", + "config": { + "hosts": [ + "http://elasticsearch:9200" + ], + "index": "user-profiles", + "query_template": "user_lookup.json", + "fields": { + "department": "user_dept", + "role": "user_role" + } + }, + "comments": [] + }, + { + "id": "filter_if_55", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[event_type] =~ /^(login|logout|password_change)$/", + "plugins": [ + { + "id": "filter_mutate_56", + "type": "filter", + "plugin": "mutate", + "config": { + "add_tag": [ + "auth_event" + ] + }, + "comments": [] + } + ], + "else_ifs": [ + { + "condition": "[event_type] =~ /^(create|update|delete)$/", + "plugins": [ + { + "id": "filter_mutate_57", + "type": "filter", + "plugin": "mutate", + "config": { + "add_tag": [ + "data_operation" + ] + }, + "comments": [] + } + ] + } + ], + "else": null + } + } + ] + }, + { + "condition": "[type] == \"webhook\"", + "plugins": [ + { + "id": "filter_if_58", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[headers][user-agent]", + "plugins": [ + { + "id": "filter_useragent_59", + "type": "filter", + "plugin": "useragent", + "config": { + "source": "[headers][user-agent]", + "target": "webhook_ua" + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_if_60", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[headers][x-signature]", + "plugins": [ + { + "id": "filter_ruby_61", + "type": "filter", + "plugin": "ruby", + "config": { + "init": "require \"openssl\"", + "code": "\n sig = event.get(\"[headers][x-signature]\")\n payload = event.get(\"message\").to_s\n secret = ENV[\"WEBHOOK_SECRET\"]\n expected = \"sha256=\" + OpenSSL::HMAC.hexdigest(\"SHA256\", secret, payload)\n\t\t\t\t\t\t\n if sig == expected\n event.set(\"sig_valid\", true)\n else\n event.set(\"sig_valid\", false)\n event.tag(\"invalid_signature\")\n end\n " + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_aggregate_62", + "type": "filter", + "plugin": "aggregate", + "config": { + "task_id": "%{[headers][x-forwarded-for]}", + "code": "\n map[\"count\"] ||= 0\n map[\"count\"] += 1\n event.set(\"request_count\", map[\"count\"])\n ", + "timeout": "60" + }, + "comments": [] + }, + { + "id": "filter_if_63", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[request_count] and [request_count] > 100", + "plugins": [ + { + "id": "filter_mutate_64", + "type": "filter", + "plugin": "mutate", + "config": { + "add_tag": [ + "rate_limit_exceeded" + ] + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + } + ] + } + ], + "else": null + } + }, + { + "id": "filter_if_65", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[message] =~ /=/", + "plugins": [ + { + "id": "filter_kv_66", + "type": "filter", + "plugin": "kv", + "config": { + "source": "message", + "field_split": "&", + "value_split": "=", + "target": "parsed" + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_if_67", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[user_agent] and ![ua]", + "plugins": [ + { + "id": "filter_useragent_68", + "type": "filter", + "plugin": "useragent", + "config": { + "source": "user_agent", + "target": "ua" + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_if_69", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[source_ip]", + "plugins": [ + { + "id": "filter_dns_70", + "type": "filter", + "plugin": "dns", + "config": { + "reverse": [ + "source_ip" + ], + "action": "append", + "nameserver": [ + "8.8.8.8" + ], + "hit_cache_size": "10000", + "hit_cache_ttl": "3600" + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_prune_71", + "type": "filter", + "plugin": "prune", + "config": { + "whitelist_names": [ + "^@", + "^_", + "type", + "tags", + "message" + ] + }, + "comments": [] + }, + { + "id": "filter_mutate_72", + "type": "filter", + "plugin": "mutate", + "config": { + "add_field": { + "env": "${ENVIRONMENT:prod}", + "cluster": "${CLUSTER:default}" + }, + "remove_field": [ + "@version" + ] + }, + "comments": [] + }, + { + "id": "filter_if_73", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[env] == \"production\" and ([tags] and \"debug\" in [tags])", + "plugins": [ + { + "id": "filter_drop_74", + "type": "filter", + "plugin": "drop", + "config": {}, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_throttle_75", + "type": "filter", + "plugin": "throttle", + "config": { + "before_count": "3", + "after_count": "1", + "period": "60", + "key": "%{fingerprint}", + "add_tag": [ + "throttled" + ] + }, + "comments": [] + }, + { + "id": "filter_if_76", + "type": "filter", + "plugin": "if", + "config": { + "condition": "\"critical_threat\" in [tags]", + "plugins": [ + { + "id": "filter_clone_77", + "type": "filter", + "plugin": "clone", + "config": { + "clones": [ + "siem" + ], + "add_field": { + "cloned": "true" + } + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_metrics_78", + "type": "filter", + "plugin": "metrics", + "config": { + "meter": [ + "events" + ], + "add_tag": [ + "metric" + ], + "flush_interval": "30", + "rates": [ + 1, + 5, + 15 + ] + }, + "comments": [] + }, + { + "id": "filter_if_79", + "type": "filter", + "plugin": "if", + "config": { + "condition": "([severity] == \"critical\" or [severity] == \"error\") and ([status] >= 500 or [threat_score] >= 90)", + "plugins": [ + { + "id": "filter_mutate_80", + "type": "filter", + "plugin": "mutate", + "config": { + "add_field": { + "priority": "P1", + "oncall": "true" + }, + "add_tag": [ + "p1" + ] + }, + "comments": [] + } + ], + "else_ifs": [ + { + "condition": "([severity] == \"warning\") and ([status] >= 400 or [threat_score] >= 70)", + "plugins": [ + { + "id": "filter_mutate_81", + "type": "filter", + "plugin": "mutate", + "config": { + "add_field": { + "priority": "P2" + }, + "add_tag": [ + "p2" + ] + }, + "comments": [] + } + ] + } + ], + "else": null + } + }, + { + "id": "filter_fingerprint_82", + "type": "filter", + "plugin": "fingerprint", + "config": { + "source": "message", + "target": "event_hash", + "method": "MURMUR3" + }, + "comments": [] + }, + { + "id": "filter_uuid_83", + "type": "filter", + "plugin": "uuid", + "config": { + "target": "event_id" + }, + "comments": [] + }, + { + "id": "filter_ruby_84", + "type": "filter", + "plugin": "ruby", + "config": { + "code": "\n event.set(\"processed_at\", Time.now.utc.iso8601)\n event.set(\"pipeline_v\", \"2.0\")\n " + }, + "comments": [] + } + ], + "output": [ + { + "id": "output_if_85", + "type": "output", + "plugin": "if", + "config": { + "condition": "\"throttled\" not in [tags] and \"metric\" not in [tags]", + "plugins": [ + { + "id": "output_elasticsearch_86", + "type": "output", + "plugin": "elasticsearch", + "config": { + "hosts": [ + "https://es1:9200", + "https://es2:9200" + ], + "user": "${ES_USER}", + "password": "${ES_PASS}", + "ssl": "true", + "cacert": "/etc/logstash/certs/ca.crt", + "index": "%{type}-%{+YYYY.MM.dd}", + "document_id": "%{event_id}", + "pipeline": "enrich", + "ilm_enabled": "true", + "ilm_rollover_alias": "%{type}", + "ilm_pattern": "{now/d}-000001", + "ilm_policy": "logs-policy", + "http_compression": "true" + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "output_if_87", + "type": "output", + "plugin": "if", + "config": { + "condition": "\"siem\" in [tags]", + "plugins": [ + { + "id": "output_elasticsearch_88", + "type": "output", + "plugin": "elasticsearch", + "config": { + "hosts": [ + "https://siem-es:9200" + ], + "user": "${SIEM_USER}", + "password": "${SIEM_PASS}", + "ssl": "true", + "cacert": "/etc/logstash/certs/siem-ca.crt", + "index": "security-%{+YYYY.MM.dd}" + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "output_if_89", + "type": "output", + "plugin": "if", + "config": { + "condition": "[env] == \"production\"", + "plugins": [ + { + "id": "output_s3_90", + "type": "output", + "plugin": "s3", + "config": { + "access_key_id": "${AWS_KEY}", + "secret_access_key": "${AWS_SECRET}", + "region": "us-east-1", + "bucket": "logs-archive", + "size_file": "104857600", + "time_file": "15", + "codec": { + "json_lines": {} + }, + "prefix": "logs/%{type}/year=%{+YYYY}/month=%{+MM}/day=%{+dd}", + "encoding": "gzip", + "server_side_encryption": "true" + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "output_kafka_91", + "type": "output", + "plugin": "kafka", + "config": { + "bootstrap_servers": "kafka1:9092,kafka2:9092", + "topic_id": "processed-%{type}", + "codec": { + "json": {} + }, + "compression_type": "snappy", + "acks": "all", + "security_protocol": "SASL_SSL", + "sasl_mechanism": "SCRAM-SHA-512", + "sasl_jaas_config": "org.apache.kafka.common.security.scram.ScramLoginModule required username='${KAFKA_USER}' password='${KAFKA_PASS}';" + }, + "comments": [] + }, + { + "id": "output_if_92", + "type": "output", + "plugin": "if", + "config": { + "condition": "\"p1\" in [tags] or \"critical\" in [tags]", + "plugins": [ + { + "id": "output_redis_93", + "type": "output", + "plugin": "redis", + "config": { + "host": [ + "redis1", + "redis2" + ], + "port": "26379", + "password": "${REDIS_PASS}", + "data_type": "list", + "key": "alerts:critical" + }, + "comments": [] + }, + { + "id": "output_http_94", + "type": "output", + "plugin": "http", + "config": { + "url": "https://alerts.example.com/api/events", + "http_method": "post", + "format": "json", + "headers": { + "Authorization": "Bearer ${ALERT_TOKEN}", + "Content-Type": "application/json" + }, + "automatic_retries": "3" + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "output_if_95", + "type": "output", + "plugin": "if", + "config": { + "condition": "[env] != \"production\"", + "plugins": [ + { + "id": "output_file_96", + "type": "output", + "plugin": "file", + "config": { + "path": "/var/log/logstash/debug-%{type}.log", + "codec": { + "json_lines": {} + } + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "output_if_97", + "type": "output", + "plugin": "if", + "config": { + "condition": "\"metric\" in [tags]", + "plugins": [ + { + "id": "output_graphite_98", + "type": "output", + "plugin": "graphite", + "config": { + "host": "graphite", + "port": "2003", + "metrics_format": "logstash.%{env}.%{type}.count", + "fields_are_metrics": "true" + }, + "comments": [] + }, + { + "id": "output_influxdb_99", + "type": "output", + "plugin": "influxdb", + "config": { + "host": "influxdb", + "port": "8086", + "db": "metrics", + "user": "${INFLUX_USER}", + "password": "${INFLUX_PASS}", + "measurement": "%{type}_metrics", + "use_event_fields_for_data_points": "true" + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "output_if_100", + "type": "output", + "plugin": "if", + "config": { + "condition": "[type] == \"rabbitmq\"", + "plugins": [ + { + "id": "output_mongodb_101", + "type": "output", + "plugin": "mongodb", + "config": { + "uri": "mongodb://${MONGO_USER}:${MONGO_PASS}@mongo:27017/logs", + "database": "logs", + "collection": "%{type}", + "isodate": "true", + "bulk": "true", + "bulk_size": "100" + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "output_if_102", + "type": "output", + "plugin": "if", + "config": { + "condition": "\"p1\" in [tags]", + "plugins": [ + { + "id": "output_email_103", + "type": "output", + "plugin": "email", + "config": { + "to": "oncall@example.com", + "from": "alerts@example.com", + "subject": "P1 Alert: %{type}", + "body": "Event: %{event_id}\nTime: %{@timestamp}\nSeverity: %{severity}\nMessage: %{message}", + "address": "smtp.example.com", + "port": "587", + "use_tls": "true", + "username": "${SMTP_USER}", + "password": "${SMTP_PASS}" + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "output_tcp_104", + "type": "output", + "plugin": "tcp", + "config": { + "host": "logstash-secondary", + "port": "5005", + "codec": { + "json_lines": {} + } + }, + "comments": [] + }, + { + "id": "output_stdout_105", + "type": "output", + "plugin": "stdout", + "config": { + "codec": { + "rubydebug": {} + } + }, + "comments": [] + } + ] +} diff --git a/tests/Common/unit/conversion_data/components/test-data-types.json b/tests/Common/unit/conversion_data/components/test-data-types.json new file mode 100644 index 0000000..f83671f --- /dev/null +++ b/tests/Common/unit/conversion_data/components/test-data-types.json @@ -0,0 +1,52 @@ +{ + "input": [], + "filter": [ + { + "id": "filter_grok_0", + "type": "filter", + "plugin": "grok", + "config": { + "match": { + "test": "test" + }, + "pattern_definitions": { + "1": "2", + "test": "test", + "asd": "asf" + }, + "patterns_dir": [ + "test" + ], + "tag_on_failure": [] + }, + "comments": [] + }, + { + "id": "filter_grok_1", + "type": "filter", + "plugin": "grok", + "config": { + "match": { + "test": [ + "test", + "test2" + ] + }, + "pattern_definitions": { + "test": "test" + }, + "patterns_dir": "test" + }, + "comments": [] + }, + { + "id": "filter_comment_2", + "type": "filter", + "plugin": "comment", + "config": { + "text": "test\nmulti\nrow" + } + } + ], + "output": [] +} diff --git a/tests/Common/unit/conversion_data/components/test-datatypes.json b/tests/Common/unit/conversion_data/components/test-datatypes.json new file mode 100644 index 0000000..f83671f --- /dev/null +++ b/tests/Common/unit/conversion_data/components/test-datatypes.json @@ -0,0 +1,52 @@ +{ + "input": [], + "filter": [ + { + "id": "filter_grok_0", + "type": "filter", + "plugin": "grok", + "config": { + "match": { + "test": "test" + }, + "pattern_definitions": { + "1": "2", + "test": "test", + "asd": "asf" + }, + "patterns_dir": [ + "test" + ], + "tag_on_failure": [] + }, + "comments": [] + }, + { + "id": "filter_grok_1", + "type": "filter", + "plugin": "grok", + "config": { + "match": { + "test": [ + "test", + "test2" + ] + }, + "pattern_definitions": { + "test": "test" + }, + "patterns_dir": "test" + }, + "comments": [] + }, + { + "id": "filter_comment_2", + "type": "filter", + "plugin": "comment", + "config": { + "text": "test\nmulti\nrow" + } + } + ], + "output": [] +} diff --git a/tests/Common/unit/conversion_data/components/test-devopsschool-1.json b/tests/Common/unit/conversion_data/components/test-devopsschool-1.json new file mode 100644 index 0000000..b879d77 --- /dev/null +++ b/tests/Common/unit/conversion_data/components/test-devopsschool-1.json @@ -0,0 +1,51 @@ +{ + "input": [ + { + "id": "input_beats_0", + "type": "input", + "plugin": "beats", + "config": { + "port": "5044" + }, + "comments": [] + } + ], + "filter": [ + { + "id": "filter_grok_1", + "type": "filter", + "plugin": "grok", + "config": { + "match": { + "message": "%{COMBINEDAPACHELOG}" + } + }, + "comments": [] + } + ], + "output": [ + { + "id": "output_elasticsearch_2", + "type": "output", + "plugin": "elasticsearch", + "config": { + "hosts": [ + "http://elasticsearch:9200" + ], + "index": "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" + }, + "comments": [] + }, + { + "id": "output_stdout_3", + "type": "output", + "plugin": "stdout", + "config": { + "codec": { + "rubydebug": {} + } + }, + "comments": [] + } + ] +} diff --git a/tests/Common/unit/conversion_data/components/test-devopsschool-2.json b/tests/Common/unit/conversion_data/components/test-devopsschool-2.json new file mode 100644 index 0000000..9a82ca7 --- /dev/null +++ b/tests/Common/unit/conversion_data/components/test-devopsschool-2.json @@ -0,0 +1,39 @@ +{ + "input": [ + { + "id": "input_beats_0", + "type": "input", + "plugin": "beats", + "config": { + "port": "5044" + }, + "comments": [] + } + ], + "filter": [ + { + "id": "filter_grok_1", + "type": "filter", + "plugin": "grok", + "config": { + "match": { + "message": "%{SYSLOGLINE}" + } + }, + "comments": [] + } + ], + "output": [ + { + "id": "output_stdout_2", + "type": "output", + "plugin": "stdout", + "config": { + "codec": { + "rubydebug": {} + } + }, + "comments": [] + } + ] +} diff --git a/tests/Common/unit/conversion_data/components/test-devopsschool-4.json b/tests/Common/unit/conversion_data/components/test-devopsschool-4.json new file mode 100644 index 0000000..6ad22f2 --- /dev/null +++ b/tests/Common/unit/conversion_data/components/test-devopsschool-4.json @@ -0,0 +1,62 @@ +{ + "input": [ + { + "id": "input_file_0", + "type": "input", + "plugin": "file", + "config": { + "path": "/var/log/apache2/access.log", + "start_position": "beginning", + "sincedb_path": "/dev/null" + }, + "comments": [] + } + ], + "filter": [ + { + "id": "filter_grok_1", + "type": "filter", + "plugin": "grok", + "config": { + "match": { + "message": "%{COMBINEDAPACHELOG}" + } + }, + "comments": [] + }, + { + "id": "filter_date_2", + "type": "filter", + "plugin": "date", + "config": { + "match": [ + "timestamp", + "dd/MMM/yyyy:HH:mm:ss Z" + ] + }, + "comments": [] + }, + { + "id": "filter_geoip_3", + "type": "filter", + "plugin": "geoip", + "config": { + "source": "clientip" + }, + "comments": [] + } + ], + "output": [ + { + "id": "output_elasticsearch_4", + "type": "output", + "plugin": "elasticsearch", + "config": { + "hosts": [ + "localhost:9200" + ] + }, + "comments": [] + } + ] +} diff --git a/tests/Common/unit/conversion_data/components/test-devopsschool-5.json b/tests/Common/unit/conversion_data/components/test-devopsschool-5.json new file mode 100644 index 0000000..e16f5da --- /dev/null +++ b/tests/Common/unit/conversion_data/components/test-devopsschool-5.json @@ -0,0 +1,48 @@ +{ + "input": [ + { + "id": "input_beats_0", + "type": "input", + "plugin": "beats", + "config": { + "port": "5044" + }, + "comments": [] + } + ], + "filter": [ + { + "id": "filter_grok_1", + "type": "filter", + "plugin": "grok", + "config": { + "match": { + "message": "%{COMBINEDAPACHELOG}" + } + }, + "comments": [] + }, + { + "id": "filter_geoip_2", + "type": "filter", + "plugin": "geoip", + "config": { + "source": "clientip" + }, + "comments": [] + } + ], + "output": [ + { + "id": "output_elasticsearch_3", + "type": "output", + "plugin": "elasticsearch", + "config": { + "hosts": [ + "localhost:9200" + ] + }, + "comments": [] + } + ] +} diff --git a/tests/Common/unit/conversion_data/components/test-elasticdocs-apache.json b/tests/Common/unit/conversion_data/components/test-elasticdocs-apache.json new file mode 100644 index 0000000..d373e91 --- /dev/null +++ b/tests/Common/unit/conversion_data/components/test-elasticdocs-apache.json @@ -0,0 +1,86 @@ +{ + "input": [ + { + "id": "input_file_0", + "type": "input", + "plugin": "file", + "config": { + "path": "/tmp/access_log", + "start_position": "beginning" + }, + "comments": [] + } + ], + "filter": [ + { + "id": "filter_if_1", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[path] =~ \"access\"", + "plugins": [ + { + "id": "filter_mutate_2", + "type": "filter", + "plugin": "mutate", + "config": { + "replace": { + "type": "apache_access" + } + }, + "comments": [] + }, + { + "id": "filter_grok_3", + "type": "filter", + "plugin": "grok", + "config": { + "match": { + "message": "%{COMBINEDAPACHELOG}" + } + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_date_4", + "type": "filter", + "plugin": "date", + "config": { + "match": [ + "timestamp", + "dd/MMM/yyyy:HH:mm:ss Z" + ] + }, + "comments": [] + } + ], + "output": [ + { + "id": "output_elasticsearch_5", + "type": "output", + "plugin": "elasticsearch", + "config": { + "hosts": [ + "localhost:9200" + ] + }, + "comments": [] + }, + { + "id": "output_stdout_6", + "type": "output", + "plugin": "stdout", + "config": { + "codec": { + "rubydebug": {} + } + }, + "comments": [] + } + ] +} diff --git a/tests/Common/unit/conversion_data/components/test-elasticdocs-configuring_filters.json b/tests/Common/unit/conversion_data/components/test-elasticdocs-configuring_filters.json new file mode 100644 index 0000000..ce29631 --- /dev/null +++ b/tests/Common/unit/conversion_data/components/test-elasticdocs-configuring_filters.json @@ -0,0 +1,60 @@ +{ + "input": [ + { + "id": "input_stdin_0", + "type": "input", + "plugin": "stdin", + "config": {}, + "comments": [] + } + ], + "filter": [ + { + "id": "filter_grok_1", + "type": "filter", + "plugin": "grok", + "config": { + "match": { + "message": "%{COMBINEDAPACHELOG}" + } + }, + "comments": [] + }, + { + "id": "filter_date_2", + "type": "filter", + "plugin": "date", + "config": { + "match": [ + "timestamp", + "dd/MMM/yyyy:HH:mm:ss Z" + ] + }, + "comments": [] + } + ], + "output": [ + { + "id": "output_elasticsearch_3", + "type": "output", + "plugin": "elasticsearch", + "config": { + "hosts": [ + "localhost:9200" + ] + }, + "comments": [] + }, + { + "id": "output_stdout_4", + "type": "output", + "plugin": "stdout", + "config": { + "codec": { + "rubydebug": {} + } + }, + "comments": [] + } + ] +} diff --git a/tests/Common/unit/conversion_data/components/test-elasticdocs-syslog.json b/tests/Common/unit/conversion_data/components/test-elasticdocs-syslog.json new file mode 100644 index 0000000..bfa06fe --- /dev/null +++ b/tests/Common/unit/conversion_data/components/test-elasticdocs-syslog.json @@ -0,0 +1,92 @@ +{ + "input": [ + { + "id": "input_tcp_0", + "type": "input", + "plugin": "tcp", + "config": { + "port": "5000", + "type": "syslog" + }, + "comments": [] + }, + { + "id": "input_udp_1", + "type": "input", + "plugin": "udp", + "config": { + "port": "5000", + "type": "syslog" + }, + "comments": [] + } + ], + "filter": [ + { + "id": "filter_if_2", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[type] == \"syslog\"", + "plugins": [ + { + "id": "filter_grok_3", + "type": "filter", + "plugin": "grok", + "config": { + "match": { + "message": "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\\[%{POSINT:syslog_pid}\\])?: %{GREEDYDATA:syslog_message}" + }, + "add_field": [ + "received_at", + "%{@timestamp}", + "received_from", + "%{host}" + ] + }, + "comments": [] + }, + { + "id": "filter_date_4", + "type": "filter", + "plugin": "date", + "config": { + "match": [ + "syslog_timestamp", + "MMM d HH:mm:ss", + "MMM dd HH:mm:ss" + ] + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + } + ], + "output": [ + { + "id": "output_elasticsearch_5", + "type": "output", + "plugin": "elasticsearch", + "config": { + "hosts": [ + "localhost:9200" + ] + }, + "comments": [] + }, + { + "id": "output_stdout_6", + "type": "output", + "plugin": "stdout", + "config": { + "codec": { + "rubydebug": {} + } + }, + "comments": [] + } + ] +} diff --git a/tests/Common/unit/conversion_data/components/test-es-input.json b/tests/Common/unit/conversion_data/components/test-es-input.json new file mode 100644 index 0000000..c3c26fe --- /dev/null +++ b/tests/Common/unit/conversion_data/components/test-es-input.json @@ -0,0 +1,48 @@ +{ + "input": [ + { + "id": "input_elasticsearch_0", + "type": "input", + "plugin": "elasticsearch", + "config": { + "api_key": "test", + "cloud_id": "test", + "index": "kibana_sample_data_ecommerce", + "query": "{\"query\":{\"match_all\":{}}}", + "slices": "6", + "ssl_enabled": "true", + "connect_timeout_seconds": "120", + "request_timeout_seconds": "600", + "socket_timeout_seconds": "600" + }, + "comments": [] + } + ], + "filter": [ + { + "id": "filter_mutate_1", + "type": "filter", + "plugin": "mutate", + "config": { + "add_field": { + "test": "test" + } + }, + "comments": [] + } + ], + "output": [ + { + "id": "output_csv_2", + "type": "output", + "plugin": "csv", + "config": { + "fields": [ + "test" + ], + "path": "/home/ubuntu/test.csv" + }, + "comments": [] + } + ] +} diff --git a/tests/Common/unit/conversion_data/components/test-ls-repo-mysql.json b/tests/Common/unit/conversion_data/components/test-ls-repo-mysql.json new file mode 100644 index 0000000..2a09593 --- /dev/null +++ b/tests/Common/unit/conversion_data/components/test-ls-repo-mysql.json @@ -0,0 +1,156 @@ +{ + "input": [ + { + "id": "input_beats_0", + "type": "input", + "plugin": "beats", + "config": { + "port": 5044, + "host": "0.0.0.0" + }, + "comments": [] + } + ], + "filter": [ + { + "id": "filter_if_1", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[fileset][module] == \"mysql\"", + "plugins": [ + { + "id": "filter_if_2", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[fileset][name] == \"error\"", + "plugins": [ + { + "id": "filter_grok_3", + "type": "filter", + "plugin": "grok", + "config": { + "match": { + "message": [ + "%{LOCALDATETIME:[mysql][error][timestamp]} (\\[%{DATA:[mysql][error][level]}\\] )?%{GREEDYDATA:[mysql][error][message]}", + "%{TIMESTAMP_ISO8601:[mysql][error][timestamp]} %{NUMBER:[mysql][error][thread_id]} \\[%{DATA:[mysql][error][level]}\\] %{GREEDYDATA:[mysql][error][message1]}", + "%{GREEDYDATA:[mysql][error][message2]}" + ] + }, + "pattern_definitions": { + "LOCALDATETIME": "[0-9]+ %{TIME}" + }, + "remove_field": "message" + }, + "comments": [] + }, + { + "id": "filter_mutate_4", + "type": "filter", + "plugin": "mutate", + "config": { + "rename": { + "[mysql][error][message1]": "[mysql][error][message]" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_5", + "type": "filter", + "plugin": "mutate", + "config": { + "rename": { + "[mysql][error][message2]": "[mysql][error][message]" + } + }, + "comments": [] + }, + { + "id": "filter_date_6", + "type": "filter", + "plugin": "date", + "config": { + "match": [ + "[mysql][error][timestamp]", + "ISO8601", + "YYMMdd H:m:s" + ], + "remove_field": "[mysql][error][time]" + }, + "comments": [] + } + ], + "else_ifs": [ + { + "condition": "[fileset][name] == \"slowlog\"", + "plugins": [ + { + "id": "filter_grok_7", + "type": "filter", + "plugin": "grok", + "config": { + "match": { + "message": [ + "^# User@Host: %{USER:[mysql][slowlog][user]}(\\[[^\\]]+\\])? @ %{HOSTNAME:[mysql][slowlog][host]} \\[(IP:[mysql][slowlog][ip])?\\](\\s*Id:\\s* %{NUMBER:[mysql][slowlog][id]})?\n# Query_time: %{NUMBER:[mysql][slowlog][query_time][sec]}\\s* Lock_time: %{NUMBER:[mysql][slowlog][lock_time][sec]}\\s* Rows_sent: %{NUMBER:[mysql][slowlog][rows_sent]}\\s* Rows_examined: %{NUMBER:[mysql][slowlog][rows_examined]}\n(SET timestamp=%{NUMBER:[mysql][slowlog][timestamp]};\n)?%{GREEDYMULTILINE:[mysql][slowlog][query]}" + ] + }, + "pattern_definitions": { + "GREEDYMULTILINE": "(.|\n)*" + }, + "remove_field": "message" + }, + "comments": [] + }, + { + "id": "filter_date_8", + "type": "filter", + "plugin": "date", + "config": { + "match": [ + "[mysql][slowlog][timestamp]", + "UNIX" + ] + }, + "comments": [] + }, + { + "id": "filter_mutate_9", + "type": "filter", + "plugin": "mutate", + "config": { + "gsub": [ + "[mysql][slowlog][query]", + "\n# Time: [0-9]+ [0-9][0-9]:[0-9][0-9]:[0-9][0-9](\\.[0-9]+)?$", + "" + ] + }, + "comments": [] + } + ] + } + ], + "else": null + } + } + ], + "else_ifs": [], + "else": null + } + } + ], + "output": [ + { + "id": "output_elasticsearch_10", + "type": "output", + "plugin": "elasticsearch", + "config": { + "hosts": "localhost", + "manage_template": "false", + "index": "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" + }, + "comments": [] + } + ] +} diff --git a/tests/Common/unit/conversion_data/components/test-ls-repo-nginx.json b/tests/Common/unit/conversion_data/components/test-ls-repo-nginx.json new file mode 100644 index 0000000..3493de0 --- /dev/null +++ b/tests/Common/unit/conversion_data/components/test-ls-repo-nginx.json @@ -0,0 +1,156 @@ +{ + "input": [ + { + "id": "input_beats_0", + "type": "input", + "plugin": "beats", + "config": { + "port": 5044, + "host": "0.0.0.0" + }, + "comments": [] + } + ], + "filter": [ + { + "id": "filter_if_1", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[fileset][module] == \"nginx\"", + "plugins": [ + { + "id": "filter_if_2", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[fileset][name] == \"access\"", + "plugins": [ + { + "id": "filter_grok_3", + "type": "filter", + "plugin": "grok", + "config": { + "match": { + "message": [ + "%{IPORHOST:[nginx][access][remote_ip]} - %{DATA:[nginx][access][user_name]} \\[%{HTTPDATE:[nginx][access][time]}\\] \"%{WORD:[nginx][access][method]} %{DATA:[nginx][access][url]} HTTP/%{NUMBER:[nginx][access][http_version]}\" %{NUMBER:[nginx][access][response_code]} %{NUMBER:[nginx][access][body_sent][bytes]} \"%{DATA:[nginx][access][referrer]}\" \"%{DATA:[nginx][access][agent]}\"" + ] + }, + "remove_field": "message" + }, + "comments": [] + }, + { + "id": "filter_mutate_4", + "type": "filter", + "plugin": "mutate", + "config": { + "add_field": { + "read_timestamp": "%{@timestamp}" + } + }, + "comments": [] + }, + { + "id": "filter_date_5", + "type": "filter", + "plugin": "date", + "config": { + "match": [ + "[nginx][access][time]", + "dd/MMM/YYYY:H:m:s Z" + ], + "remove_field": "[nginx][access][time]" + }, + "comments": [] + }, + { + "id": "filter_useragent_6", + "type": "filter", + "plugin": "useragent", + "config": { + "source": "[nginx][access][agent]", + "target": "[nginx][access][user_agent]", + "remove_field": "[nginx][access][agent]" + }, + "comments": [] + }, + { + "id": "filter_geoip_7", + "type": "filter", + "plugin": "geoip", + "config": { + "source": "[nginx][access][remote_ip]", + "target": "[nginx][access][geoip]" + }, + "comments": [] + } + ], + "else_ifs": [ + { + "condition": "[fileset][name] == \"error\"", + "plugins": [ + { + "id": "filter_grok_8", + "type": "filter", + "plugin": "grok", + "config": { + "match": { + "message": [ + "%{DATA:[nginx][error][time]} \\[%{DATA:[nginx][error][level]}\\] %{NUMBER:[nginx][error][pid]}#%{NUMBER:[nginx][error][tid]}: (\\*%{NUMBER:[nginx][error][connection_id]} )?%{GREEDYDATA:[nginx][error][message]}" + ] + }, + "remove_field": "message" + }, + "comments": [] + }, + { + "id": "filter_mutate_9", + "type": "filter", + "plugin": "mutate", + "config": { + "rename": { + "@timestamp": "read_timestamp" + } + }, + "comments": [] + }, + { + "id": "filter_date_10", + "type": "filter", + "plugin": "date", + "config": { + "match": [ + "[nginx][error][time]", + "YYYY/MM/dd H:m:s" + ], + "remove_field": "[nginx][error][time]" + }, + "comments": [] + } + ] + } + ], + "else": null + } + } + ], + "else_ifs": [], + "else": null + } + } + ], + "output": [ + { + "id": "output_elasticsearch_11", + "type": "output", + "plugin": "elasticsearch", + "config": { + "hosts": "localhost", + "manage_template": "false", + "index": "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" + }, + "comments": [] + } + ] +} diff --git a/tests/Common/unit/conversion_data/components/test-ls-repo-system.json b/tests/Common/unit/conversion_data/components/test-ls-repo-system.json new file mode 100644 index 0000000..94114af --- /dev/null +++ b/tests/Common/unit/conversion_data/components/test-ls-repo-system.json @@ -0,0 +1,135 @@ +{ + "input": [ + { + "id": "input_beats_0", + "type": "input", + "plugin": "beats", + "config": { + "port": 5044, + "host": "0.0.0.0" + }, + "comments": [] + } + ], + "filter": [ + { + "id": "filter_if_1", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[fileset][module] == \"system\"", + "plugins": [ + { + "id": "filter_if_2", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[fileset][name] == \"auth\"", + "plugins": [ + { + "id": "filter_grok_3", + "type": "filter", + "plugin": "grok", + "config": { + "match": { + "message": [ + "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\\[%{POSINT:[system][auth][pid]}\\])?: %{DATA:[system][auth][ssh][event]} %{DATA:[system][auth][ssh][method]} for (invalid user )?%{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]} port %{NUMBER:[system][auth][ssh][port]} ssh2(: %{GREEDYDATA:[system][auth][ssh][signature]})?", + "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\\[%{POSINT:[system][auth][pid]}\\])?: %{DATA:[system][auth][ssh][event]} user %{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]}", + "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\\[%{POSINT:[system][auth][pid]}\\])?: Did not receive identification string from %{IPORHOST:[system][auth][ssh][dropped_ip]}", + "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sudo(?:\\[%{POSINT:[system][auth][pid]}\\])?: \\s*%{DATA:[system][auth][user]} :( %{DATA:[system][auth][sudo][error]} ;)? TTY=%{DATA:[system][auth][sudo][tty]} ; PWD=%{DATA:[system][auth][sudo][pwd]} ; USER=%{DATA:[system][auth][sudo][user]} ; COMMAND=%{GREEDYDATA:[system][auth][sudo][command]}", + "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} groupadd(?:\\[%{POSINT:[system][auth][pid]}\\])?: new group: name=%{DATA:system.auth.groupadd.name}, GID=%{NUMBER:system.auth.groupadd.gid}", + "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} useradd(?:\\[%{POSINT:[system][auth][pid]}\\])?: new user: name=%{DATA:[system][auth][useradd][name]}, UID=%{NUMBER:[system][auth][useradd][uid]}, GID=%{NUMBER:[system][auth][useradd][gid]}, home=%{DATA:[system][auth][useradd][home]}, shell=%{DATA:[system][auth][useradd][shell]}$", + "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} %{DATA:[system][auth][program]}(?:\\[%{POSINT:[system][auth][pid]}\\])?: %{GREEDYMULTILINE:[system][auth][message]}" + ] + }, + "pattern_definitions": { + "GREEDYMULTILINE": "(.|\n)*" + }, + "remove_field": "message" + }, + "comments": [] + }, + { + "id": "filter_date_4", + "type": "filter", + "plugin": "date", + "config": { + "match": [ + "[system][auth][timestamp]", + "MMM d HH:mm:ss", + "MMM dd HH:mm:ss" + ] + }, + "comments": [] + }, + { + "id": "filter_geoip_5", + "type": "filter", + "plugin": "geoip", + "config": { + "source": "[system][auth][ssh][ip]", + "target": "[system][auth][ssh][geoip]" + }, + "comments": [] + } + ], + "else_ifs": [ + { + "condition": "[fileset][name] == \"syslog\"", + "plugins": [ + { + "id": "filter_grok_6", + "type": "filter", + "plugin": "grok", + "config": { + "match": { + "message": [ + "%{SYSLOGTIMESTAMP:[system][syslog][timestamp]} %{SYSLOGHOST:[system][syslog][hostname]} %{DATA:[system][syslog][program]}(?:\\[%{POSINT:[system][syslog][pid]}\\])?: %{GREEDYMULTILINE:[system][syslog][message]}" + ] + }, + "pattern_definitions": { + "GREEDYMULTILINE": "(.|\n)*" + }, + "remove_field": "message" + }, + "comments": [] + }, + { + "id": "filter_date_7", + "type": "filter", + "plugin": "date", + "config": { + "match": [ + "[system][syslog][timestamp]", + "MMM d HH:mm:ss", + "MMM dd HH:mm:ss" + ] + }, + "comments": [] + } + ] + } + ], + "else": null + } + } + ], + "else_ifs": [], + "else": null + } + } + ], + "output": [ + { + "id": "output_elasticsearch_8", + "type": "output", + "plugin": "elasticsearch", + "config": { + "hosts": "localhost", + "manage_template": "false", + "index": "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" + }, + "comments": [] + } + ] +} diff --git a/tests/Common/unit/conversion_data/components/test-multiline-ruby-with-hash.json b/tests/Common/unit/conversion_data/components/test-multiline-ruby-with-hash.json new file mode 100644 index 0000000..3d0be39 --- /dev/null +++ b/tests/Common/unit/conversion_data/components/test-multiline-ruby-with-hash.json @@ -0,0 +1,60 @@ +{ + "input": [ + { + "id": "input_stdin_0", + "type": "input", + "plugin": "stdin", + "config": { + "codec": { + "line": {} + } + }, + "comments": [] + } + ], + "filter": [ + { + "id": "filter_ruby_1", + "type": "filter", + "plugin": "ruby", + "config": { + "code": "\nrequire \"digest\"\nmsg = event.get(\"message\").to_s\nevent.set(\"hash\", Digest::MD5.hexdigest(msg))\n# this # is NOT a comment \u2014 it is inside a single-quoted string\nif event.get(\"level\") == \"ERROR\"\n\tevent.set(\"alert\", true)\n\tevent.set(\"severity\", \"high\")\nelsif event.get(\"level\") == \"WARN\"\n\tevent.set(\"severity\", \"medium\")\nelse\n\tevent.set(\"severity\", \"low\")\nend\n# another hash # mark inside the string \u2014 still not a comment\n\t\t" + }, + "comments": [] + }, + { + "id": "filter_ruby_2", + "type": "filter", + "plugin": "ruby", + "config": { + "init": "\nrequire \"openssl\"\nrequire \"base64\"\n# init comment inside single-quoted string\n@secret = ENV[\"SIGNING_SECRET\"] || \"default\"\n\t\t", + "code": "\npayload = event.get(\"message\").to_s\nsig = Base64.strict_encode64(\n\tOpenSSL::HMAC.digest(\"SHA256\", @secret, payload)\n)\nevent.set(\"signature\", sig)\n\t\t" + }, + "comments": [] + }, + { + "id": "filter_mutate_3", + "type": "filter", + "plugin": "mutate", + "config": { + "add_field": { + "pipeline": "ruby-test" + } + }, + "comments": [] + } + ], + "output": [ + { + "id": "output_stdout_4", + "type": "output", + "plugin": "stdout", + "config": { + "codec": { + "rubydebug": {} + } + }, + "comments": [] + } + ] +} diff --git a/tests/Common/unit/conversion_data/components/test-nested-conditionals-comments.json b/tests/Common/unit/conversion_data/components/test-nested-conditionals-comments.json new file mode 100644 index 0000000..20eaa18 --- /dev/null +++ b/tests/Common/unit/conversion_data/components/test-nested-conditionals-comments.json @@ -0,0 +1,266 @@ +{ + "input": [ + { + "id": "input_stdin_0", + "type": "input", + "plugin": "stdin", + "config": {}, + "comments": [] + } + ], + "filter": [ + { + "id": "filter_comment_1", + "type": "filter", + "plugin": "comment", + "config": { + "text": "Top-level comment before any conditional" + } + }, + { + "id": "filter_if_2", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[type] == \"web\"", + "plugins": [ + { + "id": "filter_comment_3", + "type": "filter", + "plugin": "comment", + "config": { + "text": "Comment inside first if branch" + } + }, + { + "id": "filter_if_4", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[status] >= 500", + "plugins": [ + { + "id": "filter_comment_5", + "type": "filter", + "plugin": "comment", + "config": { + "text": "Comment inside nested if" + } + }, + { + "id": "filter_mutate_6", + "type": "filter", + "plugin": "mutate", + "config": { + "add_tag": [ + "server_error" + ], + "add_field": { + "severity": "high" + } + }, + "comments": [] + }, + { + "id": "filter_comment_7", + "type": "filter", + "plugin": "comment", + "config": { + "text": "Comment after plugin inside nested if" + } + }, + { + "id": "filter_if_8", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[status] == 503", + "plugins": [ + { + "id": "filter_mutate_9", + "type": "filter", + "plugin": "mutate", + "config": { + "add_tag": [ + "service_unavailable" + ] + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_comment_10", + "type": "filter", + "plugin": "comment", + "config": { + "text": "Trailing comment inside nested if" + } + } + ], + "else_ifs": [ + { + "condition": "[status] >= 400", + "plugins": [ + { + "id": "filter_comment_11", + "type": "filter", + "plugin": "comment", + "config": { + "text": "Comment inside else-if" + } + }, + { + "id": "filter_mutate_12", + "type": "filter", + "plugin": "mutate", + "config": { + "add_tag": [ + "client_error" + ], + "add_field": { + "severity": "medium" + } + }, + "comments": [] + }, + { + "id": "filter_comment_13", + "type": "filter", + "plugin": "comment", + "config": { + "text": "Trailing comment inside else-if" + } + } + ] + } + ], + "else": { + "plugins": [ + { + "id": "filter_comment_14", + "type": "filter", + "plugin": "comment", + "config": { + "text": "Comment inside else" + } + }, + { + "id": "filter_mutate_15", + "type": "filter", + "plugin": "mutate", + "config": { + "add_tag": [ + "success" + ], + "add_field": { + "severity": "low" + } + }, + "comments": [] + } + ] + } + } + }, + { + "id": "filter_comment_16", + "type": "filter", + "plugin": "comment", + "config": { + "text": "Comment at end of outer if block" + } + } + ], + "else_ifs": [ + { + "condition": "[type] == \"db\"", + "plugins": [ + { + "id": "filter_comment_17", + "type": "filter", + "plugin": "comment", + "config": { + "text": "Comment at start of else-if block" + } + }, + { + "id": "filter_mutate_18", + "type": "filter", + "plugin": "mutate", + "config": { + "add_field": { + "source": "database" + } + }, + "comments": [] + } + ] + } + ], + "else": { + "plugins": [ + { + "id": "filter_comment_19", + "type": "filter", + "plugin": "comment", + "config": { + "text": "Comment in final else" + } + }, + { + "id": "filter_drop_20", + "type": "filter", + "plugin": "drop", + "config": {}, + "comments": [] + } + ] + } + } + }, + { + "id": "filter_comment_21", + "type": "filter", + "plugin": "comment", + "config": { + "text": "Comment between conditional and next plugin at section level" + } + }, + { + "id": "filter_mutate_22", + "type": "filter", + "plugin": "mutate", + "config": { + "add_field": { + "processed_by": "logstash" + } + }, + "comments": [] + }, + { + "id": "filter_comment_23", + "type": "filter", + "plugin": "comment", + "config": { + "text": "Trailing section-level comment" + } + } + ], + "output": [ + { + "id": "output_stdout_24", + "type": "output", + "plugin": "stdout", + "config": { + "codec": { + "rubydebug": {} + } + }, + "comments": [] + } + ] +} diff --git a/tests/Common/unit/conversion_data/components/test-regex-conditions.json b/tests/Common/unit/conversion_data/components/test-regex-conditions.json new file mode 100644 index 0000000..e86286f --- /dev/null +++ b/tests/Common/unit/conversion_data/components/test-regex-conditions.json @@ -0,0 +1,228 @@ +{ + "input": [ + { + "id": "input_syslog_0", + "type": "input", + "plugin": "syslog", + "config": { + "port": 514 + }, + "comments": [] + } + ], + "filter": [ + { + "id": "filter_if_1", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[message] =~ /^ERROR/", + "plugins": [ + { + "id": "filter_mutate_2", + "type": "filter", + "plugin": "mutate", + "config": { + "add_tag": [ + "error" + ] + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_if_3", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[message] =~ /\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}/", + "plugins": [ + { + "id": "filter_date_4", + "type": "filter", + "plugin": "date", + "config": { + "match": [ + "message", + "ISO8601" + ], + "target": "@timestamp" + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_if_5", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[host] =~ /^(web|app|db)-\\d+\\.example\\.com$/", + "plugins": [ + { + "id": "filter_mutate_6", + "type": "filter", + "plugin": "mutate", + "config": { + "add_field": { + "internal": "true" + } + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_if_7", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[message] !~ /^$/", + "plugins": [ + { + "id": "filter_grok_8", + "type": "filter", + "plugin": "grok", + "config": { + "match": { + "message": "%{GREEDYDATA:content}" + } + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_if_9", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[path] =~ /\\/api\\/v[12]\\// and [method] == \"POST\"", + "plugins": [ + { + "id": "filter_mutate_10", + "type": "filter", + "plugin": "mutate", + "config": { + "add_tag": [ + "api_write" + ] + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_if_11", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[status] =~ /^5\\d\\d$/", + "plugins": [ + { + "id": "filter_mutate_12", + "type": "filter", + "plugin": "mutate", + "config": { + "add_tag": [ + "server_error" + ] + }, + "comments": [] + } + ], + "else_ifs": [ + { + "condition": "[status] =~ /^4\\d\\d$/", + "plugins": [ + { + "id": "filter_mutate_13", + "type": "filter", + "plugin": "mutate", + "config": { + "add_tag": [ + "client_error" + ] + }, + "comments": [] + } + ] + } + ], + "else": null + } + }, + { + "id": "filter_if_14", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[user_agent] =~ /(?i)bot|crawler|spider/", + "plugins": [ + { + "id": "filter_drop_15", + "type": "filter", + "plugin": "drop", + "config": {}, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + } + ], + "output": [ + { + "id": "output_if_16", + "type": "output", + "plugin": "if", + "config": { + "condition": "\"error\" in [tags]", + "plugins": [ + { + "id": "output_file_17", + "type": "output", + "plugin": "file", + "config": { + "path": "/var/log/errors.log", + "codec": { + "json": {} + } + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "output_stdout_18", + "type": "output", + "plugin": "stdout", + "config": { + "codec": { + "rubydebug": {} + } + }, + "comments": [] + } + ] +} diff --git a/tests/Common/unit/conversion_data/components/test-sample-nginx.json b/tests/Common/unit/conversion_data/components/test-sample-nginx.json new file mode 100644 index 0000000..a3b972d --- /dev/null +++ b/tests/Common/unit/conversion_data/components/test-sample-nginx.json @@ -0,0 +1,183 @@ +{ + "input": [ + { + "id": "input_stdin_0", + "type": "input", + "plugin": "stdin", + "config": { + "codec": { + "line": {} + } + }, + "comments": [] + } + ], + "filter": [ + { + "id": "filter_mutate_1", + "type": "filter", + "plugin": "mutate", + "config": { + "add_field": { + "event.dataset": "nginx.access", + "service.name": "nginx" + } + }, + "comments": [] + }, + { + "id": "filter_grok_2", + "type": "filter", + "plugin": "grok", + "config": { + "match": { + "message": [ + "%{IPORHOST:source.address} %{DATA:nginx.ident} %{DATA:user.name} \\[%{HTTPDATE:nginx.access.time}\\] \"%{WORD:http.request.method} %{DATA:url.original} HTTP/%{NUMBER:http.version}\" %{NUMBER:http.response.status_code:int} (?:%{NUMBER:http.response.body.bytes:int}|-) \"%{DATA:http.request.referrer}\" \"%{DATA:user_agent.original}\" %{NUMBER:nginx.access.request_time:float}", + "%{IPORHOST:source.address} %{DATA:nginx.ident} %{DATA:user.name} \\[%{HTTPDATE:nginx.access.time}\\] \"%{WORD:http.request.method} %{DATA:url.original} HTTP/%{NUMBER:http.version}\" %{NUMBER:http.response.status_code:int} (?:%{NUMBER:http.response.body.bytes:int}|-) \"%{DATA:http.request.referrer}\" \"%{DATA:user_agent.original}\"", + "%{IPORHOST:source.address} %{DATA:nginx.ident} %{DATA:user.name} \\[%{HTTPDATE:nginx.access.time}\\] \"%{WORD:http.request.method} %{DATA:url.original} HTTP/%{NUMBER:http.version}\" %{NUMBER:http.response.status_code:int} (?:%{NUMBER:http.response.body.bytes:int}|-) \"%{DATA:http.request.referrer}\" \"%{DATA:user_agent.original}\" (?:rt=%{NUMBER:nginx.access.request_time:float}\\s+uct=%{NUMBER:nginx.access.upstream_connect_time:float}\\s+uht=%{NUMBER:nginx.access.upstream_header_time:float}\\s+urt=%{NUMBER:nginx.access.upstream_response_time:float})" + ] + }, + "tag_on_failure": [ + "_grok_nginx_access_fail" + ] + }, + "comments": [] + }, + { + "id": "filter_date_3", + "type": "filter", + "plugin": "date", + "config": { + "match": [ + "nginx.access.time", + "dd/MMM/yyyy:HH:mm:ss Z" + ], + "target": "@timestamp" + }, + "comments": [] + }, + { + "id": "filter_urldecode_4", + "type": "filter", + "plugin": "urldecode", + "config": { + "field": "url.original" + }, + "comments": [] + }, + { + "id": "filter_dissect_5", + "type": "filter", + "plugin": "dissect", + "config": { + "mapping": { + "url.original": "%{url.path}?%{url.query}" + } + }, + "comments": [] + }, + { + "id": "filter_useragent_6", + "type": "filter", + "plugin": "useragent", + "config": { + "source": "user_agent.original", + "target": "user_agent" + }, + "comments": [] + }, + { + "id": "filter_mutate_7", + "type": "filter", + "plugin": "mutate", + "config": { + "copy": { + "source.address": "source.ip" + } + }, + "comments": [] + }, + { + "id": "filter_geoip_8", + "type": "filter", + "plugin": "geoip", + "config": { + "source": "source.ip", + "target": "source.geo", + "tag_on_failure": [ + "_geoip_fail" + ] + }, + "comments": [] + }, + { + "id": "filter_mutate_9", + "type": "filter", + "plugin": "mutate", + "config": { + "gsub": [ + "http.request.referrer", + "^-$", + "", + "user.name", + "^-$", + "" + ] + }, + "comments": [] + }, + { + "id": "filter_if_10", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[http][response][status_code] and [http][response][status_code] >= 500", + "plugins": [ + { + "id": "filter_mutate_11", + "type": "filter", + "plugin": "mutate", + "config": { + "add_tag": [ + "nginx_server_error" + ] + }, + "comments": [] + } + ], + "else_ifs": [ + { + "condition": "[http][response][status_code] and [http][response][status_code] >= 400", + "plugins": [ + { + "id": "filter_mutate_12", + "type": "filter", + "plugin": "mutate", + "config": { + "add_tag": [ + "nginx_client_error" + ] + }, + "comments": [] + } + ] + } + ], + "else": null + } + } + ], + "output": [ + { + "id": "output_stdout_13", + "type": "output", + "plugin": "stdout", + "config": { + "codec": { + "rubydebug": {} + } + }, + "comments": [] + } + ] +} diff --git a/tests/Common/unit/conversion_data/components/test-snmp-v0.2.json b/tests/Common/unit/conversion_data/components/test-snmp-v0.2.json new file mode 100644 index 0000000..2d17fe9 --- /dev/null +++ b/tests/Common/unit/conversion_data/components/test-snmp-v0.2.json @@ -0,0 +1,212 @@ +{ + "input": [ + { + "id": "input_snmp_0", + "type": "input", + "plugin": "snmp", + "config": { + "hosts": [ + { + "host": "udp:1.2.3.4/161", + "version": "3", + "timeout": 1000, + "retries": 2 + } + ], + "interval": "30", + "security_name": "test", + "security_level": "authPriv", + "ecs_compatibility": "disabled", + "oid_mapping_format": "dotted_string", + "auth_protocol": "sha", + "auth_pass": "test", + "priv_protocol": "aes", + "priv_pass": "test", + "get": [ + "1.3.6.1.4.1.9.2.1.57.0", + "1.3.6.1.4.1.9.9.48.1.1.1.5.1", + "1.3.6.1.4.1.9.9.48.1.1.1.6.1", + "1.3.6.1.2.1.1.1.0", + "1.3.6.1.2.1.1.5.0", + "1.3.6.1.2.1.1.2.0", + "1.3.6.1.2.1.1.3.0" + ], + "tables": [ + { + "name": "cdpCacheTable", + "columns": [ + "1.3.6.1.4.1.9.9.23.1.2.1.1.1", + "1.3.6.1.4.1.9.9.23.1.2.1.1.6", + "1.3.6.1.4.1.9.9.23.1.2.1.1.7", + "1.3.6.1.4.1.9.9.23.1.2.1.1.8", + "1.3.6.1.4.1.9.9.23.1.2.1.1.9", + "1.3.6.1.4.1.9.9.23.1.2.1.1.5", + "1.3.6.1.4.1.9.9.23.1.2.1.1.4" + ] + }, + { + "name": "sensors", + "columns": [ + "1.3.6.1.4.1.9.9.13.1.3.1.2", + "1.3.6.1.4.1.9.9.13.1.3.1.3", + "1.3.6.1.4.1.9.9.13.1.3.1.4", + "1.3.6.1.4.1.9.9.13.1.3.1.5", + "1.3.6.1.4.1.9.9.13.1.3.1.6" + ] + }, + { + "name": "fans", + "columns": [ + "1.3.6.1.4.1.9.9.13.1.4.1.2", + "1.3.6.1.4.1.9.9.13.1.4.1.3" + ] + }, + { + "name": "interfaces", + "columns": [ + "1.3.6.1.2.1.2.2.1.1", + "1.3.6.1.2.1.2.2.1.2", + "1.3.6.1.2.1.2.2.1.3", + "1.3.6.1.2.1.2.2.1.7", + "1.3.6.1.2.1.2.2.1.8", + "1.3.6.1.2.1.31.1.1.1.1", + "1.3.6.1.2.1.31.1.1.1.18", + "1.3.6.1.2.1.31.1.1.1.15", + "1.3.6.1.2.1.2.2.1.5", + "1.3.6.1.2.1.2.2.1.6", + "1.3.6.1.2.1.2.2.1.4", + "1.3.6.1.2.1.31.1.1.1.6", + "1.3.6.1.2.1.31.1.1.1.10", + "1.3.6.1.2.1.31.1.1.1.9", + "1.3.6.1.2.1.31.1.1.1.13", + "1.3.6.1.2.1.31.1.1.1.7", + "1.3.6.1.2.1.31.1.1.1.11", + "1.3.6.1.2.1.31.1.1.1.8", + "1.3.6.1.2.1.31.1.1.1.12", + "1.3.6.1.2.1.2.2.1.9", + "1.3.6.1.2.1.17.7.1.4.5.1.1", + "1.3.6.1.2.1.2.2.1.14", + "1.3.6.1.2.1.2.2.1.20", + "1.3.6.1.2.1.2.2.1.13", + "1.3.6.1.2.1.2.2.1.19" + ] + } + ] + }, + "comments": [] + } + ], + "filter": [ + { + "id": "filter_mutate_1", + "type": "filter", + "plugin": "mutate", + "config": { + "rename": { + "host": "[host][hostname]" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_2", + "type": "filter", + "plugin": "mutate", + "config": { + "rename": { + "1.3.6.1.4.1.9.2.1.57.0": "[system][cpu][total][norm][pct]", + "1.3.6.1.4.1.9.9.48.1.1.1.5.1": "[system][memory][actual][used][bytes]", + "1.3.6.1.4.1.9.9.48.1.1.1.6.1": "[system][memory][actual][free][bytes]", + "1.3.6.1.2.1.1.1.0": "[host][description]", + "1.3.6.1.2.1.1.5.0": "[host][name]", + "1.3.6.1.2.1.1.2.0": "[host][id]", + "1.3.6.1.2.1.1.3.0": "[host][uptime]" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_3", + "type": "filter", + "plugin": "mutate", + "config": { + "add_field": { + "[network][name]": "home-segment-1 (192.168.4.0/24)", + "[metricset][module]": "system" + } + }, + "comments": [] + }, + { + "id": "filter_ruby_4", + "type": "filter", + "plugin": "ruby", + "config": { + "code": " v = event.get(\"[system][cpu][total][norm][pct]\")\n if v\n event.set(\"[system][cpu][total][norm][pct]\", v.to_f / 100.0)\n end" + }, + "comments": [] + }, + { + "id": "filter_ruby_5", + "type": "filter", + "plugin": "ruby", + "config": { + "code": "\n used = event.get(\"[system][memory][actual][used][bytes]\")\n free = event.get(\"[system][memory][actual][free][bytes]\")\n\t\t\n if used && free\n used_f = used.to_f\n free_f = free.to_f\n total_f = used_f + free_f\n\t\t\n if total_f > 0\n event.set(\"[system][memory][total]\", total_f)\n event.set(\"[system][memory][actual][used][pct]\", (used_f / total_f))\n event.set(\"[system][memory][actual][free][pct]\", (free_f / total_f))\n end\n end\n " + }, + "comments": [] + }, + { + "id": "filter_ruby_6", + "type": "filter", + "plugin": "ruby", + "config": { + "code": "rows = event.get('[cdpCacheTable]')\nif rows.is_a?(Array)\n host_name = event.get('[host][name]')\n host_hostname = event.get('[host][hostname]')\n network_name = event.get('[network][name]')\n timestamp = event.get('@timestamp')\n rows.each do |row|\n next unless row.is_a?(Hash)\n row['cdpCacheIfIndex'] = row.delete('1.3.6.1.4.1.9.9.23.1.2.1.1.1')\n row['cdpCacheDeviceId'] = row.delete('1.3.6.1.4.1.9.9.23.1.2.1.1.6')\n row['cdpCacheDevicePort'] = row.delete('1.3.6.1.4.1.9.9.23.1.2.1.1.7')\n row['cdpCachePlatform'] = row.delete('1.3.6.1.4.1.9.9.23.1.2.1.1.8')\n row['cdpCacheCapabilities'] = row.delete('1.3.6.1.4.1.9.9.23.1.2.1.1.9')\n row['cdpCacheVersion'] = row.delete('1.3.6.1.4.1.9.9.23.1.2.1.1.5')\n row['cdpCacheAddress'] = row.delete('1.3.6.1.4.1.9.9.23.1.2.1.1.4')\n new_event = LogStash::Event.new({\n '@timestamp' => timestamp,\n 'host' => { 'name' => host_name, 'hostname' => host_hostname },\n 'network' => { 'name' => network_name },\n 'table' => row,\n 'metricset' => { 'module' => 'snmp' },\n 'event' => { 'kind' => 'cdpcachetable' }\n })\n new_event_block.call(new_event)\n end\n event.remove('[cdpCacheTable]')\n event.set('[event][kind]', 'metrics')\nend" + }, + "comments": [] + }, + { + "id": "filter_ruby_7", + "type": "filter", + "plugin": "ruby", + "config": { + "code": "rows = event.get('[sensors]')\nif rows.is_a?(Array)\n host_name = event.get('[host][name]')\n host_hostname = event.get('[host][hostname]')\n network_name = event.get('[network][name]')\n timestamp = event.get('@timestamp')\n rows.each do |row|\n next unless row.is_a?(Hash)\n row['description'] = row.delete('1.3.6.1.4.1.9.9.13.1.3.1.2')\n row['temp_celsius'] = row.delete('1.3.6.1.4.1.9.9.13.1.3.1.3')\n row['temp_threshold'] = row.delete('1.3.6.1.4.1.9.9.13.1.3.1.4')\n row['temp_last_shutdown'] = row.delete('1.3.6.1.4.1.9.9.13.1.3.1.5')\n row['state'] = row.delete('1.3.6.1.4.1.9.9.13.1.3.1.6')\n new_event = LogStash::Event.new({\n '@timestamp' => timestamp,\n 'host' => { 'name' => host_name, 'hostname' => host_hostname },\n 'network' => { 'name' => network_name },\n 'table' => row,\n 'metricset' => { 'module' => 'snmp' },\n 'event' => { 'kind' => 'sensors' }\n })\n new_event_block.call(new_event)\n end\n event.remove('[sensors]')\n event.set('[event][kind]', 'metrics')\nend" + }, + "comments": [] + }, + { + "id": "filter_ruby_8", + "type": "filter", + "plugin": "ruby", + "config": { + "code": "rows = event.get('[fans]')\nif rows.is_a?(Array)\n host_name = event.get('[host][name]')\n host_hostname = event.get('[host][hostname]')\n network_name = event.get('[network][name]')\n timestamp = event.get('@timestamp')\n rows.each do |row|\n next unless row.is_a?(Hash)\n row['description'] = row.delete('1.3.6.1.4.1.9.9.13.1.4.1.2')\n row['state'] = row.delete('1.3.6.1.4.1.9.9.13.1.4.1.3')\n new_event = LogStash::Event.new({\n '@timestamp' => timestamp,\n 'host' => { 'name' => host_name, 'hostname' => host_hostname },\n 'network' => { 'name' => network_name },\n 'table' => row,\n 'metricset' => { 'module' => 'snmp' },\n 'event' => { 'kind' => 'fans' }\n })\n new_event_block.call(new_event)\n end\n event.remove('[fans]')\n event.set('[event][kind]', 'metrics')\nend" + }, + "comments": [] + }, + { + "id": "filter_ruby_9", + "type": "filter", + "plugin": "ruby", + "config": { + "code": "rows = event.get('[interfaces]')\nif rows.is_a?(Array)\n host_name = event.get('[host][name]')\n host_hostname = event.get('[host][hostname]')\n network_name = event.get('[network][name]')\n timestamp = event.get('@timestamp')\n rows.each do |row|\n next unless row.is_a?(Hash)\n row['ifIndex'] = row.delete('1.3.6.1.2.1.2.2.1.1')\n row['ifDescr'] = row.delete('1.3.6.1.2.1.2.2.1.2')\n row['ifType'] = row.delete('1.3.6.1.2.1.2.2.1.3')\n row['ifAdminStatus'] = row.delete('1.3.6.1.2.1.2.2.1.7')\n row['ifOperStatus'] = row.delete('1.3.6.1.2.1.2.2.1.8')\n row['ifName'] = row.delete('1.3.6.1.2.1.31.1.1.1.1')\n row['ifAlias'] = row.delete('1.3.6.1.2.1.31.1.1.1.18')\n row['ifHighSpeed'] = row.delete('1.3.6.1.2.1.31.1.1.1.15')\n row['ifSpeed'] = row.delete('1.3.6.1.2.1.2.2.1.5')\n row['ifPhysAddress'] = row.delete('1.3.6.1.2.1.2.2.1.6')\n row['ifMtu'] = row.delete('1.3.6.1.2.1.2.2.1.4')\n row['ifHCInOctets'] = row.delete('1.3.6.1.2.1.31.1.1.1.6')\n row['ifHCOutOctets'] = row.delete('1.3.6.1.2.1.31.1.1.1.10')\n row['ifHCInBroadcastPkts'] = row.delete('1.3.6.1.2.1.31.1.1.1.9')\n row['ifHCOutBroadcastPkts'] = row.delete('1.3.6.1.2.1.31.1.1.1.13')\n row['ifHCInUcastPkts'] = row.delete('1.3.6.1.2.1.31.1.1.1.7')\n row['ifHCOutUcastPkts'] = row.delete('1.3.6.1.2.1.31.1.1.1.11')\n row['ifHCInMulticastPkts'] = row.delete('1.3.6.1.2.1.31.1.1.1.8')\n row['ifHCOutMulticastPkts'] = row.delete('1.3.6.1.2.1.31.1.1.1.12')\n row['ifLastChange'] = row.delete('1.3.6.1.2.1.2.2.1.9')\n row['dot1qPvid'] = row.delete('1.3.6.1.2.1.17.7.1.4.5.1.1')\n row['ifInErrors'] = row.delete('1.3.6.1.2.1.2.2.1.14')\n row['ifOutErrors'] = row.delete('1.3.6.1.2.1.2.2.1.20')\n row['ifInDiscards'] = row.delete('1.3.6.1.2.1.2.2.1.13')\n row['ifOutDiscards'] = row.delete('1.3.6.1.2.1.2.2.1.19')\n new_event = LogStash::Event.new({\n '@timestamp' => timestamp,\n 'host' => { 'name' => host_name, 'hostname' => host_hostname },\n 'network' => { 'name' => network_name },\n 'table' => row,\n 'metricset' => { 'module' => 'snmp' },\n 'event' => { 'kind' => 'interfaces' }\n })\n new_event_block.call(new_event)\n end\n event.remove('[interfaces]')\n event.set('[event][kind]', 'metrics')\nend" + }, + "comments": [] + } + ], + "output": [ + { + "id": "output_elasticsearch_10", + "type": "output", + "plugin": "elasticsearch", + "config": { + "data_stream": "true", + "data_stream_type": "metrics", + "data_stream_namespace": "default", + "data_stream_dataset": "snmp.polling", + "cloud_id": "test", + "user": "test", + "password": "test" + }, + "comments": [] + } + ] +} diff --git a/tests/Common/unit/conversion_data/components/test-string-escaping.json b/tests/Common/unit/conversion_data/components/test-string-escaping.json new file mode 100644 index 0000000..1216586 --- /dev/null +++ b/tests/Common/unit/conversion_data/components/test-string-escaping.json @@ -0,0 +1,60 @@ +{ + "input": [], + "filter": [ + { + "id": "filter_mutate_0", + "type": "filter", + "plugin": "mutate", + "config": { + "add_field": { + "double_quoted_with_hash": "value # not a comment", + "with_brackets": "data [in] brackets {and} braces", + "with_arrow": "key => value pattern", + "env_ref": "${MY_VAR}", + "sprintf_ref": "prefix-%{field_name}" + } + }, + "comments": [] + }, + { + "id": "filter_grok_1", + "type": "filter", + "plugin": "grok", + "config": { + "match": { + "message": "%{IP:client} \\[%{HTTPDATE:ts}\\] \"%{WORD:method} %{URIPATHPARAM:path}" + }, + "pattern_definitions": { + "CUSTOM_IP": "\\b(?:\\d{1,3}\\.){3}\\d{1,3}\\b" + } + }, + "comments": [] + }, + { + "id": "filter_mutate_2", + "type": "filter", + "plugin": "mutate", + "config": { + "rename": { + "@timestamp": "event_time", + "host": "source_host" + } + }, + "comments": [] + } + ], + "output": [ + { + "id": "output_file_3", + "type": "output", + "plugin": "file", + "config": { + "path": "/var/log/output/%{type}/%{+YYYY}/%{+MM}/%{+dd}.log", + "codec": { + "json": {} + } + }, + "comments": [] + } + ] +} diff --git a/tests/Common/unit/conversion_data/components/test-twitter.json b/tests/Common/unit/conversion_data/components/test-twitter.json new file mode 100644 index 0000000..e2dcd67 --- /dev/null +++ b/tests/Common/unit/conversion_data/components/test-twitter.json @@ -0,0 +1,89 @@ +{ + "input": [ + { + "id": "input_comment_0", + "type": "input", + "plugin": "comment", + "config": { + "text": "This is the sample pipeline whose screenshots are used in\nthe Pipeline Viewer documentation (../pipeline-viewer.asciidoc)\n\nWhenever the Pipeline Viewer UI changes, run this pipeline and\nopen in the new UI to take updated screenshots.\n\nNote: you will have to setup the environment variables used\nbelow. Refer to the Twitter Logstash Input plugin documentation\nfor their expected values" + } + }, + { + "id": "input_twitter_1", + "type": "input", + "plugin": "twitter", + "config": { + "id": "tweet harvester", + "consumer_key": "${TWITTER_API_CONSUMER_KEY}", + "consumer_secret": "${TWITTER_API_CONSUMER_SECRET}", + "keywords": [ + "rain", + "monsoon", + "shower", + "drizzle" + ], + "oauth_token": "${TWITTER_API_OAUTH_TOKEN}", + "oauth_token_secret": "${TWITTER_API_OAUTH_TOKEN_SECRET}" + }, + "comments": [] + } + ], + "filter": [ + { + "id": "filter_grok_2", + "type": "filter", + "plugin": "grok", + "config": { + "match": { + "message": "%{WORD:is_rt}" + } + }, + "comments": [] + }, + { + "id": "filter_if_3", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[is_rt] == \"RT\"", + "plugins": [ + { + "id": "filter_drop_4", + "type": "filter", + "plugin": "drop", + "config": { + "id": "drop_all_RTs" + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + } + ], + "output": [ + { + "id": "output_stdout_5", + "type": "output", + "plugin": "stdout", + "config": { + "codec": { + "dots": {} + } + }, + "comments": [] + }, + { + "id": "output_elasticsearch_6", + "type": "output", + "plugin": "elasticsearch", + "config": { + "user": "elastic", + "password": "changeme", + "index": "tweets" + }, + "comments": [] + } + ] +} diff --git a/tests/Common/unit/conversion_data/components/test_complex1.json b/tests/Common/unit/conversion_data/components/test_complex1.json new file mode 100644 index 0000000..ccaef73 --- /dev/null +++ b/tests/Common/unit/conversion_data/components/test_complex1.json @@ -0,0 +1,952 @@ +{ + "input": [ + { + "id": "input_comment_0", + "type": "input", + "plugin": "comment", + "config": { + "text": "\n\"LogstashUI kitchen sink\" pipeline\nGoal: be extremely feature-rich while staying within known-valid plugin options.\n\nBeats / Elastic Agent style shippers" + } + }, + { + "id": "input_beats_1", + "type": "input", + "plugin": "beats", + "config": { + "id": "in_beats_5044", + "port": "5044", + "add_field": { + "ingest_transport": "beats" + }, + "tags": [ + "from_beats" + ] + }, + "comments": [] + }, + { + "id": "input_comment_2", + "type": "input", + "plugin": "comment", + "config": { + "text": "JSON-over-TCP (common for app logs)" + } + }, + { + "id": "input_tcp_3", + "type": "input", + "plugin": "tcp", + "config": { + "id": "in_tcp_json_5514", + "port": "5514", + "mode": "server", + "codec": { + "json": {} + }, + "add_field": { + "ingest_transport": "tcp" + }, + "tags": [ + "from_tcp" + ] + }, + "comments": [] + }, + { + "id": "input_comment_4", + "type": "input", + "plugin": "comment", + "config": { + "text": "Syslog-ish UDP" + } + }, + { + "id": "input_udp_5", + "type": "input", + "plugin": "udp", + "config": { + "id": "in_udp_5515", + "port": "5515", + "codec": { + "plain": {} + }, + "add_field": { + "ingest_transport": "udp" + }, + "tags": [ + "from_udp" + ] + }, + "comments": [] + }, + { + "id": "input_comment_6", + "type": "input", + "plugin": "comment", + "config": { + "text": "HTTP event intake (webhooks, apps posting JSON, etc.)" + } + }, + { + "id": "input_http_7", + "type": "input", + "plugin": "http", + "config": { + "id": "in_http_8080", + "port": "8080", + "codec": { + "json": {} + }, + "add_field": { + "ingest_transport": "http" + }, + "tags": [ + "from_http" + ] + }, + "comments": [] + }, + { + "id": "input_comment_8", + "type": "input", + "plugin": "comment", + "config": { + "text": "Local dev/testing input" + } + }, + { + "id": "input_stdin_9", + "type": "input", + "plugin": "stdin", + "config": { + "id": "in_stdin", + "codec": { + "line": {} + }, + "add_field": { + "ingest_transport": "stdin" + }, + "tags": [ + "from_stdin" + ] + }, + "comments": [] + }, + { + "id": "input_comment_10", + "type": "input", + "plugin": "comment", + "config": { + "text": "Synthetic test data (makes it easy to validate end-to-end quickly)" + } + }, + { + "id": "input_generator_11", + "type": "input", + "plugin": "generator", + "config": { + "id": "in_generator", + "lines": [ + "Feb 21 09:12:01 host1 sshd[123]: Failed password for invalid user admin from 10.1.2.3 port 51234 ssh2", + "{\"@timestamp\":\"2026-02-21T14:12:02Z\",\"message\":\"GET /health 200\",\"source_ip\":\"8.8.8.8\",\"user_agent\":\"Mozilla/5.0\"}", + "level=info service=api latency_ms=42 source_ip=192.168.1.50 msg=\"request completed\"" + ], + "count": "1", + "add_field": { + "ingest_transport": "generator" + }, + "tags": [ + "from_generator" + ] + }, + "comments": [] + } + ], + "filter": [ + { + "id": "filter_comment_12", + "type": "filter", + "plugin": "comment", + "config": { + "text": "\nNormalize a few shared fields\n" + } + }, + { + "id": "filter_mutate_13", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "f_mutate_bootstrap", + "add_field": { + "[@metadata][pipeline]": "logstashui_kitchen_sink", + "event.module": "logstashui" + } + }, + "comments": [] + }, + { + "id": "filter_comment_14", + "type": "filter", + "plugin": "comment", + "config": { + "text": "Keep a canonical message field" + } + }, + { + "id": "filter_if_15", + "type": "filter", + "plugin": "if", + "config": { + "condition": "![message] and [event][original]", + "plugins": [ + { + "id": "filter_mutate_16", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "f_mutate_event_original_to_message", + "copy": { + "[event][original]": "message" + } + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_comment_17", + "type": "filter", + "plugin": "comment", + "config": { + "text": "\nTry to parse JSON *if* message looks like JSON (common when tcp/udp/plain feed JSON strings)\n" + } + }, + { + "id": "filter_if_18", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[message] =~ \"^[[:space:]]*\\\\{\"", + "plugins": [ + { + "id": "filter_json_19", + "type": "filter", + "plugin": "json", + "config": { + "id": "f_json_from_message", + "source": "message", + "target": "json", + "tag_on_failure": [ + "_jsonparsefailure_message" + ] + }, + "comments": [] + }, + { + "id": "filter_comment_20", + "type": "filter", + "plugin": "comment", + "config": { + "text": "If json parsed, promote a few expected keys (only if present)" + } + }, + { + "id": "filter_if_21", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[json][@timestamp]", + "plugins": [ + { + "id": "filter_mutate_22", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "f_promote_json_ts", + "copy": { + "[json][@timestamp]": "@timestamp" + } + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_if_23", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[json][source_ip]", + "plugins": [ + { + "id": "filter_mutate_24", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "f_promote_json_source_ip", + "copy": { + "[json][source_ip]": "source_ip" + } + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_if_25", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[json][user_agent]", + "plugins": [ + { + "id": "filter_mutate_26", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "f_promote_json_ua", + "copy": { + "[json][user_agent]": "user_agent" + } + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_comment_27", + "type": "filter", + "plugin": "comment", + "config": { + "text": "\nSyslog-ish parsing (UDP and some TCP)\n" + } + }, + { + "id": "filter_if_28", + "type": "filter", + "plugin": "if", + "config": { + "condition": "\"from_udp\" in [tags] or \"from_tcp\" in [tags]", + "plugins": [ + { + "id": "filter_comment_29", + "type": "filter", + "plugin": "comment", + "config": { + "text": "Try dissect first (fast) and fall back to grok" + } + }, + { + "id": "filter_dissect_30", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "f_dissect_syslogish", + "mapping": { + "message": "%{syslog_timestamp} %{syslog_host} %{syslog_program}[%{syslog_pid}]: %{syslog_message}" + }, + "tag_on_failure": [ + "_dissectfailure_syslogish" + ] + }, + "comments": [] + }, + { + "id": "filter_if_31", + "type": "filter", + "plugin": "if", + "config": { + "condition": "\"_dissectfailure_syslogish\" in [tags]", + "plugins": [ + { + "id": "filter_grok_32", + "type": "filter", + "plugin": "grok", + "config": { + "id": "f_grok_syslogish", + "match": { + "message": [ + "%{SYSLOGTIMESTAMP:syslog_timestamp} %{HOSTNAME:syslog_host} %{DATA:syslog_program}(?:\\[%{POSINT:syslog_pid}\\])?: %{GREEDYDATA:syslog_message}" + ] + }, + "tag_on_failure": [ + "_grokparsefailure_syslogish" + ] + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_comment_33", + "type": "filter", + "plugin": "comment", + "config": { + "text": "If we extracted a syslog timestamp, use it" + } + }, + { + "id": "filter_if_34", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[syslog_timestamp]", + "plugins": [ + { + "id": "filter_date_35", + "type": "filter", + "plugin": "date", + "config": { + "id": "f_date_syslog", + "match": [ + "syslog_timestamp", + "MMM d HH:mm:ss", + "MMM dd HH:mm:ss" + ], + "tag_on_failure": [ + "_dateparsefailure_syslog" + ] + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_comment_36", + "type": "filter", + "plugin": "comment", + "config": { + "text": "\nkey=value parsing for \u201cflat\u201d log lines\n" + } + }, + { + "id": "filter_if_37", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[message] =~ \"([A-Za-z0-9_.-]+)=([^\\\"]\\\\S+|\\\"[^\\\"]*\\\")\"", + "plugins": [ + { + "id": "filter_kv_38", + "type": "filter", + "plugin": "kv", + "config": { + "id": "f_kv_message", + "source": "message", + "trim_key": " ", + "trim_value": " ", + "value_split": "=", + "field_split_pattern": "\\s+", + "tag_on_failure": [ + "_kvfailure_message" + ] + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_comment_39", + "type": "filter", + "plugin": "comment", + "config": { + "text": "\nBasic typing / normalization\n" + } + }, + { + "id": "filter_mutate_40", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "f_mutate_normalize", + "rename": { + "msg": "message_short" + }, + "convert": { + "latency_ms": "integer" + }, + "lowercase": [ + "level" + ] + }, + "comments": [] + }, + { + "id": "filter_comment_41", + "type": "filter", + "plugin": "comment", + "config": { + "text": "\nEnrichments: useragent, geoip, cidr, dns\n" + } + }, + { + "id": "filter_if_42", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[user_agent]", + "plugins": [ + { + "id": "filter_useragent_43", + "type": "filter", + "plugin": "useragent", + "config": { + "id": "f_useragent", + "source": "user_agent", + "target": "user_agent_parsed" + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_comment_44", + "type": "filter", + "plugin": "comment", + "config": { + "text": "Canonicalize IP into source_ip if it exists elsewhere" + } + }, + { + "id": "filter_if_45", + "type": "filter", + "plugin": "if", + "config": { + "condition": "![source_ip] and [source][ip]", + "plugins": [ + { + "id": "filter_mutate_46", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "f_copy_source_ip", + "copy": { + "[source][ip]": "source_ip" + } + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_if_47", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[source_ip]", + "plugins": [ + { + "id": "filter_comment_48", + "type": "filter", + "plugin": "comment", + "config": { + "text": "Tag private vs public" + } + }, + { + "id": "filter_cidr_49", + "type": "filter", + "plugin": "cidr", + "config": { + "id": "f_cidr_private", + "address": [ + "%{source_ip}" + ], + "network": [ + "10.0.0.0/8", + "172.16.0.0/12", + "192.168.0.0/16" + ], + "add_tag": [ + "src_private" + ] + }, + "comments": [] + }, + { + "id": "filter_comment_50", + "type": "filter", + "plugin": "comment", + "config": { + "text": "GeoIP typically only makes sense for public IPs, so do it only if not private-tagged" + } + }, + { + "id": "filter_if_51", + "type": "filter", + "plugin": "if", + "config": { + "condition": "\"src_private\" not in [tags]", + "plugins": [ + { + "id": "filter_geoip_52", + "type": "filter", + "plugin": "geoip", + "config": { + "id": "f_geoip", + "source": "source_ip", + "target": "source_geo" + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_comment_53", + "type": "filter", + "plugin": "comment", + "config": { + "text": "Reverse DNS lookup; replace source_ip with hostname when possible (or leave as-is)" + } + }, + { + "id": "filter_dns_54", + "type": "filter", + "plugin": "dns", + "config": { + "id": "f_dns_reverse", + "reverse": [ + "source_ip" + ], + "action": "replace" + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_comment_55", + "type": "filter", + "plugin": "comment", + "config": { + "text": "\nTranslate severity/level into a normalized numeric\n" + } + }, + { + "id": "filter_translate_56", + "type": "filter", + "plugin": "translate", + "config": { + "id": "f_translate_level_to_severity", + "source": "level", + "target": "severity", + "dictionary": { + "trace": "0", + "debug": "1", + "info": "2", + "warn": "3", + "error": "4", + "fatal": "5" + }, + "fallback": "2" + }, + "comments": [] + }, + { + "id": "filter_mutate_57", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "f_convert_severity_int", + "convert": { + "severity": "integer" + } + }, + "comments": [] + }, + { + "id": "filter_comment_58", + "type": "filter", + "plugin": "comment", + "config": { + "text": "\nStable fingerprint for dedup / correlation\n" + } + }, + { + "id": "filter_fingerprint_59", + "type": "filter", + "plugin": "fingerprint", + "config": { + "id": "f_fingerprint_message", + "source": [ + "message" + ], + "method": "MURMUR3", + "target": "[@metadata][fingerprint]" + }, + "comments": [] + }, + { + "id": "filter_comment_60", + "type": "filter", + "plugin": "comment", + "config": { + "text": "\nExample branching: treat auth-ish messages specially\n" + } + }, + { + "id": "filter_if_61", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[syslog_program] == \"sshd\" or [message] =~ \"(?i)failed password|authentication failure|invalid user\"", + "plugins": [ + { + "id": "filter_mutate_62", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "f_tag_auth", + "add_tag": [ + "category_auth" + ], + "add_field": { + "event.category": "authentication" + } + }, + "comments": [] + } + ], + "else_ifs": [ + { + "condition": "[message] =~ \"(?i)GET\\\\s+/health|/ready|/live\"", + "plugins": [ + { + "id": "filter_mutate_63", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "f_tag_health", + "add_tag": [ + "category_healthcheck" + ], + "add_field": { + "event.category": "availability" + } + }, + "comments": [] + } + ] + } + ], + "else": { + "plugins": [ + { + "id": "filter_mutate_64", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "f_tag_generic", + "add_tag": [ + "category_generic" + ] + }, + "comments": [] + } + ] + } + } + }, + { + "id": "filter_comment_65", + "type": "filter", + "plugin": "comment", + "config": { + "text": "\nPrune down noisy fields (keeps top-level essentials)\n" + } + }, + { + "id": "filter_prune_66", + "type": "filter", + "plugin": "prune", + "config": { + "id": "f_prune", + "whitelist_names": [ + "^@timestamp$", + "^message$", + "^message_short$", + "^host$", + "^source_ip$", + "^source_geo$", + "^severity$", + "^level$", + "^tags$", + "^event\\..*$", + "^user_agent.*$", + "^syslog_.*$", + "^ingest_transport$" + ] + }, + "comments": [] + } + ], + "output": [ + { + "id": "output_comment_67", + "type": "output", + "plugin": "comment", + "config": { + "text": "Always see something in console during dev" + } + }, + { + "id": "output_stdout_68", + "type": "output", + "plugin": "stdout", + "config": { + "id": "out_stdout_rubydebug", + "codec": { + "rubydebug": { + "metadata": "true" + } + } + }, + "comments": [] + }, + { + "id": "output_comment_69", + "type": "output", + "plugin": "comment", + "config": { + "text": "Write to disk (great for debugging replay)" + } + }, + { + "id": "output_file_70", + "type": "output", + "plugin": "file", + "config": { + "id": "out_file_jsonl", + "path": "/tmp/logstashui-%{+YYYY.MM.dd}.jsonl", + "codec": { + "json_lines": {} + } + }, + "comments": [] + }, + { + "id": "output_comment_71", + "type": "output", + "plugin": "comment", + "config": { + "text": "Elasticsearch (local default)" + } + }, + { + "id": "output_elasticsearch_72", + "type": "output", + "plugin": "elasticsearch", + "config": { + "id": "out_es_local", + "hosts": [ + "http://localhost:9200" + ], + "index": "logstashui-%{+YYYY.MM.dd}", + "ilm_enabled": "false" + }, + "comments": [] + }, + { + "id": "output_comment_73", + "type": "output", + "plugin": "comment", + "config": { + "text": "Webhook back to your UI/API (example)" + } + }, + { + "id": "output_http_74", + "type": "output", + "plugin": "http", + "config": { + "id": "out_http_callback", + "url": "http://localhost:9000/logstash/callback", + "http_method": "post", + "format": "json" + }, + "comments": [] + }, + { + "id": "output_comment_75", + "type": "output", + "plugin": "comment", + "config": { + "text": "Kafka (example)" + } + }, + { + "id": "output_kafka_76", + "type": "output", + "plugin": "kafka", + "config": { + "id": "out_kafka", + "bootstrap_servers": "localhost:9092", + "topic_id": "logstashui-events" + }, + "comments": [] + }, + { + "id": "output_comment_77", + "type": "output", + "plugin": "comment", + "config": { + "text": "Pipeline-to-pipeline (requires another pipeline with pipeline input address => \"downstream\")" + } + }, + { + "id": "output_pipeline_78", + "type": "output", + "plugin": "pipeline", + "config": { + "id": "out_pipeline_downstream", + "send_to": [ + "downstream" + ] + }, + "comments": [] + } + ] +} diff --git a/tests/Common/unit/conversion_data/components/test_elasticdocs-conditional.json b/tests/Common/unit/conversion_data/components/test_elasticdocs-conditional.json new file mode 100644 index 0000000..39c72eb --- /dev/null +++ b/tests/Common/unit/conversion_data/components/test_elasticdocs-conditional.json @@ -0,0 +1,116 @@ +{ + "input": [ + { + "id": "input_file_0", + "type": "input", + "plugin": "file", + "config": { + "path": "/tmp/*_log" + }, + "comments": [] + } + ], + "filter": [ + { + "id": "filter_if_1", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[path] =~ \"access\"", + "plugins": [ + { + "id": "filter_mutate_2", + "type": "filter", + "plugin": "mutate", + "config": { + "replace": { + "type": "apache_access" + } + }, + "comments": [] + }, + { + "id": "filter_grok_3", + "type": "filter", + "plugin": "grok", + "config": { + "match": { + "message": "%{COMBINEDAPACHELOG}" + } + }, + "comments": [] + }, + { + "id": "filter_date_4", + "type": "filter", + "plugin": "date", + "config": { + "match": [ + "timestamp", + "dd/MMM/yyyy:HH:mm:ss Z" + ] + }, + "comments": [] + } + ], + "else_ifs": [ + { + "condition": "[path] =~ \"error\"", + "plugins": [ + { + "id": "filter_mutate_5", + "type": "filter", + "plugin": "mutate", + "config": { + "replace": { + "type": "apache_error" + } + }, + "comments": [] + } + ] + } + ], + "else": { + "plugins": [ + { + "id": "filter_mutate_6", + "type": "filter", + "plugin": "mutate", + "config": { + "replace": { + "type": "random_logs" + } + }, + "comments": [] + } + ] + } + } + } + ], + "output": [ + { + "id": "output_elasticsearch_7", + "type": "output", + "plugin": "elasticsearch", + "config": { + "hosts": [ + "localhost:9200" + ] + }, + "comments": [] + }, + { + "id": "output_stdout_8", + "type": "output", + "plugin": "stdout", + "config": { + "codec": { + "rubydebug": {} + } + }, + "comments": [] + } + ] +} diff --git a/tests/Common/unit/conversion_data/components/text-complex4.json b/tests/Common/unit/conversion_data/components/text-complex4.json new file mode 100644 index 0000000..da47a4e --- /dev/null +++ b/tests/Common/unit/conversion_data/components/text-complex4.json @@ -0,0 +1,386 @@ +{ + "input": [ + { + "id": "input_generator_0", + "type": "input", + "plugin": "generator", + "config": { + "id": "gen_edgeA", + "count": 1, + "lines": [ + "2026-02-22T01:23:45Z level=INFO service=api trace.id=abc123 method=GET path=\"/api/v2/items/42\" ip=8.8.8.8 ua=\"Mozilla/5.0 (Windows NT 10.0; Win64; x64)\" msg=\"hello\\world\"" + ], + "add_field": { + "[@metadata][source]": "generator", + "event.original": "%{message}" + } + }, + "comments": [] + } + ], + "filter": [ + { + "id": "filter_comment_1", + "type": "filter", + "plugin": "comment", + "config": { + "text": "Fast split: timestamp + remainder" + } + }, + { + "id": "filter_dissect_2", + "type": "filter", + "plugin": "dissect", + "config": { + "id": "dissect_ts_rest", + "mapping": { + "message": "%{ts} %{rest}" + }, + "tag_on_failure": [ + "_dissectfailure_ts_rest" + ] + }, + "comments": [] + }, + { + "id": "filter_date_3", + "type": "filter", + "plugin": "date", + "config": { + "id": "date_ts", + "match": [ + "ts", + "ISO8601" + ], + "tag_on_failure": [ + "_dateparsefailure_ts" + ] + }, + "comments": [] + }, + { + "id": "filter_comment_4", + "type": "filter", + "plugin": "comment", + "config": { + "text": "Parse key=value in rest" + } + }, + { + "id": "filter_kv_5", + "type": "filter", + "plugin": "kv", + "config": { + "id": "kv_rest", + "source": "rest", + "trim_key": " ", + "trim_value": " ", + "value_split": "=", + "field_split_pattern": "\\s+", + "include_brackets": "false", + "tag_on_failure": [ + "_kvfailure_rest" + ] + }, + "comments": [] + }, + { + "id": "filter_comment_6", + "type": "filter", + "plugin": "comment", + "config": { + "text": "Normalize: remove surrounding quotes on selected fields (common log format)" + } + }, + { + "id": "filter_mutate_7", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "mutate_strip_quotes", + "gsub": [ + "path", + "^\"|\"$", + "", + "ua", + "^\"|\"$", + "", + "msg", + "^\"|\"$", + "" + ] + }, + "comments": [] + }, + { + "id": "filter_comment_8", + "type": "filter", + "plugin": "comment", + "config": { + "text": "Promote a few fields into ECS-ish places" + } + }, + { + "id": "filter_mutate_9", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "mutate_promote", + "rename": { + "ip": "[source][ip]", + "ua": "[user_agent][original]", + "method": "[http][request][method]", + "path": "[url][path]", + "trace.id": "[trace][id]" + }, + "lowercase": [ + "level" + ] + }, + "comments": [] + }, + { + "id": "filter_comment_10", + "type": "filter", + "plugin": "comment", + "config": { + "text": "Route key uses nested refs in sprintf (great UI test)" + } + }, + { + "id": "filter_mutate_11", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "mutate_route_key", + "add_field": { + "route_key": "%{[@metadata][source]}::%{[service]}::%{[http][request][method]}::%{[url][path]}" + } + }, + "comments": [] + }, + { + "id": "filter_comment_12", + "type": "filter", + "plugin": "comment", + "config": { + "text": "Regex literals (escaped slashes)" + } + }, + { + "id": "filter_if_13", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[url][path] =~ /^\\/api\\/v2\\/items\\/[0-9]+$/", + "plugins": [ + { + "id": "filter_mutate_14", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "tag_items", + "add_tag": [ + "route_items" + ] + }, + "comments": [] + } + ], + "else_ifs": [ + { + "condition": "[url][path] =~ /^\\/api\\/v2\\/[A-Za-z0-9._-]+$/", + "plugins": [ + { + "id": "filter_mutate_15", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "tag_api_generic", + "add_tag": [ + "route_api_generic" + ] + }, + "comments": [] + } + ] + } + ], + "else": { + "plugins": [ + { + "id": "filter_mutate_16", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "tag_other", + "add_tag": [ + "route_other" + ] + }, + "comments": [] + } + ] + } + } + }, + { + "id": "filter_comment_17", + "type": "filter", + "plugin": "comment", + "config": { + "text": "useragent parsing" + } + }, + { + "id": "filter_if_18", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[user_agent][original]", + "plugins": [ + { + "id": "filter_useragent_19", + "type": "filter", + "plugin": "useragent", + "config": { + "id": "ua_parse", + "source": "[user_agent][original]", + "target": "[user_agent][parsed]" + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_comment_20", + "type": "filter", + "plugin": "comment", + "config": { + "text": "geoip on public source.ip" + } + }, + { + "id": "filter_if_21", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[source][ip]", + "plugins": [ + { + "id": "filter_geoip_22", + "type": "filter", + "plugin": "geoip", + "config": { + "id": "geoip_source", + "source": "[source][ip]", + "target": "[source][geo]" + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_comment_23", + "type": "filter", + "plugin": "comment", + "config": { + "text": "fingerprint based on nested refs + message" + } + }, + { + "id": "filter_fingerprint_24", + "type": "filter", + "plugin": "fingerprint", + "config": { + "id": "fp_event", + "source": [ + "route_key", + "message" + ], + "method": "MURMUR3", + "target": "[@metadata][fp]" + }, + "comments": [] + }, + { + "id": "filter_comment_25", + "type": "filter", + "plugin": "comment", + "config": { + "text": "Replace literal backslash with slash in msg (escape-heavy but valid)" + } + }, + { + "id": "filter_mutate_26", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "mutate_gsub_backslash", + "gsub": [ + "msg", + "\\\\", + "/" + ] + }, + "comments": [] + }, + { + "id": "filter_prune_27", + "type": "filter", + "plugin": "prune", + "config": { + "id": "prune_edgeA", + "whitelist_names": [ + "^@timestamp$", + "^message$", + "^tags$", + "^level$", + "^service$", + "^route_key$", + "^trace\\..*$", + "^http\\..*$", + "^url\\..*$", + "^source\\..*$", + "^user_agent\\..*$", + "^msg$", + "^@metadata\\..*$" + ] + }, + "comments": [] + } + ], + "output": [ + { + "id": "output_stdout_28", + "type": "output", + "plugin": "stdout", + "config": { + "codec": { + "rubydebug": { + "metadata": "true" + } + } + }, + "comments": [] + }, + { + "id": "output_file_29", + "type": "output", + "plugin": "file", + "config": { + "path": "/tmp/edgeA-%{+YYYY.MM.dd}.jsonl", + "codec": { + "json_lines": {} + } + }, + "comments": [] + } + ] +} diff --git a/tests/Common/unit/conversion_data/components/text-complex5.json b/tests/Common/unit/conversion_data/components/text-complex5.json new file mode 100644 index 0000000..1e75f10 --- /dev/null +++ b/tests/Common/unit/conversion_data/components/text-complex5.json @@ -0,0 +1,94 @@ +{ + "input": [ + { + "id": "input_generator_0", + "type": "input", + "plugin": "generator", + "config": { + "id": "gen_edgecase_3", + "count": 1, + "lines": [ + "path=\"C:\\Program Files\\App\\\" msg=\"quote:\" and backslash:\\\\\"" + ] + }, + "comments": [] + } + ], + "filter": [ + { + "id": "filter_comment_1", + "type": "filter", + "plugin": "comment", + "config": { + "text": "This ruby code contains both quote styles and backslashes." + } + }, + { + "id": "filter_ruby_2", + "type": "filter", + "plugin": "ruby", + "config": { + "id": "ruby_edgecase_3", + "code": "\n # Double quotes inside single-quoted LSCL string\n event.set(\"[edge][note]\", \"He said: \"hello\"\")\n # Single quote inside Ruby string\n event.set(\"[edge][apostrophe]\", \"it's fine\")\n # Trailing backslash in a field value (nasty for serializers)\n event.set(\"[edge][trail]\", \"C:\temp\")\n " + }, + "comments": [] + }, + { + "id": "filter_comment_3", + "type": "filter", + "plugin": "comment", + "config": { + "text": "Parse key=val" + } + }, + { + "id": "filter_kv_4", + "type": "filter", + "plugin": "kv", + "config": { + "id": "kv_edgecase_3", + "source": "message", + "value_split": "=", + "field_split_pattern": "\\s+" + }, + "comments": [] + }, + { + "id": "filter_comment_5", + "type": "filter", + "plugin": "comment", + "config": { + "text": "Replace literal backslash \"\\\" with \"/\"" + } + }, + { + "id": "filter_mutate_6", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "gsub_edgecase_3", + "gsub": [ + "path", + "\\\\", + "/" + ] + }, + "comments": [] + } + ], + "output": [ + { + "id": "output_stdout_7", + "type": "output", + "plugin": "stdout", + "config": { + "codec": { + "rubydebug": { + "metadata": "true" + } + } + }, + "comments": [] + } + ] +} diff --git a/tests/Common/unit/conversion_data/components/text-complex6.json b/tests/Common/unit/conversion_data/components/text-complex6.json new file mode 100644 index 0000000..429bb20 --- /dev/null +++ b/tests/Common/unit/conversion_data/components/text-complex6.json @@ -0,0 +1,469 @@ +{ + "input": [ + { + "id": "input_kafka_0", + "type": "input", + "plugin": "kafka", + "config": { + "id": "input_kafka_1", + "topics": [ + "critical_business_events", + "low_latency_metrics" + ], + "bootstrap_servers": "kafka1:9092,kafka2:9092", + "group_id": "logstash_critical_group", + "codec": { + "json_lines": {} + }, + "type": "business_event", + "tags": [ + "kafka_input", + "critical" + ], + "max_poll_records": "500" + }, + "comments": [] + }, + { + "id": "input_redis_1", + "type": "input", + "plugin": "redis", + "config": { + "id": "input_redis_1", + "host": "redis-cache.example.com", + "port": "6379", + "data_type": "list", + "key": "service_log_queue", + "type": "service_log", + "tags": [ + "redis_input", + "service_data" + ], + "codec": { + "plain": {} + } + }, + "comments": [] + }, + { + "id": "input_tcp_2", + "type": "input", + "plugin": "tcp", + "config": { + "id": "input_tcp_1", + "port": "9999", + "type": "audit_log", + "ssl_enable": "true", + "ssl_cert": "/etc/logstash/certs/logstash.crt", + "ssl_key": "/etc/logstash/certs/logstash.key", + "ssl_verify": "true", + "codec": { + "json": { + "delimiter": "\n" + } + }, + "tags": [ + "tcp_input", + "sensitive" + ] + }, + "comments": [] + } + ], + "filter": [ + { + "id": "filter_if_3", + "type": "filter", + "plugin": "if", + "config": { + "condition": "\"service_data\" in [tags]", + "plugins": [ + { + "id": "filter_json_4", + "type": "filter", + "plugin": "json", + "config": { + "id": "filter_json_1", + "source": "message", + "target": "parsed_service_log", + "remove_field": [ + "message" + ], + "add_tag": [ + "json_attempt" + ] + }, + "comments": [] + }, + { + "id": "filter_if_5", + "type": "filter", + "plugin": "if", + "config": { + "condition": "\"_jsonparsefailure\" in [tags]", + "plugins": [ + { + "id": "filter_grok_6", + "type": "filter", + "plugin": "grok", + "config": { + "id": "filter_grok_1", + "match": { + "message": "(?%{TIMESTAMP_ISO8601}) %{DATA:service_id} \\[%{LOGLEVEL:level}] %{NUMBER:req_id:int} - %{GREEDYDATA:log_msg}" + }, + "add_tag": [ + "grok_fallback_success" + ], + "remove_tag": [ + "_jsonparsefailure" + ] + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_if_7", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[parsed_service_log][sensitive_data] == true or [tags] =~ /_grokparsefailure/", + "plugins": [ + { + "id": "filter_mutate_8", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_1", + "gsub": [ + "message", + "(\\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Z|a-z]{2,}\\b)", + "email_masked" + ] + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_mutate_9", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_2", + "rename": { + "[parsed_service_log][level]": "log_level" + }, + "add_field": { + "correlation_id": "%{[parsed_service_log][request_id]}" + } + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_if_10", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[correlation_id] and [log_level]", + "plugins": [ + { + "id": "filter_aggregate_11", + "type": "filter", + "plugin": "aggregate", + "config": { + "id": "filter_aggregate_1", + "task_id": "%{correlation_id}", + "code": "\n if event.get('log_level') == 'START'\n map['start_time'] = event.get('@timestamp').time.to_f\n map['service'] = event.get('service_id')\n event.cancel\n elsif event.get('log_level') == 'END' and map['start_time']\n end_time = event.get('@timestamp').time.to_f\n duration = (end_time - map['start_time']) * 1000 # Duration in ms\n event.set('request_duration_ms', duration.round(3))\n event.set('service_name', map['service'])\n event.set('type', 'request_summary')\n end\n ", + "map_action": "create_or_update", + "push_map_as_event_on_timeout": "true", + "timeout": "60", + "timeout_code": "event.set('error_reason', 'Unmatched_START_Event')", + "timeout_task_id_field": "unmatched_correlation_id" + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_if_12", + "type": "filter", + "plugin": "if", + "config": { + "condition": "\"critical\" in [tags] or [type] == \"audit_log\"", + "plugins": [ + { + "id": "filter_translate_13", + "type": "filter", + "plugin": "translate", + "config": { + "id": "filter_translate_1", + "field": "tenant_id", + "destination": "tenant_name", + "dictionary_path": "/etc/logstash/dicts/tenant_map.yml", + "fallback": "Unknown_Tenant", + "refresh_interval": "600" + }, + "comments": [] + }, + { + "id": "filter_mutate_14", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_3", + "convert": { + "transaction_amount": "float" + }, + "remove_field": [ + "host", + "port" + ] + }, + "comments": [] + }, + { + "id": "filter_date_15", + "type": "filter", + "plugin": "date", + "config": { + "id": "filter_date_1", + "match": [ + "[event_time]", + "ISO8601", + "UNIX_MS" + ], + "target": "@timestamp", + "remove_tag": [ + "_dateparsefailure" + ] + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_if_16", + "type": "filter", + "plugin": "if", + "config": { + "condition": "(\"_grokparsefailure\" in [tags] or \"_jsonparsefailure\" in [tags]) and [type] != \"audit_log\"", + "plugins": [ + { + "id": "filter_mutate_17", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_4", + "add_tag": [ + "dlq_candidate", + "parsing_error" + ], + "add_field": { + "dlq_reason": "Parsing_Failed" + } + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_mutate_18", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_5", + "remove_tag": [ + "_jsonparsefailure", + "_grokparsefailure", + "_dateparsefailure" + ] + }, + "comments": [] + } + ], + "output": [ + { + "id": "output_if_19", + "type": "output", + "plugin": "if", + "config": { + "condition": "[tenant_name] =~ /^PRIORITY_/", + "plugins": [ + { + "id": "output_elasticsearch_20", + "type": "output", + "plugin": "elasticsearch", + "config": { + "id": "output_elasticsearch_1", + "hosts": [ + "https://es-priority:9200" + ], + "index": "tenant_priority-%{tenant_name}-%{+YYYY.MM}", + "workers": "1", + "manage_template": "false" + }, + "comments": [] + } + ], + "else_ifs": [ + { + "condition": "[tenant_name]", + "plugins": [ + { + "id": "output_elasticsearch_21", + "type": "output", + "plugin": "elasticsearch", + "config": { + "id": "output_elasticsearch_2", + "hosts": [ + "https://es-main:9200" + ], + "index": "tenant_general-%{+YYYY.MM.dd}", + "dlq_enabled": "true", + "dlq_path": "/var/lib/logstash/dlq" + }, + "comments": [] + } + ] + } + ], + "else": null + } + }, + { + "id": "output_if_22", + "type": "output", + "plugin": "if", + "config": { + "condition": "\"dlq_candidate\" in [tags]", + "plugins": [ + { + "id": "output_file_23", + "type": "output", + "plugin": "file", + "config": { + "id": "output_file_1", + "path": "/var/log/logstash/error_logs/dlq_parsing_failures.log", + "codec": { + "json_lines": { + "target": "original_event" + } + }, + "add_tag": [ + "s3_backup" + ] + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "output_if_24", + "type": "output", + "plugin": "if", + "config": { + "condition": "[type] == \"audit_log\" or [type] == \"request_summary\" or \"s3_backup\" in [tags]", + "plugins": [ + { + "id": "output_s3_25", + "type": "output", + "plugin": "s3", + "config": { + "id": "output_s3_1", + "bucket": "logstash-archive-bucket", + "region": "us-west-2", + "time_file": "15", + "size_file": "50", + "codec": { + "json_lines": {} + }, + "temporary_directory": "/tmp/logstash_s3_tmp" + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "output_if_26", + "type": "output", + "plugin": "if", + "config": { + "condition": "\"unmatched_correlation_id\" in [tags]", + "plugins": [ + { + "id": "output_tcp_27", + "type": "output", + "plugin": "tcp", + "config": { + "id": "output_tcp_1", + "host": "graylog-server.example.com", + "port": "12201", + "codec": { + "gelf": { + "level": 1, + "short_message": "Log Aggregation Timeout/Error: %{unmatched_correlation_id}" + } + } + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "output_if_28", + "type": "output", + "plugin": "if", + "config": { + "condition": "[log_level] =~ /(START|END|FATAL)/", + "plugins": [ + { + "id": "output_stdout_29", + "type": "output", + "plugin": "stdout", + "config": { + "id": "output_stdout_1", + "codec": { + "rubydebug": { + "metadata": "true" + } + } + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + } + ] +} diff --git a/tests/Common/unit/conversion_data/components/text-complex7.json b/tests/Common/unit/conversion_data/components/text-complex7.json new file mode 100644 index 0000000..60bdd43 --- /dev/null +++ b/tests/Common/unit/conversion_data/components/text-complex7.json @@ -0,0 +1,558 @@ +{ + "input": [ + { + "id": "input_beats_0", + "type": "input", + "plugin": "beats", + "config": { + "id": "input_beats_1", + "port": "5044", + "ssl": "true", + "ssl_certificate": "/etc/logstash/certs/logstash.crt", + "ssl_key": "/etc/logstash/certs/logstash.key", + "codec": { + "json": {} + }, + "tags": [ + "beats_input", + "app_log" + ] + }, + "comments": [] + }, + { + "id": "input_udp_1", + "type": "input", + "plugin": "udp", + "config": { + "id": "input_udp_1", + "port": "5140", + "buffer_size": "8192", + "codec": { + "plain": { + "charset": "UTF-8" + } + }, + "type": "network_flow", + "tags": [ + "udp_input", + "unstructured" + ] + }, + "comments": [] + }, + { + "id": "input_jdbc_2", + "type": "input", + "plugin": "jdbc", + "config": { + "id": "input_jdbc_1", + "jdbc_driver_library": "/usr/share/logstash/logstash-core/lib/jars/postgresql-42.2.8.jar", + "jdbc_driver_class": "org.postgresql.Driver", + "jdbc_connection_string": "jdbc:postgresql://db.example.com:5432/config_db", + "jdbc_user": "logstash_user", + "jdbc_password": "${JDBC_PASSWORD}", + "schedule": "0 * * * *", + "statement": "SELECT id, user_name, config_item, change_timestamp FROM config_changes WHERE change_timestamp > :sql_last_value ORDER BY change_timestamp ASC", + "use_column_value": "true", + "tracking_column": "change_timestamp", + "tracking_column_type": "timestamp", + "last_run_metadata_path": "/var/lib/logstash/.jdbc_last_run_config_db", + "type": "config_audit", + "tags": [ + "jdbc_input", + "audit" + ] + }, + "comments": [] + } + ], + "filter": [ + { + "id": "filter_mutate_3", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_1", + "rename": { + "@timestamp": "log_recv_time" + }, + "add_field": { + "severity": "INFO" + } + }, + "comments": [] + }, + { + "id": "filter_if_4", + "type": "filter", + "plugin": "if", + "config": { + "condition": "\"app_log\" in [tags]", + "plugins": [ + { + "id": "filter_if_5", + "type": "filter", + "plugin": "if", + "config": { + "condition": "\"_jsonparsefailure\" in [tags]", + "plugins": [ + { + "id": "filter_grok_6", + "type": "filter", + "plugin": "grok", + "config": { + "id": "filter_grok_1", + "match": { + "message": "(?\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}\\.\\d{3}) \\[%{DATA:thread}] %{LOGLEVEL:log_level} %{DATA:logger} - %{GREEDYDATA:log_message}" + }, + "add_tag": [ + "grok_fallback_success" + ], + "remove_tag": [ + "_jsonparsefailure" + ] + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_if_7", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[log_level]", + "plugins": [ + { + "id": "filter_mutate_8", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_2", + "uppercase": [ + "log_level" + ], + "copy": { + "log_level": "severity" + } + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_if_9", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[severity] =~ /(WARNING|ERROR|FATAL)/", + "plugins": [ + { + "id": "filter_geoip_10", + "type": "filter", + "plugin": "geoip", + "config": { + "id": "filter_geoip_1", + "source": "[fields][source_ip]", + "target": "geo", + "database": "/etc/logstash/geoip/GeoLite2-City.mmdb", + "remove_field": [ + "continent_code", + "location" + ] + }, + "comments": [] + }, + { + "id": "filter_translate_11", + "type": "filter", + "plugin": "translate", + "config": { + "id": "filter_translate_1", + "field": "[service_code]", + "destination": "service_name", + "dictionary_path": "/etc/logstash/dictionaries/service_codes.csv", + "fallback": "Unknown Service", + "refresh_interval": "300" + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "filter_mutate_12", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_3", + "remove_field": [ + "message", + "agent" + ] + }, + "comments": [] + } + ], + "else_ifs": [ + { + "condition": "[type] == \"network_flow\"", + "plugins": [ + { + "id": "filter_grok_13", + "type": "filter", + "plugin": "grok", + "config": { + "id": "filter_grok_2", + "match": { + "message": "%{NETFLOW_V9}" + }, + "on_failure": [ + "_netflowparsefailure" + ] + }, + "comments": [] + }, + { + "id": "filter_if_14", + "type": "filter", + "plugin": "if", + "config": { + "condition": "\"_netflowparsefailure\" in [tags]", + "plugins": [ + { + "id": "filter_mutate_15", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_4", + "add_tag": [ + "unparsed_flow" + ], + "remove_tag": [ + "_grokparsefailure", + "_netflowparsefailure" + ], + "copy": { + "message": "unparsed_data" + }, + "replace": { + "message": "Truncated unparsed flow data." + } + }, + "comments": [] + } + ], + "else_ifs": [], + "else": { + "plugins": [ + { + "id": "filter_aggregate_16", + "type": "filter", + "plugin": "aggregate", + "config": { + "id": "filter_aggregate_1", + "task_id": "%{source_ip}_%{destination_ip}_%{protocol}", + "code": "map['total_packets'] ||= 0; map['total_packets'] += event.get('packets').to_i; map['total_bytes'] ||= 0; map['total_bytes'] += event.get('bytes').to_i", + "map_action": "create_or_update", + "push_map_as_event_on_timeout": "true", + "timeout": "120", + "timeout_task_id_field": "aggregated_flow_id", + "timeout_tags": [ + "_aggregate_timeout" + ] + }, + "comments": [] + } + ] + } + } + } + ] + }, + { + "condition": "[type] == \"config_audit\"", + "plugins": [ + { + "id": "filter_if_17", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[change_timestamp]", + "plugins": [ + { + "id": "filter_date_18", + "type": "filter", + "plugin": "date", + "config": { + "id": "filter_date_1", + "match": [ + "change_timestamp", + "YYYY-MM-dd HH:mm:ss.SSSSSS" + ], + "target": "@timestamp", + "remove_field": [ + "change_timestamp" + ] + }, + "comments": [] + }, + { + "id": "filter_if_19", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[user_name] != \"system\"", + "plugins": [ + { + "id": "filter_ruby_20", + "type": "filter", + "plugin": "ruby", + "config": { + "id": "filter_ruby_1", + "code": "event.set('user_hash', Digest::MD5.hexdigest(event.get('user_name')))" + }, + "comments": [] + }, + { + "id": "filter_mutate_21", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_5", + "remove_field": [ + "user_name" + ] + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + } + ], + "else_ifs": [], + "else": null + } + } + ] + } + ], + "else": null + } + }, + { + "id": "filter_if_22", + "type": "filter", + "plugin": "if", + "config": { + "condition": "\"_grokparsefailure\" in [tags] or \"_jsonparsefailure\" in [tags]", + "plugins": [ + { + "id": "filter_mutate_23", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_6", + "add_field": { + "log_status": "FAILED_TO_PARSE" + } + }, + "comments": [] + } + ], + "else_ifs": [], + "else": { + "plugins": [ + { + "id": "filter_mutate_24", + "type": "filter", + "plugin": "mutate", + "config": { + "id": "filter_mutate_7", + "add_field": { + "log_status": "PROCESSED" + } + }, + "comments": [] + } + ] + } + } + }, + { + "id": "filter_if_25", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[log_recv_time] < now() - 86400000", + "plugins": [ + { + "id": "filter_drop_26", + "type": "filter", + "plugin": "drop", + "config": { + "id": "filter_drop_1" + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + } + ], + "output": [ + { + "id": "output_if_27", + "type": "output", + "plugin": "if", + "config": { + "condition": "[log_status] == \"PROCESSED\" and [severity] =~ /(ERROR|FATAL)/", + "plugins": [ + { + "id": "output_elasticsearch_28", + "type": "output", + "plugin": "elasticsearch", + "config": { + "id": "output_elasticsearch_1", + "hosts": [ + "https://es-hot.example.com:9200" + ], + "index": "high-priority-%{+YYYY.MM.dd}", + "user": "logstash_writer", + "password": "secure_password", + "ssl": "true", + "cacert": "/etc/logstash/certs/ca.crt", + "action": "index" + }, + "comments": [] + } + ], + "else_ifs": [ + { + "condition": "[log_status] == \"PROCESSED\"", + "plugins": [ + { + "id": "output_elasticsearch_29", + "type": "output", + "plugin": "elasticsearch", + "config": { + "id": "output_elasticsearch_2", + "hosts": [ + "https://es-warm.example.com:9200" + ], + "index": "general-logs-%{+YYYY.MM.dd}", + "user": "logstash_writer", + "password": "secure_password", + "ssl": "true", + "cacert": "/etc/logstash/certs/ca.crt", + "workers": "4", + "ilm_enabled": "false" + }, + "comments": [] + } + ] + } + ], + "else": null + } + }, + { + "id": "output_if_30", + "type": "output", + "plugin": "if", + "config": { + "condition": "[log_status] == \"FAILED_TO_PARSE\"", + "plugins": [ + { + "id": "output_file_31", + "type": "output", + "plugin": "file", + "config": { + "id": "output_file_1", + "path": "/var/log/logstash/dlq_failures.json", + "codec": { + "json": { + "pretty": "true" + } + }, + "add_tag": [ + "dlq_routed" + ] + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "output_if_32", + "type": "output", + "plugin": "if", + "config": { + "condition": "\"_aggregate_timeout\" in [tags]", + "plugins": [ + { + "id": "output_tcp_33", + "type": "output", + "plugin": "tcp", + "config": { + "id": "output_tcp_1", + "host": "alert-sys.example.com", + "port": "6514", + "codec": { + "gelf": { + "protocol": "TCP", + "short_message": "Aggregated Flow Timeout: %{aggregated_flow_id}" + } + }, + "socket_timeout": "5" + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + }, + { + "id": "output_if_34", + "type": "output", + "plugin": "if", + "config": { + "condition": "rand(100) < 1", + "plugins": [ + { + "id": "output_stdout_35", + "type": "output", + "plugin": "stdout", + "config": { + "id": "output_stdout_1", + "codec": { + "rubydebug": { + "metadata": "true" + } + } + }, + "comments": [] + } + ], + "else_ifs": [], + "else": null + } + } + ] +} diff --git a/tests/Common/unit/conversion_data/components/text-ls-repo-nginx-error.json b/tests/Common/unit/conversion_data/components/text-ls-repo-nginx-error.json new file mode 100644 index 0000000..b247032 --- /dev/null +++ b/tests/Common/unit/conversion_data/components/text-ls-repo-nginx-error.json @@ -0,0 +1,183 @@ +{ + "input": [ + { + "id": "input_stdin_0", + "type": "input", + "plugin": "stdin", + "config": { + "codec": { + "line": {} + } + }, + "comments": [] + } + ], + "filter": [ + { + "id": "filter_mutate_1", + "type": "filter", + "plugin": "mutate", + "config": { + "add_field": { + "event.dataset": "nginx.access", + "service.name": "nginx" + } + }, + "comments": [] + }, + { + "id": "filter_grok_2", + "type": "filter", + "plugin": "grok", + "config": { + "match": { + "message": [ + "%{IPORHOST:source.address} %{DATA:nginx.ident} %{DATA:user.name} \\[%{HTTPDATE:nginx.access.time}\\] \"%{WORD:http.request.method} %{DATA:url.original} HTTP/%{NUMBER:http.version}\" %{NUMBER:http.response.status_code:int} (?:%{NUMBER:http.response.body.bytes:int}|-) \"%{DATA:http.request.referrer}\" \"%{DATA:user_agent.original}\" %{NUMBER:nginx.access.request_time:float}", + "%{IPORHOST:source.address} %{DATA:nginx.ident} %{DATA:user.name} \\[%{HTTPDATE:nginx.access.time}\\] \"%{WORD:http.request.method} %{DATA:url.original} HTTP/%{NUMBER:http.version}\" %{NUMBER:http.response.status_code:int} (?:%{NUMBER:http.response.body.bytes:int}|-) \"%{DATA:http.request.referrer}\" \"%{DATA:user_agent.original}\"", + "%{IPORHOST:source.address} %{DATA:nginx.ident} %{DATA:user.name} \\[%{HTTPDATE:nginx.access.time}\\] \"%{WORD:http.request.method} %{DATA:url.original} HTTP/%{NUMBER:http.version}\" %{NUMBER:http.response.status_code:int} (?:%{NUMBER:http.response.body.bytes:int}|-) \"%{DATA:http.request.referrer}\" \"%{DATA:user_agent.original}\" (?:rt=%{NUMBER:nginx.access.request_time:float}\\s+uct=%{NUMBER:nginx.access.upstream_connect_time:float}\\s+uht=%{NUMBER:nginx.access.upstream_header_time:float}\\s+urt=%{NUMBER:nginx.access.upstream_response_time:float})" + ] + }, + "tag_on_failure": [ + "_grok_nginx_access_fail" + ] + }, + "comments": [] + }, + { + "id": "filter_date_3", + "type": "filter", + "plugin": "date", + "config": { + "match": [ + "nginx.access.time", + "dd/MMM/yyyy:HH:mm:ss Z" + ], + "target": "@timestamp" + }, + "comments": [] + }, + { + "id": "filter_urldecode_4", + "type": "filter", + "plugin": "urldecode", + "config": { + "field": "url.original" + }, + "comments": [] + }, + { + "id": "filter_dissect_5", + "type": "filter", + "plugin": "dissect", + "config": { + "mapping": { + "url.original": "%{url.path}?%{url.query}" + } + }, + "comments": [] + }, + { + "id": "filter_useragent_6", + "type": "filter", + "plugin": "useragent", + "config": { + "source": "user_agent.original", + "target": "user_agent" + }, + "comments": [] + }, + { + "id": "filter_mutate_7", + "type": "filter", + "plugin": "mutate", + "config": { + "copy": { + "source.address": "source.ip" + } + }, + "comments": [] + }, + { + "id": "filter_geoip_8", + "type": "filter", + "plugin": "geoip", + "config": { + "source": "source.ip", + "target": "source.geo", + "tag_on_failure": [ + "_geoip_fail" + ] + }, + "comments": [] + }, + { + "id": "filter_mutate_9", + "type": "filter", + "plugin": "mutate", + "config": { + "gsub": [ + "http.request.referrer", + "^-$", + "", + "user.name", + "^-$", + "" + ] + }, + "comments": [] + }, + { + "id": "filter_if_10", + "type": "filter", + "plugin": "if", + "config": { + "condition": "[http][response][status_code] >= 500", + "plugins": [ + { + "id": "filter_mutate_11", + "type": "filter", + "plugin": "mutate", + "config": { + "add_tag": [ + "nginx_server_error" + ] + }, + "comments": [] + } + ], + "else_ifs": [ + { + "condition": "[http][response][status_code] >= 400", + "plugins": [ + { + "id": "filter_mutate_12", + "type": "filter", + "plugin": "mutate", + "config": { + "add_tag": [ + "nginx_client_error" + ] + }, + "comments": [] + } + ] + } + ], + "else": null + } + } + ], + "output": [ + { + "id": "output_stdout_13", + "type": "output", + "plugin": "stdout", + "config": { + "codec": { + "rubydebug": {} + } + }, + "comments": [] + } + ] +} diff --git a/tests/Common/unit/conversion_data/pipelines/ls-repo-apache2.conf b/tests/Common/unit/conversion_data/pipelines/ls-repo-apache2.conf new file mode 100644 index 0000000..bf0eb1d --- /dev/null +++ b/tests/Common/unit/conversion_data/pipelines/ls-repo-apache2.conf @@ -0,0 +1,69 @@ +input { + beats { + port => 5044 + host => "0.0.0.0" + } +} +filter { + if [fileset][module] == "apache2" { + if [fileset][name] == "access" { + grok { + match => { + "message" => [ + '%{IPORHOST:[apache2][access][remote_ip]} - %{DATA:[apache2][access][user_name]} \[%{HTTPDATE:[apache2][access][time]}\] "%{WORD:[apache2][access][method]} %{DATA:[apache2][access][url]} HTTP/%{NUMBER:[apache2][access][http_version]}" %{NUMBER:[apache2][access][response_code]} %{NUMBER:[apache2][access][body_sent][bytes]}( "%{DATA:[apache2][access][referrer]}")?( "%{DATA:[apache2][access][agent]}")?', + '%{IPORHOST:[apache2][access][remote_ip]} - %{DATA:[apache2][access][user_name]} \[%{HTTPDATE:[apache2][access][time]}\] "-" %{NUMBER:[apache2][access][response_code]} -' + ] + } + remove_field => "message" + } + mutate { + add_field => { + "read_timestamp" => "%{@timestamp}" + } + } + date { + match => ["[apache2][access][time]", "dd/MMM/YYYY:H:m:s Z"] + remove_field => "[apache2][access][time]" + } + useragent { + source => "[apache2][access][agent]" + target => "[apache2][access][user_agent]" + remove_field => "[apache2][access][agent]" + } + geoip { + source => "[apache2][access][remote_ip]" + target => "[apache2][access][geoip]" + } + } + else if [fileset][name] == "error" { + grok { + match => { + "message" => [ + "\[%{APACHE_TIME:[apache2][error][timestamp]}\] \[%{LOGLEVEL:[apache2][error][level]}\]( \[client %{IPORHOST:[apache2][error][client]}\])? %{GREEDYDATA:[apache2][error][message]}", + "\[%{APACHE_TIME:[apache2][error][timestamp]}\] \[%{DATA:[apache2][error][module]}:%{LOGLEVEL:[apache2][error][level]}\] \[pid %{NUMBER:[apache2][error][pid]}(:tid %{NUMBER:[apache2][error][tid]})?\]( \[client %{IPORHOST:[apache2][error][client]}\])? %{GREEDYDATA:[apache2][error][message1]}" + ] + } + pattern_definitions => { + "APACHE_TIME" => "%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR}" + } + remove_field => "message" + } + mutate { + rename => { + "[apache2][error][message1]" => "[apache2][error][message]" + } + } + date { + match => ["[apache2][error][timestamp]", "EEE MMM dd H:m:s YYYY", "EEE MMM dd H:m:s.SSSSSS YYYY"] + remove_field => "[apache2][error][timestamp]" + } + } + } +} +output { + elasticsearch { + hosts => "localhost" + manage_template => "false" + index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" + } +} diff --git a/tests/Common/unit/conversion_data/pipelines/test-asa-new.conf b/tests/Common/unit/conversion_data/pipelines/test-asa-new.conf new file mode 100644 index 0000000..dbd9def --- /dev/null +++ b/tests/Common/unit/conversion_data/pipelines/test-asa-new.conf @@ -0,0 +1,897 @@ +input { + udp { + id => "input_udp_1" + port => "5119" + } + cloudwatch { + } +} +filter { + mutate { + id => "filter_mutate_1" + rename => { + "message" => "log.original" + "host" => "observer.ip" + } + copy => { + "host" => "sysloghost" + } + } + grok { + id => "filter_grok_1" + match => { + "log.original" => [ + "%{CISCO_TAGGED_SYSLOG} %{GREEDYDATA:message}", + "^<%{POSINT:syslog_pri}>%{DATA}: %%{DATA:ciscotag}: %{GREEDYDATA:message}", + "^<%{POSINT:syslog_pri}>%%{DATA:ciscotag}: %{GREEDYDATA:message}" + ] + } + } + grok { + id => "filter_grok_2" + match => { + "ciscotag" => [ + "%{WORD}-%{INT:event.severity}-%{INT:event.code}", + "%{WORD}-%{WORD}-%{INT:event.severity}-%{INT:event.code}" + ] + } + } + mutate { + id => "filter_mutate_2" + add_field => { + "event.action" => "firewall-rule" + } + } + if [event.code] == "105012" { + grok { + id => "filter_grok_3" + match => { + "message" => [ + "Teardown dynamic %{WORD:network.transport} translation from %{DATA:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} to %{DATA:cisco.destination_interface}:%{IP:destination.address}/%{INT:destination.port} duration %{DATA:cisco.duration_hms}$" + ] + } + } + } + else if [event.code] == "106001" { + dissect { + id => "filter_dissect_1" + mapping => { + "message" => "%{network.direction} %{network.transport} connection %{event.outcome} from %{source.address}/%{source.port} to %{destination.address}/%{destination.port} flags %{} on interface %{source_interface}" + } + } + mutate { + id => "filter_mutate_3" + add_field => { + "event.category" => "network_traffic" + } + } + } + else if [event.code] == "106002" { + dissect { + id => "filter_dissect_2" + mapping => { + "message" => "%{network.transport} Connection %{event.outcome} by %{network.direction} list %{cisco.list_id} src %{source.address} dest %{destination.address}" + } + } + mutate { + id => "filter_mutate_4" + add_field => { + "event.category" => "network_traffic" + } + } + } + else if [event.code] == "106006" { + dissect { + id => "filter_dissect_3" + mapping => { + "message" => "%{event.outcome} %{network.direction} %{network.transport} from %{source.address}/%{source.port} to %{destination.address}/%{destination.port} on interface %{cisco.source_interface}" + } + } + mutate { + id => "filter_mutate_5" + add_field => { + "event.category" => "network_traffic" + } + } + } + else if [event.code] == "106007" { + dissect { + id => "filter_dissect_4" + mapping => { + "message" => "%{event.outcome} %{network.direction} %{network.transport} from %{source.address}/%{source.port} to %{destination.address}/%{destination.port} due to %{network.protocol} %{}" + } + } + mutate { + id => "filter_mutate_6" + add_field => { + "event.category" => "network_traffic" + } + } + } + else if [event.code] == "106010" { + dissect { + id => "filter_dissect_5" + mapping => { + "message" => "%{event.outcome} %{network.direction} %{network.transport} src %{cisco.source_interface}:%{source.address}/%{source.port} %{} dst %{cisco.destination_interface}:%{destination.address}/%{destination.port} %{}" + } + } + mutate { + id => "filter_mutate_7" + add_field => { + "event.category" => "network_traffic" + } + } + } + else if [event.code] == "106013" { + dissect { + id => "filter_dissect_6" + mapping => { + "message" => "Dropping echo request from %{source.address} to PAT address %{destination.address}" + } + } + mutate { + id => "filter_mutate_8" + add_field => { + "network.transport" => "icmp" + "network.direction" => "inbound" + } + } + } + else if [event.code] == "106014" { + dissect { + id => "filter_dissect_7" + mapping => { + "message" => "%{event.outcome} %{network.direction} %{network.transport} src %{cisco.source_interface}:%{source.address} %{}dst %{cisco.destination_interface}:%{destination.address} %{}" + } + } + mutate { + id => "filter_mutate_9" + add_field => { + "event.category" => "network_traffic" + } + } + } + else if [event.code] == "106015" { + dissect { + id => "filter_dissect_8" + mapping => { + "message" => "%{event.outcome} %{network.transport} (no connection) from %{source.address}/%{source.port} to %{destination.address}/%{destination.port} flags %{} on interface %{cisco.source_interface}" + } + } + mutate { + id => "filter_mutate_10" + add_field => { + "event.category" => "nat_translation" + } + } + } + else if [event.code] == "106016" { + dissect { + id => "filter_dissect_9" + mapping => { + "message" => "%{event.outcome} IP spoof from (%{source.address}) to %{destination.address} on interface %{cisco.source_interface}" + } + } + } + else if [event.code] == "106017" { + dissect { + id => "filter_dissect_10" + mapping => { + "message" => "%{event.outcome} IP due to Land Attack from %{source.address} to %{destination.address}" + } + } + } + else if [event.code] == "106018" { + dissect { + id => "filter_dissect_11" + mapping => { + "message" => "%{network.transport} packet type %{cisco.icmp_type} %{event.outcome} by %{network.direction} list %{cisco.list_id} src %{source.address} dest %{destination.address}" + } + } + } + else if [event.code] == "106020" { + dissect { + id => "filter_dissect_12" + mapping => { + "message" => "%{event.outcome} IP teardrop fragment (size = %{}, offset = %{}) from %{source.address} to %{destination.address}" + } + } + } + else if [event.code] == "106021" { + dissect { + id => "filter_dissect_13" + mapping => { + "message" => "%{event.outcome} %{network.transport} reverse path check from %{source.address} to %{destination.address} on interface %{cisco.source_interface}" + } + } + } + else if [event.code] == "106022" { + dissect { + id => "filter_dissect_14" + mapping => { + "message" => "%{event.outcome} %{network.transport} connection spoof from %{source.address} to %{destination.address} on interface %{cisco.source_interface}" + } + } + } + else if [event.code] == "106023" { + grok { + id => "filter_grok_4" + match => { + "message" => [ + '%{WORD:event.outcome} %{WORD:network.transport} src %{WORD:cisco.source_interface}:%{IPORHOST:source.address}?(/%{INT:source.port}) dst %{WORD:cisco.destination_interface}:%{IPORHOST:destination.address}?(/%{INT:destination.port}) by access-group "%{DATA:cisco.list_id}"', + '%{WORD:event.outcome} %{WORD:network.transport} src %{WORD:cisco.source_interface}:%{IPORHOST:source.address} dst %{WORD:destination.direction}:%{IPORHOST:destination.address} \(%{DATA}\) by access-group "%{DATA:cisco.list_id}"', + '%{WORD:event.outcome} %{WORD:network.transport} src %{WORD:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} dst %{WORD:cisco.destination.interface}:%{IPORHOST:destination.address}/%{INT:destination.port} by access-group "%{DATA:cisco.list_id}"' + ] + } + } + mutate { + id => "filter_mutate_11" + add_field => { + "event.category" => "network_traffic" + } + } + } + else if [event.code] == "106027" { + dissect { + id => "filter_dissect_15" + mapping => { + "message" => '%{} %{event.outcome} src %{source.address} dst %{destination.address} by access-group "%{cisco.list_id}"%{}' + } + } + } + else if [event.code] == "106100" { + dissect { + id => "filter_dissect_16" + mapping => { + "message" => "access-list %{cisco.list_id} %{event.outcome} %{network.transport} %{cisco.source_interface}/%{source.address}(%{source.port}) -> %{cisco.destination_interface}/%{destination.address}(%{destination.port}) %{}" + } + } + } + else if [event.code] == "106102" { + dissect { + id => "filter_dissect_17" + mapping => { + "message" => "access-list %{cisco.list_id} %{event.outcome} %{network.transport} for user %{cisco.username} %{cisco.source_interface}/%{source.address} %{source.port} %{cisco.destination_interface}/%{destination.address} %{destination.port} %{}" + } + } + } + else if [event.code] == "106103" { + dissect { + id => "filter_dissect_18" + mapping => { + "message" => "access-list %{cisco.list_id} %{event.outcome} %{network.transport} for user %{cisco.username} %{cisco.source_interface}/%{source.address} %{source.port} %{cisco.destination_interface}/%{destination.address} %{destination.port} %{}" + } + } + } + else if [event.code] == "113004" { + grok { + id => "filter_grok_5" + match => { + "message" => [ + "AAA user accounting %{WORD:cisco.auth_outcome} : server =%{SPACE}%{IP:source.address} : user =%{SPACE}%{DATA:source.user.name}$" + ] + } + } + mutate { + id => "filter_mutate_12" + add_field => { + "event.category" => "authentication" + } + } + if [cisco.auth_outcome] == "Successful" { + mutate { + id => "filter_mutate_13" + add_field => { + "event.action" => "authentication_success" + } + } + } + else { + mutate { + id => "filter_mutate_14" + add_field => { + "event.action" => "authentication_failure" + } + } + } + } + else if [event.code] == "302015" or [event.code] == "302013" { + grok { + id => "filter_grok_6" + match => { + "message" => [ + "Built %{WORD:network.direction} %{WORD:network.transport} connection %{INT} for %{DATA:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} \(%{IP}|\) to %{DATA:cisco.destination_interface}:%{IPORHOST:destination.address}/%{INT:destination.port} \(%{DATA}\)", + "Built %{WORD:network.direction} %{WORD:network.transport} connection %{INT} for %{DATA:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} \(%{DATA}\)?(\(%{DATA:cisco.source_username}\)) to %{DATA:cisco.destination_interface}:%{IPORHOST:destination.address}/%{INT:destination.port} \(%{DATA}\) ?(\(%{DATA:cisco.username}\))", + "Built %{WORD:network.direction} %{WORD:network.transport} connection %{INT:cisco.connection_id} for %{WORD:cisco.source_interface}:%{IPORHOST:source.address}\/%{INT:source.port} \(%{DATA}\) to %{WORD:cisco.destination_interface}:%{IPORHOST:destination.address}/%{INT:destination.port}" + ] + } + } + mutate { + id => "filter_mutate_15" + add_field => { + "event.category" => "nat_translation" + } + } + } + else if [event.code] == "110003" { + grok { + id => "filter_grok_7" + match => { + "message" => [ + "%{DATA:cisco.event_error} for %{WORD:network.transport} from %{DATA:cisco.source_interface}:%{IP:source.address}\/%{INT:source.port} to %{DATA:cisco.destination_interface}:%{IP:destination.address}\/%{INT:destination.port}" + ] + } + } + mutate { + id => "filter_mutate_16" + add_field => { + "event.category" => "error" + } + } + } + else if [event.code] == "113019" { + grok { + id => "filter_grok_8" + match => { + "message" => [ + "Group = %{DATA:cisco.group}, Username = %{DATA:user.name}, IP = %{IP:cisco.client_vpn_ip}, %{DATA:cisco.client_vpn_action}\. Session Type: %{DATA:cisco.session_type}, Duration: %{DATA:cisco.duration}, Bytes xmt: %{INT:cisco.vpn_transmit_byte_summary}, Bytes rcv: %{INT:cisco.vpn_receive_byte_summary}, Reason: %{DATA:cisco.client_vpn_outcome}$" + ] + } + } + } + else if [event.code] == "304001" { + dissect { + id => "filter_dissect_19" + mapping => { + "message" => "%{source.address} %{}ccessed URL %{destination.address}:%{url.original}" + } + } + mutate { + id => "filter_mutate_17" + add_field => { + "event.outcome" => "allow" + } + } + } + else if [event.code] == "304002" { + dissect { + id => "filter_dissect_20" + mapping => { + "message" => "Access %{event.outcome} URL %{url.original} SRC %{source.address} %{}EST %{destination.address} on interface %{cisco.source_interface}" + } + } + } + else if [event.code] == "305011" { + grok { + id => "filter_grok_9" + match => { + "message" => [ + "Built dynamic %{WORD:network.transport} translation from %{DATA:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} to %{DATA:cisco.destination_interface}:%{IP:destination.address}/%{INT:destination.port}" + ] + } + } + mutate { + id => "filter_mutate_18" + add_field => { + "event.category" => "nat_translation" + } + } + } + else if [event.code] == "305012" { + grok { + id => "filter_grok_10" + match => { + "message" => [ + "Teardown dynamic %{WORD:network.transport} translation from %{DATA:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} to %{DATA:cisco.destination_interface}:%{IP:destination.address}/%{INT:destination.port}" + ] + } + } + mutate { + id => "filter_mutate_19" + add_field => { + "event.category" => "nat_translation" + } + } + } + else if [event.code] == "313001" { + dissect { + id => "filter_dissect_21" + mapping => { + "message" => "%{event.outcome} %{network.transport} type=%{cisco.icmp_type}, code=%{cisco.icmp_code} from %{source.address} on interface %{cisco.source_interface}" + } + } + } + else if [event.code] == "313004" { + dissect { + id => "filter_dissect_22" + mapping => { + "message" => "%{event.outcome} %{network.transport} type=%{cisco.icmp_type}, from%{}addr %{source.address} on interface %{cisco.source_interface} to %{destination.address}: no matching session" + } + } + } + else if [event.code] == "313005" { + dissect { + id => "filter_dissect_23" + mapping => { + "message" => "No matching connection for %{network.transport} error message: %{} on %{cisco.source_interface} interface.%{}riginal IP payload: %{}" + } + } + } + else if [event.code] == "313008" { + dissect { + id => "filter_dissect_24" + mapping => { + "message" => "%{event.outcome} %{network.transport} type=%{cisco.icmp_type} , code=%{cisco.icmp_code} from %{source.address} on interface %{cisco.source_interface}" + } + } + } + else if [event.code] == "313009" { + dissect { + id => "filter_dissect_25" + mapping => { + "message" => "%{event.outcome} invalid %{network.transport} code %{cisco.icmp_code} , for %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}" + } + } + } + else if [event.code] == "322001" { + dissect { + id => "filter_dissect_26" + mapping => { + "message" => "%{event.outcome} MAC address %{source.mac}, possible spoof attempt on interface %{cisco.source_interface}" + } + } + } + else if [event.code] == "338001" { + dissect { + id => "filter_dissect_27" + mapping => { + "message" => "Dynamic filter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{source.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" + } + } + } + else if [event.code] == "338002" { + dissect { + id => "filter_dissect_28" + mapping => { + "message" => "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{destination.domain}" + } + } + mutate { + id => "filter_mutate_20" + add_field => { + "server.domain" => "[destination.domain]" + } + } + } + else if [event.code] == "338003" { + dissect { + id => "filter_dissect_29" + mapping => { + "message" => "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" + } + } + } + else if [event.code] == "338004" { + dissect { + id => "filter_dissect_30" + mapping => { + "message" => "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" + } + } + } + else if [event.code] == "338005" { + dissect { + id => "filter_dissect_31" + mapping => { + "message" => "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{source.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" + } + } + mutate { + id => "filter_mutate_21" + add_field => { + "server.domain" => "[source.domain]" + } + } + } + else if [event.code] == "338006" { + dissect { + id => "filter_dissect_32" + mapping => { + "message" => "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{destination.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" + } + } + mutate { + id => "filter_mutate_22" + add_field => { + "server.domain" => "server.domain" + } + } + } + else if [event.code] == "338007" { + dissect { + id => "filter_dissect_33" + mapping => { + "message" => "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" + } + } + } + else if [event.code] == "338008" { + dissect { + id => "filter_dissect_34" + mapping => { + "message" => "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" + } + } + } + else if [event.code] == "338101" { + dissect { + id => "filter_dissect_35" + mapping => { + "message" => "Dynamic %{}ilter %{event.outcome} white%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{source.domain}" + } + } + mutate { + id => "filter_mutate_23" + add_field => { + "server.domain" => "server.domain" + } + } + } + else if [event.code] == "338102" { + dissect { + id => "filter_dissect_36" + mapping => { + "message" => "Dynamic %{}ilter %{event.outcome} white%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{destination.domain}" + } + } + mutate { + id => "filter_mutate_24" + add_field => { + "server.domain" => "server.domain" + } + } + } + else if [event.code] == "338103" { + dissect { + id => "filter_dissect_37" + mapping => { + "message" => "Dynamic %{}ilter %{event.outcome} white%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{}" + } + } + } + else if [event.code] == "338104" { + dissect { + id => "filter_dissect_38" + mapping => { + "message" => "Dynamic %{}ilter %{event.outcome} white%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{}" + } + } + } + else if [event.code] == "338201" { + dissect { + id => "filter_dissect_39" + mapping => { + "message" => "Dynamic %{}ilter %{event.outcome} grey%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{source.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" + } + } + mutate { + id => "filter_mutate_25" + add_field => { + "server.domain" => "server.domain" + } + } + } + else if [event.code] == "338202" { + dissect { + id => "filter_dissect_40" + mapping => { + "message" => "Dynamic %{}ilter %{event.outcome} grey%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{destination.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" + } + } + mutate { + id => "filter_mutate_26" + add_field => { + "server.domain" => "server.domain" + } + } + } + else if [event.code] == "338203" { + dissect { + id => "filter_dissect_41" + mapping => { + "message" => "Dynamic %{}ilter %{event.outcome} grey%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{source.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" + } + } + mutate { + id => "filter_mutate_27" + add_field => { + "server.domain" => "server.domain" + } + } + } + else if [event.code] == "338204" { + dissect { + id => "filter_dissect_42" + mapping => { + "message" => "Dynamic %{}ilter %{event.outcome} grey%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{destination.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" + } + } + mutate { + id => "filter_mutate_28" + add_field => { + "server.domain" => "server.domain" + } + } + } + else if [event.code] == "338301" { + dissect { + id => "filter_dissect_43" + mapping => { + "message" => "Intercepted DNS reply for domain %{source.domain} from %{cisco.source_interface}:%{source.address}/%{source.port} to %{cisco.destination_interface}:%{destination.address}/%{destination.port}, matched %{cisco.list_id}" + } + } + mutate { + id => "filter_mutate_29" + add_field => { + "client.address" => "client.address" + } + } + mutate { + id => "filter_mutate_30" + add_field => { + "client.port" => "client.port" + } + } + mutate { + id => "filter_mutate_31" + add_field => { + "server.address" => "server.address" + } + } + mutate { + id => "filter_mutate_32" + add_field => { + "server.port" => "server.port" + } + } + } + else if [event.code] in ["302014", "302016", "302018", "302021", "302036", "302304", "302306", "302020"] { + grok { + id => "filter_grok_11" + pattern_definitions => { + "NOTCOLON" => "[^:]*" + "ECSSOURCEIPORHOST" => "(?:%{IP:source.address}|%{HOSTNAME:source.domain})" + "ECSDESTIPORHOST" => "(?:%{IP:destination.address}|%{HOSTNAME:destination.domain})" + "MAPPEDSRC" => "(?:%{DATA:cisco.mapped_source_ip}|%{HOSTNAME})" + } + match => { + "message" => [ + "Teardown %{NOTSPACE:network.transport} (?:state-bypass )?connection %{NOTSPACE:cisco.connection_id} (?:for|from) %{NOTCOLON}:%{DATA:source.address}/%{NUMBER:source.port:int}?(\(%{DATA:cisco.source_username}\)|) ?to %{NOTCOLON:cisco.destination_interface}:%{DATA:destination.address}/%{NUMBER:destination.port:int}?(\(%{DATA:cisco.source_username}\)|) ?(?:duration %{TIME:cisco.duration_hms} bytes %{NUMBER:network.bytes:int})%{GREEDYDATA}", + "Teardown %{NOTSPACE:network.transport} (?:state-bypass )?connection %{NOTSPACE:cisco.connection_id} (?:for|from) %{NOTCOLON}:%{DATA:source.address}/%{NUMBER:source.port:int} (?:%{NOTSPACE:cisco.source_username} )?to %{NOTCOLON:cisco.destination_interface}:%{DATA:destination.address}/%{NUMBER:destination.port:int} (?:%{NOTSPACE:cisco.destination_username} )?(?:duration %{TIME:cisco.duration_hms} bytes %{NUMBER:network.bytes:int})%{GREEDYDATA}", + "Teardown %{NOTSPACE:network.transport} connection for faddr (?:%{NOTCOLON:cisco.source_interface}:)?%{ECSDESTIPORHOST}/%{NUMBER} (?:%{NOTSPACE:cisco.destination_username} )?gaddr (?:%{NOTCOLON}:)?%{MAPPEDSRC}/%{NUMBER} laddr (?:%{NOTCOLON:cisco.source_interface}:)?%{ECSSOURCEIPORHOST}/%{NUMBER}(?: %{NOTSPACE:cisco.source_username})?%{GREEDYDATA}", + "Built %{WORD:network.direction} %{NOTSPACE:network.transport} connection for faddr (?:%{NOTCOLON:cisco.source_interface}:)?%{ECSDESTIPORHOST}/%{NUMBER} (?:%{NOTSPACE:cisco.destination_username} )?gaddr (?:%{NOTCOLON}:)?%{MAPPEDSRC}/%{NUMBER} laddr (?:%{NOTCOLON:cisco.source_interface}:)?%{ECSSOURCEIPORHOST}/%{NUMBER}(?: %{NOTSPACE:cisco.source_username})?%{GREEDYDATA}" + ] + } + } + mutate { + id => "filter_mutate_33" + add_field => { + "event.category" => "nat_translation" + } + } + } + else if [event.code] == "419002" { + grok { + id => "filter_grok_12" + match => { + "message" => [ + "%{DATA:cisco.event_error} from %{WORD:cisco.source_interface}:%{IPORHOST:source.address}\/%{INT:source.port} to %{WORD:cisco.destination_interface}:%{IPORHOST:destination.address}\/%{INT:destination.port}" + ] + } + } + mutate { + id => "filter_mutate_34" + add_field => { + "event.category" => "error" + } + } + } + else if [event.code] in ["733100", "752015", "752012"] { + grok { + id => "filter_grok_13" + match => { + "message" => [ + "%{GREEDYDATA:cisco.event_error}" + ] + } + } + mutate { + id => "filter_mutate_35" + add_field => { + "event.category" => "error" + } + } + } + else if [event.code] == "716002" { + grok { + id => "filter_grok_14" + match => { + "message" => [ + "Group \<%{DATA:cisco.group} User \<%{DATA:user.name}\> IP \<%{IP:cisco.client_vpn_ip}\> WebVPN session %{WORD:cisco.client_vpn_session_outcome}\: %{DATA:cisco.web_vpn_action}\." + ] + } + } + } + else if [event.code] in ['722022', '722033', '722055', '722051', '113039', '722023', '722037'] { + grok { + id => "filter_grok_15" + match => { + "message" => [ + "Group \<%{DATA:cisco.group} User \<%{DATA:user.name}\> IP \<%{IP:cisco.client_vpn_ip}\> %{GREEDYDATA:cisco.message}" + ] + } + } + } + else { + grok { + id => "filter_grok_16" + match => { + "message" => [ + "forced_failure" + ] + } + } + } + if [event.category] == "nat_translation" { + drop { + id => "filter_drop_1" + } + } + if [source.address] { + grok { + id => "filter_grok_17" + match => { + "source.address" => [ + "(?:%{IP:source.ip}|%{GREEDYDATA:source.domain})" + ] + } + } + } + if [destination.address] { + grok { + id => "filter_grok_18" + match => { + "destination.address" => [ + "(?:%{IP:destination.ip}|%{GREEDYDATA:destination.domain})" + ] + } + } + } + if [client.address] { + grok { + id => "filter_grok_19" + match => { + "client.address" => [ + "(?:%{IP:client.ip}|%{GREEDYDATA:client.domain})" + ] + } + } + } + if [server.address] { + grok { + id => "filter_grok_20" + match => { + "server.address" => [ + "(?:%{IP:server.ip}|%{GREEDYDATA:server.domain})" + ] + } + } + } + mutate { + id => "filter_mutate_36" + lowercase => ["network.transport", "network.protocol", "network.direction", "event.outcome"] + } + if [event.outcome] == "est-allowed" { + mutate { + id => "filter_mutate_37" + update => { + "event.outcome" => "allow" + } + } + } + else if [event.outcome] == "permitted" { + mutate { + id => "filter_mutate_38" + update => { + "event.outcome" => "allow" + } + } + } + else if [event.outcome] == "denied" { + mutate { + id => "filter_mutate_39" + update => { + "event.outcome" => "deny" + } + } + } + else if [event.outcome] == "dropped" { + mutate { + id => "filter_mutate_40" + update => { + "event.outcome" => "deny" + } + } + } + if [network.transport] == "icmpv6" { + mutate { + id => "filter_mutate_41" + update => { + "network.transport" => "ipv6-icmp" + } + } + } + translate { + id => "filter_translate_1" + field => "network.transport" + destination => "network.iana_number" + dictionary => { + "icmp" => "1" + "igmp" => "2" + "ipv4" => "4" + "tcp" => "6" + "egp" => "8" + "igp" => "9" + "pup" => "12" + "udp" => "17" + "rdp" => "27" + "irtp" => "28" + "dccp" => "33" + "idpr" => "35" + "ipv6" => "41" + "ipv6-route" => "43" + "ipv6-frag" => "44" + "rsvp" => "46" + "gre" => "47" + "esp" => "50" + "ipv6-icmp" => "58" + "ipv6-nonxt" => "59" + "ipv6-opts" => "60" + } + } + mutate { + id => "filter_mutate_42" + remove_field => ["ciscotag", "timestamp"] + } + mutate { + id => "filter_mutate_43" + add_field => { + "event.module" => "cisco" + "event.dataset" => "asa" + } + } + translate { + id => "filter_translate_2" + field => "[event.severity]" + destination => "[log.level]" + dictionary => { + "0" => "emergency" + "1" => "alert" + "2" => "critical" + "3" => "error" + "4" => "warning" + "5" => "notification" + "6" => "informational" + "7" => "debug" + } + } +} +output { + elasticsearch { + id => "output_elasticsearch_1" + api_key => "${es_api_key}" + hosts => "https://homedc-90e54c.es.us-east-2.aws.elastic-cloud.com:443" + index => "asa-1.2" + pipeline => "asa" + } +} diff --git a/tests/Common/unit/conversion_data/pipelines/test-asa.conf b/tests/Common/unit/conversion_data/pipelines/test-asa.conf new file mode 100644 index 0000000..dbd9def --- /dev/null +++ b/tests/Common/unit/conversion_data/pipelines/test-asa.conf @@ -0,0 +1,897 @@ +input { + udp { + id => "input_udp_1" + port => "5119" + } + cloudwatch { + } +} +filter { + mutate { + id => "filter_mutate_1" + rename => { + "message" => "log.original" + "host" => "observer.ip" + } + copy => { + "host" => "sysloghost" + } + } + grok { + id => "filter_grok_1" + match => { + "log.original" => [ + "%{CISCO_TAGGED_SYSLOG} %{GREEDYDATA:message}", + "^<%{POSINT:syslog_pri}>%{DATA}: %%{DATA:ciscotag}: %{GREEDYDATA:message}", + "^<%{POSINT:syslog_pri}>%%{DATA:ciscotag}: %{GREEDYDATA:message}" + ] + } + } + grok { + id => "filter_grok_2" + match => { + "ciscotag" => [ + "%{WORD}-%{INT:event.severity}-%{INT:event.code}", + "%{WORD}-%{WORD}-%{INT:event.severity}-%{INT:event.code}" + ] + } + } + mutate { + id => "filter_mutate_2" + add_field => { + "event.action" => "firewall-rule" + } + } + if [event.code] == "105012" { + grok { + id => "filter_grok_3" + match => { + "message" => [ + "Teardown dynamic %{WORD:network.transport} translation from %{DATA:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} to %{DATA:cisco.destination_interface}:%{IP:destination.address}/%{INT:destination.port} duration %{DATA:cisco.duration_hms}$" + ] + } + } + } + else if [event.code] == "106001" { + dissect { + id => "filter_dissect_1" + mapping => { + "message" => "%{network.direction} %{network.transport} connection %{event.outcome} from %{source.address}/%{source.port} to %{destination.address}/%{destination.port} flags %{} on interface %{source_interface}" + } + } + mutate { + id => "filter_mutate_3" + add_field => { + "event.category" => "network_traffic" + } + } + } + else if [event.code] == "106002" { + dissect { + id => "filter_dissect_2" + mapping => { + "message" => "%{network.transport} Connection %{event.outcome} by %{network.direction} list %{cisco.list_id} src %{source.address} dest %{destination.address}" + } + } + mutate { + id => "filter_mutate_4" + add_field => { + "event.category" => "network_traffic" + } + } + } + else if [event.code] == "106006" { + dissect { + id => "filter_dissect_3" + mapping => { + "message" => "%{event.outcome} %{network.direction} %{network.transport} from %{source.address}/%{source.port} to %{destination.address}/%{destination.port} on interface %{cisco.source_interface}" + } + } + mutate { + id => "filter_mutate_5" + add_field => { + "event.category" => "network_traffic" + } + } + } + else if [event.code] == "106007" { + dissect { + id => "filter_dissect_4" + mapping => { + "message" => "%{event.outcome} %{network.direction} %{network.transport} from %{source.address}/%{source.port} to %{destination.address}/%{destination.port} due to %{network.protocol} %{}" + } + } + mutate { + id => "filter_mutate_6" + add_field => { + "event.category" => "network_traffic" + } + } + } + else if [event.code] == "106010" { + dissect { + id => "filter_dissect_5" + mapping => { + "message" => "%{event.outcome} %{network.direction} %{network.transport} src %{cisco.source_interface}:%{source.address}/%{source.port} %{} dst %{cisco.destination_interface}:%{destination.address}/%{destination.port} %{}" + } + } + mutate { + id => "filter_mutate_7" + add_field => { + "event.category" => "network_traffic" + } + } + } + else if [event.code] == "106013" { + dissect { + id => "filter_dissect_6" + mapping => { + "message" => "Dropping echo request from %{source.address} to PAT address %{destination.address}" + } + } + mutate { + id => "filter_mutate_8" + add_field => { + "network.transport" => "icmp" + "network.direction" => "inbound" + } + } + } + else if [event.code] == "106014" { + dissect { + id => "filter_dissect_7" + mapping => { + "message" => "%{event.outcome} %{network.direction} %{network.transport} src %{cisco.source_interface}:%{source.address} %{}dst %{cisco.destination_interface}:%{destination.address} %{}" + } + } + mutate { + id => "filter_mutate_9" + add_field => { + "event.category" => "network_traffic" + } + } + } + else if [event.code] == "106015" { + dissect { + id => "filter_dissect_8" + mapping => { + "message" => "%{event.outcome} %{network.transport} (no connection) from %{source.address}/%{source.port} to %{destination.address}/%{destination.port} flags %{} on interface %{cisco.source_interface}" + } + } + mutate { + id => "filter_mutate_10" + add_field => { + "event.category" => "nat_translation" + } + } + } + else if [event.code] == "106016" { + dissect { + id => "filter_dissect_9" + mapping => { + "message" => "%{event.outcome} IP spoof from (%{source.address}) to %{destination.address} on interface %{cisco.source_interface}" + } + } + } + else if [event.code] == "106017" { + dissect { + id => "filter_dissect_10" + mapping => { + "message" => "%{event.outcome} IP due to Land Attack from %{source.address} to %{destination.address}" + } + } + } + else if [event.code] == "106018" { + dissect { + id => "filter_dissect_11" + mapping => { + "message" => "%{network.transport} packet type %{cisco.icmp_type} %{event.outcome} by %{network.direction} list %{cisco.list_id} src %{source.address} dest %{destination.address}" + } + } + } + else if [event.code] == "106020" { + dissect { + id => "filter_dissect_12" + mapping => { + "message" => "%{event.outcome} IP teardrop fragment (size = %{}, offset = %{}) from %{source.address} to %{destination.address}" + } + } + } + else if [event.code] == "106021" { + dissect { + id => "filter_dissect_13" + mapping => { + "message" => "%{event.outcome} %{network.transport} reverse path check from %{source.address} to %{destination.address} on interface %{cisco.source_interface}" + } + } + } + else if [event.code] == "106022" { + dissect { + id => "filter_dissect_14" + mapping => { + "message" => "%{event.outcome} %{network.transport} connection spoof from %{source.address} to %{destination.address} on interface %{cisco.source_interface}" + } + } + } + else if [event.code] == "106023" { + grok { + id => "filter_grok_4" + match => { + "message" => [ + '%{WORD:event.outcome} %{WORD:network.transport} src %{WORD:cisco.source_interface}:%{IPORHOST:source.address}?(/%{INT:source.port}) dst %{WORD:cisco.destination_interface}:%{IPORHOST:destination.address}?(/%{INT:destination.port}) by access-group "%{DATA:cisco.list_id}"', + '%{WORD:event.outcome} %{WORD:network.transport} src %{WORD:cisco.source_interface}:%{IPORHOST:source.address} dst %{WORD:destination.direction}:%{IPORHOST:destination.address} \(%{DATA}\) by access-group "%{DATA:cisco.list_id}"', + '%{WORD:event.outcome} %{WORD:network.transport} src %{WORD:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} dst %{WORD:cisco.destination.interface}:%{IPORHOST:destination.address}/%{INT:destination.port} by access-group "%{DATA:cisco.list_id}"' + ] + } + } + mutate { + id => "filter_mutate_11" + add_field => { + "event.category" => "network_traffic" + } + } + } + else if [event.code] == "106027" { + dissect { + id => "filter_dissect_15" + mapping => { + "message" => '%{} %{event.outcome} src %{source.address} dst %{destination.address} by access-group "%{cisco.list_id}"%{}' + } + } + } + else if [event.code] == "106100" { + dissect { + id => "filter_dissect_16" + mapping => { + "message" => "access-list %{cisco.list_id} %{event.outcome} %{network.transport} %{cisco.source_interface}/%{source.address}(%{source.port}) -> %{cisco.destination_interface}/%{destination.address}(%{destination.port}) %{}" + } + } + } + else if [event.code] == "106102" { + dissect { + id => "filter_dissect_17" + mapping => { + "message" => "access-list %{cisco.list_id} %{event.outcome} %{network.transport} for user %{cisco.username} %{cisco.source_interface}/%{source.address} %{source.port} %{cisco.destination_interface}/%{destination.address} %{destination.port} %{}" + } + } + } + else if [event.code] == "106103" { + dissect { + id => "filter_dissect_18" + mapping => { + "message" => "access-list %{cisco.list_id} %{event.outcome} %{network.transport} for user %{cisco.username} %{cisco.source_interface}/%{source.address} %{source.port} %{cisco.destination_interface}/%{destination.address} %{destination.port} %{}" + } + } + } + else if [event.code] == "113004" { + grok { + id => "filter_grok_5" + match => { + "message" => [ + "AAA user accounting %{WORD:cisco.auth_outcome} : server =%{SPACE}%{IP:source.address} : user =%{SPACE}%{DATA:source.user.name}$" + ] + } + } + mutate { + id => "filter_mutate_12" + add_field => { + "event.category" => "authentication" + } + } + if [cisco.auth_outcome] == "Successful" { + mutate { + id => "filter_mutate_13" + add_field => { + "event.action" => "authentication_success" + } + } + } + else { + mutate { + id => "filter_mutate_14" + add_field => { + "event.action" => "authentication_failure" + } + } + } + } + else if [event.code] == "302015" or [event.code] == "302013" { + grok { + id => "filter_grok_6" + match => { + "message" => [ + "Built %{WORD:network.direction} %{WORD:network.transport} connection %{INT} for %{DATA:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} \(%{IP}|\) to %{DATA:cisco.destination_interface}:%{IPORHOST:destination.address}/%{INT:destination.port} \(%{DATA}\)", + "Built %{WORD:network.direction} %{WORD:network.transport} connection %{INT} for %{DATA:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} \(%{DATA}\)?(\(%{DATA:cisco.source_username}\)) to %{DATA:cisco.destination_interface}:%{IPORHOST:destination.address}/%{INT:destination.port} \(%{DATA}\) ?(\(%{DATA:cisco.username}\))", + "Built %{WORD:network.direction} %{WORD:network.transport} connection %{INT:cisco.connection_id} for %{WORD:cisco.source_interface}:%{IPORHOST:source.address}\/%{INT:source.port} \(%{DATA}\) to %{WORD:cisco.destination_interface}:%{IPORHOST:destination.address}/%{INT:destination.port}" + ] + } + } + mutate { + id => "filter_mutate_15" + add_field => { + "event.category" => "nat_translation" + } + } + } + else if [event.code] == "110003" { + grok { + id => "filter_grok_7" + match => { + "message" => [ + "%{DATA:cisco.event_error} for %{WORD:network.transport} from %{DATA:cisco.source_interface}:%{IP:source.address}\/%{INT:source.port} to %{DATA:cisco.destination_interface}:%{IP:destination.address}\/%{INT:destination.port}" + ] + } + } + mutate { + id => "filter_mutate_16" + add_field => { + "event.category" => "error" + } + } + } + else if [event.code] == "113019" { + grok { + id => "filter_grok_8" + match => { + "message" => [ + "Group = %{DATA:cisco.group}, Username = %{DATA:user.name}, IP = %{IP:cisco.client_vpn_ip}, %{DATA:cisco.client_vpn_action}\. Session Type: %{DATA:cisco.session_type}, Duration: %{DATA:cisco.duration}, Bytes xmt: %{INT:cisco.vpn_transmit_byte_summary}, Bytes rcv: %{INT:cisco.vpn_receive_byte_summary}, Reason: %{DATA:cisco.client_vpn_outcome}$" + ] + } + } + } + else if [event.code] == "304001" { + dissect { + id => "filter_dissect_19" + mapping => { + "message" => "%{source.address} %{}ccessed URL %{destination.address}:%{url.original}" + } + } + mutate { + id => "filter_mutate_17" + add_field => { + "event.outcome" => "allow" + } + } + } + else if [event.code] == "304002" { + dissect { + id => "filter_dissect_20" + mapping => { + "message" => "Access %{event.outcome} URL %{url.original} SRC %{source.address} %{}EST %{destination.address} on interface %{cisco.source_interface}" + } + } + } + else if [event.code] == "305011" { + grok { + id => "filter_grok_9" + match => { + "message" => [ + "Built dynamic %{WORD:network.transport} translation from %{DATA:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} to %{DATA:cisco.destination_interface}:%{IP:destination.address}/%{INT:destination.port}" + ] + } + } + mutate { + id => "filter_mutate_18" + add_field => { + "event.category" => "nat_translation" + } + } + } + else if [event.code] == "305012" { + grok { + id => "filter_grok_10" + match => { + "message" => [ + "Teardown dynamic %{WORD:network.transport} translation from %{DATA:cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} to %{DATA:cisco.destination_interface}:%{IP:destination.address}/%{INT:destination.port}" + ] + } + } + mutate { + id => "filter_mutate_19" + add_field => { + "event.category" => "nat_translation" + } + } + } + else if [event.code] == "313001" { + dissect { + id => "filter_dissect_21" + mapping => { + "message" => "%{event.outcome} %{network.transport} type=%{cisco.icmp_type}, code=%{cisco.icmp_code} from %{source.address} on interface %{cisco.source_interface}" + } + } + } + else if [event.code] == "313004" { + dissect { + id => "filter_dissect_22" + mapping => { + "message" => "%{event.outcome} %{network.transport} type=%{cisco.icmp_type}, from%{}addr %{source.address} on interface %{cisco.source_interface} to %{destination.address}: no matching session" + } + } + } + else if [event.code] == "313005" { + dissect { + id => "filter_dissect_23" + mapping => { + "message" => "No matching connection for %{network.transport} error message: %{} on %{cisco.source_interface} interface.%{}riginal IP payload: %{}" + } + } + } + else if [event.code] == "313008" { + dissect { + id => "filter_dissect_24" + mapping => { + "message" => "%{event.outcome} %{network.transport} type=%{cisco.icmp_type} , code=%{cisco.icmp_code} from %{source.address} on interface %{cisco.source_interface}" + } + } + } + else if [event.code] == "313009" { + dissect { + id => "filter_dissect_25" + mapping => { + "message" => "%{event.outcome} invalid %{network.transport} code %{cisco.icmp_code} , for %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}" + } + } + } + else if [event.code] == "322001" { + dissect { + id => "filter_dissect_26" + mapping => { + "message" => "%{event.outcome} MAC address %{source.mac}, possible spoof attempt on interface %{cisco.source_interface}" + } + } + } + else if [event.code] == "338001" { + dissect { + id => "filter_dissect_27" + mapping => { + "message" => "Dynamic filter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{source.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" + } + } + } + else if [event.code] == "338002" { + dissect { + id => "filter_dissect_28" + mapping => { + "message" => "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{destination.domain}" + } + } + mutate { + id => "filter_mutate_20" + add_field => { + "server.domain" => "[destination.domain]" + } + } + } + else if [event.code] == "338003" { + dissect { + id => "filter_dissect_29" + mapping => { + "message" => "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" + } + } + } + else if [event.code] == "338004" { + dissect { + id => "filter_dissect_30" + mapping => { + "message" => "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" + } + } + } + else if [event.code] == "338005" { + dissect { + id => "filter_dissect_31" + mapping => { + "message" => "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{source.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" + } + } + mutate { + id => "filter_mutate_21" + add_field => { + "server.domain" => "[source.domain]" + } + } + } + else if [event.code] == "338006" { + dissect { + id => "filter_dissect_32" + mapping => { + "message" => "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{destination.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" + } + } + mutate { + id => "filter_mutate_22" + add_field => { + "server.domain" => "server.domain" + } + } + } + else if [event.code] == "338007" { + dissect { + id => "filter_dissect_33" + mapping => { + "message" => "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" + } + } + } + else if [event.code] == "338008" { + dissect { + id => "filter_dissect_34" + mapping => { + "message" => "Dynamic %{}ilter %{event.outcome} black%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" + } + } + } + else if [event.code] == "338101" { + dissect { + id => "filter_dissect_35" + mapping => { + "message" => "Dynamic %{}ilter %{event.outcome} white%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{source.domain}" + } + } + mutate { + id => "filter_mutate_23" + add_field => { + "server.domain" => "server.domain" + } + } + } + else if [event.code] == "338102" { + dissect { + id => "filter_dissect_36" + mapping => { + "message" => "Dynamic %{}ilter %{event.outcome} white%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{destination.domain}" + } + } + mutate { + id => "filter_mutate_24" + add_field => { + "server.domain" => "server.domain" + } + } + } + else if [event.code] == "338103" { + dissect { + id => "filter_dissect_37" + mapping => { + "message" => "Dynamic %{}ilter %{event.outcome} white%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{}" + } + } + } + else if [event.code] == "338104" { + dissect { + id => "filter_dissect_38" + mapping => { + "message" => "Dynamic %{}ilter %{event.outcome} white%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{}" + } + } + } + else if [event.code] == "338201" { + dissect { + id => "filter_dissect_39" + mapping => { + "message" => "Dynamic %{}ilter %{event.outcome} grey%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{source.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" + } + } + mutate { + id => "filter_mutate_25" + add_field => { + "server.domain" => "server.domain" + } + } + } + else if [event.code] == "338202" { + dissect { + id => "filter_dissect_40" + mapping => { + "message" => "Dynamic %{}ilter %{event.outcome} grey%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{destination.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" + } + } + mutate { + id => "filter_mutate_26" + add_field => { + "server.domain" => "server.domain" + } + } + } + else if [event.code] == "338203" { + dissect { + id => "filter_dissect_41" + mapping => { + "message" => "Dynamic %{}ilter %{event.outcome} grey%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}source %{} resolved from %{cisco.list_id} list: %{source.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" + } + } + mutate { + id => "filter_mutate_27" + add_field => { + "server.domain" => "server.domain" + } + } + } + else if [event.code] == "338204" { + dissect { + id => "filter_dissect_42" + mapping => { + "message" => "Dynamic %{}ilter %{event.outcome} grey%{}d %{network.transport} traffic from %{cisco.source_interface}:%{source.address}/%{source.port} (%{cisco.mapped_source_ip}/%{cisco.mapped_source_port}) to %{cisco.destination_interface}:%{destination.address}/%{destination.port} (%{cisco.mapped_destination_ip}/%{cisco.mapped_destination_port})%{}destination %{} resolved from %{cisco.list_id} list: %{destination.domain}, threat-level: %{cisco.threat_level}, category: %{cisco.threat_category}" + } + } + mutate { + id => "filter_mutate_28" + add_field => { + "server.domain" => "server.domain" + } + } + } + else if [event.code] == "338301" { + dissect { + id => "filter_dissect_43" + mapping => { + "message" => "Intercepted DNS reply for domain %{source.domain} from %{cisco.source_interface}:%{source.address}/%{source.port} to %{cisco.destination_interface}:%{destination.address}/%{destination.port}, matched %{cisco.list_id}" + } + } + mutate { + id => "filter_mutate_29" + add_field => { + "client.address" => "client.address" + } + } + mutate { + id => "filter_mutate_30" + add_field => { + "client.port" => "client.port" + } + } + mutate { + id => "filter_mutate_31" + add_field => { + "server.address" => "server.address" + } + } + mutate { + id => "filter_mutate_32" + add_field => { + "server.port" => "server.port" + } + } + } + else if [event.code] in ["302014", "302016", "302018", "302021", "302036", "302304", "302306", "302020"] { + grok { + id => "filter_grok_11" + pattern_definitions => { + "NOTCOLON" => "[^:]*" + "ECSSOURCEIPORHOST" => "(?:%{IP:source.address}|%{HOSTNAME:source.domain})" + "ECSDESTIPORHOST" => "(?:%{IP:destination.address}|%{HOSTNAME:destination.domain})" + "MAPPEDSRC" => "(?:%{DATA:cisco.mapped_source_ip}|%{HOSTNAME})" + } + match => { + "message" => [ + "Teardown %{NOTSPACE:network.transport} (?:state-bypass )?connection %{NOTSPACE:cisco.connection_id} (?:for|from) %{NOTCOLON}:%{DATA:source.address}/%{NUMBER:source.port:int}?(\(%{DATA:cisco.source_username}\)|) ?to %{NOTCOLON:cisco.destination_interface}:%{DATA:destination.address}/%{NUMBER:destination.port:int}?(\(%{DATA:cisco.source_username}\)|) ?(?:duration %{TIME:cisco.duration_hms} bytes %{NUMBER:network.bytes:int})%{GREEDYDATA}", + "Teardown %{NOTSPACE:network.transport} (?:state-bypass )?connection %{NOTSPACE:cisco.connection_id} (?:for|from) %{NOTCOLON}:%{DATA:source.address}/%{NUMBER:source.port:int} (?:%{NOTSPACE:cisco.source_username} )?to %{NOTCOLON:cisco.destination_interface}:%{DATA:destination.address}/%{NUMBER:destination.port:int} (?:%{NOTSPACE:cisco.destination_username} )?(?:duration %{TIME:cisco.duration_hms} bytes %{NUMBER:network.bytes:int})%{GREEDYDATA}", + "Teardown %{NOTSPACE:network.transport} connection for faddr (?:%{NOTCOLON:cisco.source_interface}:)?%{ECSDESTIPORHOST}/%{NUMBER} (?:%{NOTSPACE:cisco.destination_username} )?gaddr (?:%{NOTCOLON}:)?%{MAPPEDSRC}/%{NUMBER} laddr (?:%{NOTCOLON:cisco.source_interface}:)?%{ECSSOURCEIPORHOST}/%{NUMBER}(?: %{NOTSPACE:cisco.source_username})?%{GREEDYDATA}", + "Built %{WORD:network.direction} %{NOTSPACE:network.transport} connection for faddr (?:%{NOTCOLON:cisco.source_interface}:)?%{ECSDESTIPORHOST}/%{NUMBER} (?:%{NOTSPACE:cisco.destination_username} )?gaddr (?:%{NOTCOLON}:)?%{MAPPEDSRC}/%{NUMBER} laddr (?:%{NOTCOLON:cisco.source_interface}:)?%{ECSSOURCEIPORHOST}/%{NUMBER}(?: %{NOTSPACE:cisco.source_username})?%{GREEDYDATA}" + ] + } + } + mutate { + id => "filter_mutate_33" + add_field => { + "event.category" => "nat_translation" + } + } + } + else if [event.code] == "419002" { + grok { + id => "filter_grok_12" + match => { + "message" => [ + "%{DATA:cisco.event_error} from %{WORD:cisco.source_interface}:%{IPORHOST:source.address}\/%{INT:source.port} to %{WORD:cisco.destination_interface}:%{IPORHOST:destination.address}\/%{INT:destination.port}" + ] + } + } + mutate { + id => "filter_mutate_34" + add_field => { + "event.category" => "error" + } + } + } + else if [event.code] in ["733100", "752015", "752012"] { + grok { + id => "filter_grok_13" + match => { + "message" => [ + "%{GREEDYDATA:cisco.event_error}" + ] + } + } + mutate { + id => "filter_mutate_35" + add_field => { + "event.category" => "error" + } + } + } + else if [event.code] == "716002" { + grok { + id => "filter_grok_14" + match => { + "message" => [ + "Group \<%{DATA:cisco.group} User \<%{DATA:user.name}\> IP \<%{IP:cisco.client_vpn_ip}\> WebVPN session %{WORD:cisco.client_vpn_session_outcome}\: %{DATA:cisco.web_vpn_action}\." + ] + } + } + } + else if [event.code] in ['722022', '722033', '722055', '722051', '113039', '722023', '722037'] { + grok { + id => "filter_grok_15" + match => { + "message" => [ + "Group \<%{DATA:cisco.group} User \<%{DATA:user.name}\> IP \<%{IP:cisco.client_vpn_ip}\> %{GREEDYDATA:cisco.message}" + ] + } + } + } + else { + grok { + id => "filter_grok_16" + match => { + "message" => [ + "forced_failure" + ] + } + } + } + if [event.category] == "nat_translation" { + drop { + id => "filter_drop_1" + } + } + if [source.address] { + grok { + id => "filter_grok_17" + match => { + "source.address" => [ + "(?:%{IP:source.ip}|%{GREEDYDATA:source.domain})" + ] + } + } + } + if [destination.address] { + grok { + id => "filter_grok_18" + match => { + "destination.address" => [ + "(?:%{IP:destination.ip}|%{GREEDYDATA:destination.domain})" + ] + } + } + } + if [client.address] { + grok { + id => "filter_grok_19" + match => { + "client.address" => [ + "(?:%{IP:client.ip}|%{GREEDYDATA:client.domain})" + ] + } + } + } + if [server.address] { + grok { + id => "filter_grok_20" + match => { + "server.address" => [ + "(?:%{IP:server.ip}|%{GREEDYDATA:server.domain})" + ] + } + } + } + mutate { + id => "filter_mutate_36" + lowercase => ["network.transport", "network.protocol", "network.direction", "event.outcome"] + } + if [event.outcome] == "est-allowed" { + mutate { + id => "filter_mutate_37" + update => { + "event.outcome" => "allow" + } + } + } + else if [event.outcome] == "permitted" { + mutate { + id => "filter_mutate_38" + update => { + "event.outcome" => "allow" + } + } + } + else if [event.outcome] == "denied" { + mutate { + id => "filter_mutate_39" + update => { + "event.outcome" => "deny" + } + } + } + else if [event.outcome] == "dropped" { + mutate { + id => "filter_mutate_40" + update => { + "event.outcome" => "deny" + } + } + } + if [network.transport] == "icmpv6" { + mutate { + id => "filter_mutate_41" + update => { + "network.transport" => "ipv6-icmp" + } + } + } + translate { + id => "filter_translate_1" + field => "network.transport" + destination => "network.iana_number" + dictionary => { + "icmp" => "1" + "igmp" => "2" + "ipv4" => "4" + "tcp" => "6" + "egp" => "8" + "igp" => "9" + "pup" => "12" + "udp" => "17" + "rdp" => "27" + "irtp" => "28" + "dccp" => "33" + "idpr" => "35" + "ipv6" => "41" + "ipv6-route" => "43" + "ipv6-frag" => "44" + "rsvp" => "46" + "gre" => "47" + "esp" => "50" + "ipv6-icmp" => "58" + "ipv6-nonxt" => "59" + "ipv6-opts" => "60" + } + } + mutate { + id => "filter_mutate_42" + remove_field => ["ciscotag", "timestamp"] + } + mutate { + id => "filter_mutate_43" + add_field => { + "event.module" => "cisco" + "event.dataset" => "asa" + } + } + translate { + id => "filter_translate_2" + field => "[event.severity]" + destination => "[log.level]" + dictionary => { + "0" => "emergency" + "1" => "alert" + "2" => "critical" + "3" => "error" + "4" => "warning" + "5" => "notification" + "6" => "informational" + "7" => "debug" + } + } +} +output { + elasticsearch { + id => "output_elasticsearch_1" + api_key => "${es_api_key}" + hosts => "https://homedc-90e54c.es.us-east-2.aws.elastic-cloud.com:443" + index => "asa-1.2" + pipeline => "asa" + } +} diff --git a/tests/Common/unit/conversion_data/pipelines/test-boolean-numeric.conf b/tests/Common/unit/conversion_data/pipelines/test-boolean-numeric.conf new file mode 100644 index 0000000..663b5bd --- /dev/null +++ b/tests/Common/unit/conversion_data/pipelines/test-boolean-numeric.conf @@ -0,0 +1,58 @@ +input { + tcp { + port => 5000 + ssl_enable => "false" + buffer_size => 65536 + } + udp { + port => 514 + queue_size => 2000 + workers => 4 + } +} +filter { + grok { + match => { + "message" => "%{NUMBER:duration:float} %{NUMBER:status:int}" + } + keep_empty_captures => "false" + tag_on_failure => ["_grokfailure"] + timeout_millis => 30000 + break_on_match => "true" + } + mutate { + convert => { + "duration" => "float" + "status" => "integer" + "bytes" => "integer" + } + } + if [duration] > 1.5 { + mutate { + add_field => { + "slow_request" => "true" + } + } + } + if [status] >= 500 { + mutate { + add_field => { + "is_error" => "true" + "error_code" => 500 + "threshold_pct" => 0.99 + } + } + } + throttle { + before_count => 3 + after_count => 1 + period => 60 + key => "%{host}" + add_tag => ["throttled"] + } +} +output { + stdout { + codec => rubydebug + } +} diff --git a/tests/Common/unit/conversion_data/pipelines/test-comments-brace-in-comment.conf b/tests/Common/unit/conversion_data/pipelines/test-comments-brace-in-comment.conf new file mode 100644 index 0000000..e8be51d --- /dev/null +++ b/tests/Common/unit/conversion_data/pipelines/test-comments-brace-in-comment.conf @@ -0,0 +1,39 @@ +input { +} +filter { + mutate { + # add_field => {"this_key" => "this_value"} + # rename => {"old_field" => "new_field"} + # remove_field => ["field1", "field2"] + # replace => {"message" => "override: %{message}"} + add_field => { + "real_field" => "real_value" + } + remove_field => ["unwanted"] + } + grok { + # match => { "message" => "%{COMBINEDAPACHELOG}" } + # pattern_definitions => { "MY_PATTERN" => "\\w+" } + match => { + "message" => "%{GREEDYDATA:raw_message}" + } + } + date { + # match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z", "ISO8601"] + # target => "@timestamp" + match => ["timestamp", "ISO8601"] + target => "@timestamp" + } + translate { + # dictionary => { "200" => "OK", "404" => "Not Found", "500" => "Error" } + field => "status_code" + destination => "status_label" + dictionary => { + "200" => "OK" + "404" => "Not Found" + } + fallback => "Unknown" + } +} +output { +} diff --git a/tests/Common/unit/conversion_data/pipelines/test-comments-mixed.conf b/tests/Common/unit/conversion_data/pipelines/test-comments-mixed.conf new file mode 100644 index 0000000..f369943 --- /dev/null +++ b/tests/Common/unit/conversion_data/pipelines/test-comments-mixed.conf @@ -0,0 +1,57 @@ +input { + # inline on section opener + udp { + # standalone inside plugin + # add_field => {"commented_out" => "value"} standalone with braces + # inline on plugin opener + # inline on scalar value + # inline on array + port => 5140 + buffer_size => 65536 + tags => ["syslog"] + } + # inline on plugin closer -> section comment +} +filter { + # inline on filter opener + # standalone at section level + mutate { + # standalone before first pair + # standalone between pairs + # standalone at end of plugin + # inline on mutate opener + # inline on hash opener + # inline on hash pair + # inline on hash closer + # inline on array pair + add_field => { + "key1" => "value1" + "key2" => "value2" + } + remove_field => ["old"] + } + # inline on plugin closer -> section comment + # standalone between plugins + if [type] == "web" { + # standalone inside conditional + grok { + # standalone inside nested plugin + # match => {"message" => "%{GREEDYDATA}"} standalone with braces in conditional + # inline on nested plugin opener + # inline inside nested hash + # inline on nested hash closer + match => { + "message" => "%{COMBINEDAPACHELOG}" + } + } + # inline on nested plugin closer + # standalone at end of conditional block + } + drop { + } +} +output { + stdout { + codec => rubydebug + } +} diff --git a/tests/Common/unit/conversion_data/pipelines/test-comments-plugin-inline.conf b/tests/Common/unit/conversion_data/pipelines/test-comments-plugin-inline.conf new file mode 100644 index 0000000..621e651 --- /dev/null +++ b/tests/Common/unit/conversion_data/pipelines/test-comments-plugin-inline.conf @@ -0,0 +1,46 @@ +input { + udp { + # inline comment on plugin opener + # inline comment on a scalar value + # another scalar inline comment + # inline comment after an array + port => 5140 + buffer_size => 65536 + tags => ["udp", "syslog"] + } + # inline comment on plugin closer — becomes section-level comment +} +filter { + mutate { + # opener comment + # inline comment on hash opener + # inline comment on hash pair + # another hash pair comment + # inline comment on hash closer + # inline on array + # another hash opener with inline + # pair comment + # hash closer comment + add_field => { + "first" => "value1" + "second" => "value2" + } + remove_field => ["unwanted", "junk"] + rename => { + "old_name" => "new_name" + } + } + # plugin closer — section-level + drop { + # opener comment on empty plugin + } + # closer comment on empty plugin +} +output { + stdout { + # output plugin with inline + # inline on codec line + codec => rubydebug + } + # closer +} diff --git a/tests/Common/unit/conversion_data/pipelines/test-comments-section-opener.conf b/tests/Common/unit/conversion_data/pipelines/test-comments-section-opener.conf new file mode 100644 index 0000000..ce0490b --- /dev/null +++ b/tests/Common/unit/conversion_data/pipelines/test-comments-section-opener.conf @@ -0,0 +1,29 @@ +input { + # inline comment on input opener + beats { + # inline on beats opener + port => 5044 + } +} +filter { + # inline comment on filter opener + mutate { + add_field => { + "processed" => "true" + } + } + # standalone comment between plugins at section level + drop { + } +} +output { + # inline comment on output opener + elasticsearch { + hosts => ["localhost:9200"] + index => "logs-%{+YYYY.MM.dd}" + } + # standalone at section level before second output plugin + stdout { + codec => rubydebug + } +} diff --git a/tests/Common/unit/conversion_data/pipelines/test-comments-standalone-in-plugin.conf b/tests/Common/unit/conversion_data/pipelines/test-comments-standalone-in-plugin.conf new file mode 100644 index 0000000..1beb186 --- /dev/null +++ b/tests/Common/unit/conversion_data/pipelines/test-comments-standalone-in-plugin.conf @@ -0,0 +1,29 @@ +input { +} +filter { + mutate { + # This is a standalone comment at the top of a plugin block + # Standalone comment in the middle of a plugin block + # Another standalone at the bottom + add_field => { + "field1" => "value1" + } + remove_field => ["junk"] + } + grok { + # Standalone comment before the only config key + # Standalone at the end of plugin + match => { + "message" => "%{COMBINEDAPACHELOG}" + } + } + # Section-level comment between plugins + mutate { + # Leading standalone + # Second leading standalone + # Trailing standalone + uppercase => ["log_level"] + } +} +output { +} diff --git a/tests/Common/unit/conversion_data/pipelines/test-complex2.conf b/tests/Common/unit/conversion_data/pipelines/test-complex2.conf new file mode 100644 index 0000000..555150b --- /dev/null +++ b/tests/Common/unit/conversion_data/pipelines/test-complex2.conf @@ -0,0 +1,329 @@ +input { + # + # "LogstashUI kitchen sink" pipeline + # Goal: be extremely feature-rich while staying within known-valid plugin options. + # + # Beats / Elastic Agent style shippers + beats { + id => "in_beats_5044" + port => "5044" + add_field => { + "ingest_transport" => "beats" + } + tags => ["from_beats"] + } + # JSON-over-TCP (common for app logs) + tcp { + id => "in_tcp_json_5514" + port => 5514 + mode => "server" + codec => json + add_field => { + "ingest_transport" => "tcp" + } + tags => ["from_tcp"] + } + # Syslog-ish UDP + udp { + id => "in_udp_5515" + port => 5515 + codec => plain + add_field => { + "ingest_transport" => "udp" + } + tags => ["from_udp"] + } + # HTTP event intake (webhooks, apps posting JSON, etc.) + http { + id => "in_http_8080" + port => 8080 + codec => json + add_field => { + "ingest_transport" => "http" + } + tags => ["from_http"] + } + # Local dev/testing input + stdin { + id => "in_stdin" + codec => line + add_field => { + "ingest_transport" => "stdin" + } + tags => ["from_stdin"] + } + # Synthetic test data (makes it easy to validate end-to-end quickly) + generator { + id => "in_generator" + lines => ["Feb 21 09:12:01 host1 sshd[123]: Failed password for invalid user admin from 10.1.2.3 port 51234 ssh2", "{\"@timestamp\":\"2026-02-21T14:12:02Z\",\"message\":\"GET /health 200\",\"source_ip\":\"8.8.8.8\",\"user_agent\":\"Mozilla/5.0\"}", "level=info service=api latency_ms=42 source_ip=192.168.1.50 msg=\"request completed\""] + count => 1 + add_field => { + "ingest_transport" => "generator" + } + tags => ["from_generator"] + } +} +filter { + # + # Normalize a few shared fields + # + mutate { + id => "f_mutate_bootstrap" + add_field => { + "[@metadata][pipeline]" => "logstashui_kitchen_sink" + "event.module" => "logstashui" + } + } + # Keep a canonical message field + if ![message] and [event][original] { + mutate { + id => "f_mutate_event_original_to_message" + copy => { + "[event][original]" => "message" + } + } + } + # + # Try to parse JSON *if* message looks like JSON (common when tcp/udp/plain feed JSON strings) + # + if [message] =~ "^[[:space:]]*\\{" { + json { + id => "f_json_from_message" + source => "message" + target => "json" + tag_on_failure => ["_jsonparsefailure_message"] + } + # If json parsed, promote a few expected keys (only if present) + if [json][@timestamp] { + mutate { + id => "f_promote_json_ts" + copy => { + "[json][@timestamp]" => "@timestamp" + } + } + } + if [json][source_ip] { + mutate { + id => "f_promote_json_source_ip" + copy => { + "[json][source_ip]" => "source_ip" + } + } + } + if [json][user_agent] { + mutate { + id => "f_promote_json_ua" + copy => { + "[json][user_agent]" => "user_agent" + } + } + } + } + # + # Syslog-ish parsing (UDP and some TCP) + # + if "from_udp" in [tags] or "from_tcp" in [tags] { + # Try dissect first (fast) and fall back to grok + dissect { + id => "f_dissect_syslogish" + mapping => { + "message" => "%{syslog_timestamp} %{syslog_host} %{syslog_program}[%{syslog_pid}]: %{syslog_message}" + } + tag_on_failure => ["_dissectfailure_syslogish"] + } + if "_dissectfailure_syslogish" in [tags] { + grok { + id => "f_grok_syslogish" + match => { + "message" => [ + "%{SYSLOGTIMESTAMP:syslog_timestamp} %{HOSTNAME:syslog_host} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" + ] + } + tag_on_failure => ["_grokparsefailure_syslogish"] + } + } + # If we extracted a syslog timestamp, use it + if [syslog_timestamp] { + date { + id => "f_date_syslog" + match => ["syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"] + tag_on_failure => ["_dateparsefailure_syslog"] + } + } + } + # + # key=value parsing for “flat” log lines + # + if [message] =~ "([A-Za-z0-9_.-]+)=([^\"]\\S+|\"[^\"]*\")" { + kv { + id => "f_kv_message" + source => "message" + trim_key => " " + trim_value => " " + value_split => "=" + field_split_pattern => "\s+" + tag_on_failure => ["_kvfailure_message"] + } + } + # + # Basic typing / normalization + # + mutate { + id => "f_mutate_normalize" + rename => { + "msg" => "message_short" + } + convert => { + "latency_ms" => "integer" + } + lowercase => ["level"] + } + # + # Enrichments: useragent, geoip, cidr, dns + # + if [user_agent] { + useragent { + id => "f_useragent" + source => "user_agent" + target => "user_agent_parsed" + } + } + # Canonicalize IP into source_ip if it exists elsewhere + if ![source_ip] and [source][ip] { + mutate { + id => "f_copy_source_ip" + copy => { + "[source][ip]" => "source_ip" + } + } + } + if [source_ip] { + # Tag private vs public + cidr { + id => "f_cidr_private" + address => ["%{source_ip}"] + network => ["10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"] + add_tag => ["src_private"] + } + # GeoIP typically only makes sense for public IPs, so do it only if not private-tagged + if "src_private" not in [tags] { + geoip { + id => "f_geoip" + source => "source_ip" + target => "source_geo" + } + } + # Reverse DNS lookup; replace source_ip with hostname when possible (or leave as-is) + dns { + id => "f_dns_reverse" + reverse => ["source_ip"] + action => "replace" + } + } + # + # Translate severity/level into a normalized numeric + # + translate { + id => "f_translate_level_to_severity" + source => "level" + target => "severity" + dictionary => { + "trace" => "0" + "debug" => "1" + "info" => "2" + "warn" => "3" + "error" => "4" + "fatal" => "5" + } + fallback => "2" + } + mutate { + id => "f_convert_severity_int" + convert => { + "severity" => "integer" + } + } + # + # Stable fingerprint for dedup / correlation + # + fingerprint { + id => "f_fingerprint_message" + source => ["message"] + method => "MURMUR3" + target => "[@metadata][fingerprint]" + } + # + # Example branching: treat auth-ish messages specially + # + if [syslog_program] == "sshd" or [message] =~ "(?i)failed password|authentication failure|invalid user" { + mutate { + id => "f_tag_auth" + add_tag => ["category_auth"] + add_field => { + "event.category" => "authentication" + } + } + } + else if [message] =~ "(?i)GET\\s+/health|/ready|/live" { + mutate { + id => "f_tag_health" + add_tag => ["category_healthcheck"] + add_field => { + "event.category" => "availability" + } + } + } + else { + mutate { + id => "f_tag_generic" + add_tag => ["category_generic"] + } + } + # + # Prune down noisy fields (keeps top-level essentials) + # + prune { + id => "f_prune" + whitelist_names => ["^@timestamp$", "^message$", "^message_short$", "^host$", "^source_ip$", "^source_geo$", "^severity$", "^level$", "^tags$", "^event\\..*$", "^user_agent.*$", "^syslog_.*$", "^ingest_transport$"] + } +} +output { + # Always see something in console during dev + stdout { + id => "out_stdout_rubydebug" + codec => rubydebug { + metadata => "true" + } + } + # Write to disk (great for debugging replay) + file { + id => "out_file_jsonl" + path => "/tmp/logstashui-%{+YYYY.MM.dd}.jsonl" + codec => json_lines + } + # Elasticsearch (local default) + elasticsearch { + id => "out_es_local" + hosts => ["http://localhost:9200"] + index => "logstashui-%{+YYYY.MM.dd}" + ilm_enabled => "false" + } + # Webhook back to your UI/API (example) + http { + id => "out_http_callback" + url => "http://localhost:9000/logstash/callback" + http_method => "post" + format => "json" + } + # Kafka (example) + kafka { + id => "out_kafka" + bootstrap_servers => "localhost:9092" + topic_id => "logstashui-events" + } + # Pipeline-to-pipeline (requires another pipeline with pipeline input address => "downstream") + pipeline { + id => "out_pipeline_downstream" + send_to => ["downstream"] + } +} diff --git a/tests/Common/unit/conversion_data/pipelines/test-complex3.conf b/tests/Common/unit/conversion_data/pipelines/test-complex3.conf new file mode 100644 index 0000000..83e9b91 --- /dev/null +++ b/tests/Common/unit/conversion_data/pipelines/test-complex3.conf @@ -0,0 +1,652 @@ +input { + beats { + port => "5044" + ssl => "true" + ssl_certificate => "/etc/logstash/certs/server.crt" + ssl_key => "/etc/logstash/certs/server.key" + ssl_verify_mode => "force_peer" + ssl_certificate_authorities => ["/etc/logstash/certs/ca.crt"] + codec => json + type => "beats" + tags => ["beats_ssl"] + } + http { + port => "8080" + codec => json + ssl => "true" + ssl_certificate => "/etc/logstash/certs/http.crt" + ssl_key => "/etc/logstash/certs/http.key" + threads => "4" + max_pending_requests => "100" + response_headers => { + "Content-Type" => "application/json" + } + type => "webhook" + tags => ["http_api"] + } + kafka { + bootstrap_servers => "kafka1:9092,kafka2:9092,kafka3:9092" + topics => ["app-logs", "security-events", "metrics"] + group_id => "logstash-consumer" + consumer_threads => "3" + codec => avro { + schema_uri => "http://schema-registry:8081/schemas/ids/1" + } + decorate_events => "true" + security_protocol => "SASL_SSL" + sasl_mechanism => "SCRAM-SHA-512" + sasl_jaas_config => "org.apache.kafka.common.security.scram.ScramLoginModule required username='logstash' password='${KAFKA_PASS}';" + type => "kafka" + tags => ["kafka_stream"] + } + jdbc { + jdbc_driver_library => "/usr/share/logstash/vendor/jar/jdbc/postgresql.jar" + jdbc_driver_class => "org.postgresql.Driver" + jdbc_connection_string => "jdbc:postgresql://db:5432/prod" + jdbc_user => "${DB_USER}" + jdbc_password => "${DB_PASS}" + schedule => "*/5 * * * *" + statement => "SELECT * FROM events WHERE created_at > :sql_last_value" + use_column_value => "true" + tracking_column => "created_at" + tracking_column_type => "timestamp" + type => "database" + tags => ["jdbc_poll"] + } + file { + path => ["/var/log/nginx/*.log", "/var/log/app/**/*.log"] + start_position => "beginning" + sincedb_path => "/var/lib/logstash/sincedb" + codec => multiline { + pattern => "^%{TIMESTAMP_ISO8601}" + negate => "true" + what => "previous" + max_lines => 500 + } + type => "file" + tags => ["file_input"] + } + tcp { + port => "5000" + codec => json_lines + ssl_enable => "true" + ssl_cert => "/etc/logstash/certs/tcp.crt" + ssl_key => "/etc/logstash/certs/tcp.key" + type => "tcp_json" + tags => ["tcp_secure"] + } + udp { + port => "514" + codec => cef + type => "syslog" + tags => ["syslog_udp"] + } + rabbitmq { + host => "rabbitmq" + port => "5672" + user => "${RABBIT_USER}" + password => "${RABBIT_PASS}" + queue => "logs" + exchange => "logs-exchange" + exchange_type => "topic" + key => "logs.#" + durable => "true" + codec => json + type => "rabbitmq" + tags => ["amqp"] + } + redis { + host => "redis" + port => "6379" + password => "${REDIS_PASS}" + data_type => "list" + key => "logstash:queue" + codec => json + type => "redis" + tags => ["redis_queue"] + } + s3 { + bucket => "logs-archive" + region => "us-east-1" + access_key_id => "${AWS_KEY}" + secret_access_key => "${AWS_SECRET}" + interval => "300" + codec => json_lines + type => "s3" + tags => ["s3_archive"] + } + kinesis { + kinesis_stream_name => "app-stream" + region => "us-west-2" + codec => json + type => "kinesis" + tags => ["aws_kinesis"] + } +} +filter { + if [type] == "beats" { + if [agent][type] == "filebeat" { + if [log][file][path] =~ /nginx/ { + grok { + match => { + "message" => '%{IPORHOST:client_ip} - %{DATA:user} \[%{HTTPDATE:timestamp}\] "%{WORD:method} %{DATA:path} HTTP/%{NUMBER:version}" %{NUMBER:status:int} %{NUMBER:bytes:int} "%{DATA:referrer}" "%{DATA:agent}"' + } + } + date { + match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"] + target => "@timestamp" + } + useragent { + source => "agent" + target => "ua" + } + geoip { + source => "client_ip" + target => "geo" + database => "/usr/share/GeoIP/GeoLite2-City.mmdb" + } + if [status] >= 500 { + mutate { + add_tag => ["error", "server_error"] + add_field => { + "severity" => "critical" + } + } + } + else if [status] >= 400 { + mutate { + add_tag => ["error", "client_error"] + add_field => { + "severity" => "warning" + } + } + } + ruby { + code => ' + bytes = event.get("bytes").to_i + if bytes > 10485760 + event.set("size_class", "large") + elsif bytes > 1048576 + event.set("size_class", "medium") + else + event.set("size_class", "small") + end + ' + } + fingerprint { + source => ["client_ip", "path", "timestamp"] + target => "[@metadata][fingerprint]" + method => "SHA256" + } + } + else if [log][file][path] =~ /application/ { + json { + source => "message" + target => "app" + } + if [app][level] { + translate { + field => "[app][level]" + destination => "severity_num" + dictionary => { + "DEBUG" => "1" + "INFO" => "2" + "WARN" => "3" + "ERROR" => "4" + "FATAL" => "5" + } + fallback => "2" + } + mutate { + convert => { + "severity_num" => "integer" + } + } + } + if [app][exception] { + mutate { + add_tag => ["exception"] + } + ruby { + code => ' + exc = event.get("[app][exception]") + if exc.is_a?(Hash) + event.set("exception_class", exc["class"]) + event.set("exception_msg", exc["message"]) + end + ' + } + } + } + } + else if [agent][type] == "metricbeat" { + if [system][cpu] { + ruby { + code => ' + cpu = event.get("[system][cpu]") + if cpu && cpu["cores"] + total = 0.0 + cpu["cores"].each { |c| total += c["user"]["pct"].to_f if c["user"] } + avg = total / cpu["cores"].length + event.set("[system][cpu][avg_pct]", avg.round(2)) + event.tag("cpu_warning") if avg > 75 + event.tag("cpu_critical") if avg > 90 + end + ' + } + } + if [system][memory][actual][used][pct] { + ruby { + code => ' + pct = event.get("[system][memory][actual][used][pct]").to_f * 100 + event.set("mem_used_pct", pct.round(2)) + event.tag("memory_warning") if pct > 85 + event.tag("memory_critical") if pct > 95 + ' + } + } + } + } + else if [type] == "kafka" { + if [kafka][topic] == "security-events" { + json { + source => "message" + target => "security" + } + if [security][ip] { + cidr { + address => ["%{[security][ip]}"] + network => ["10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"] + add_tag => ["internal_ip"] + } + if "internal_ip" not in [tags] { + geoip { + source => "[security][ip]" + target => "threat_geo" + database => "/usr/share/GeoIP/GeoLite2-City.mmdb" + } + geoip { + source => "[security][ip]" + target => "threat_asn" + database => "/usr/share/GeoIP/GeoLite2-ASN.mmdb" + default_database_type => "ASN" + } + } + } + if [security][event_type] { + translate { + field => "[security][event_type]" + destination => "threat_score" + dictionary => { + "brute_force" => "75" + "sql_injection" => "90" + "xss" => "85" + "unauthorized" => "80" + "privilege_escalation" => "95" + "malware" => "100" + } + fallback => "50" + } + mutate { + convert => { + "threat_score" => "integer" + } + } + if [threat_score] >= 90 { + mutate { + add_tag => ["critical_threat"] + } + } + } + ruby { + code => ' + score = event.get("threat_score").to_i + is_ext = event.get("tags").include?("internal_ip") ? 0 : 20 + composite = score + is_ext + event.set("composite_risk", [composite, 100].min) + + if composite >= 100 + event.set("risk", "critical") + elsif composite >= 80 + event.set("risk", "high") + elsif composite >= 60 + event.set("risk", "medium") + else + event.set("risk", "low") + end + ' + } + } + else if [kafka][topic] == "metrics" { + dissect { + mapping => { + "metric" => "%{env}.%{dc}.%{host}.%{service}.%{type}.%{name}" + } + } + if [type] == "response_time" { + ruby { + code => ' + val = event.get("value").to_f + if val > 5000 + event.set("perf_status", "critical") + elsif val > 2000 + event.set("perf_status", "slow") + else + event.set("perf_status", "normal") + end + ' + } + } + aggregate { + task_id => "%{service}_%{name}" + code => ' + map["count"] ||= 0 + map["sum"] ||= 0.0 + map["min"] ||= Float::INFINITY + map["max"] ||= -Float::INFINITY + + val = event.get("value").to_f + map["count"] += 1 + map["sum"] += val + map["min"] = [map["min"], val].min + map["max"] = [map["max"], val].max + + avg = map["sum"] / map["count"] + event.set("rolling_avg", avg.round(2)) + event.set("rolling_min", map["min"]) + event.set("rolling_max", map["max"]) + ' + timeout => "300" + } + } + } + else if [type] == "database" { + if [event_data] { + json { + source => "event_data" + target => "evt" + } + } + date { + match => ["created_at", "ISO8601", "yyyy-MM-dd HH:mm:ss"] + target => "@timestamp" + } + elasticsearch { + hosts => ["http://elasticsearch:9200"] + index => "user-profiles" + query_template => "user_lookup.json" + fields => { + "department" => "user_dept" + "role" => "user_role" + } + } + if [event_type] =~ /^(login|logout|password_change)$/ { + mutate { + add_tag => ["auth_event"] + } + } + else if [event_type] =~ /^(create|update|delete)$/ { + mutate { + add_tag => ["data_operation"] + } + } + } + else if [type] == "webhook" { + if [headers][user-agent] { + useragent { + source => "[headers][user-agent]" + target => "webhook_ua" + } + } + if [headers][x-signature] { + ruby { + init => 'require "openssl"' + code => ' + sig = event.get("[headers][x-signature]") + payload = event.get("message").to_s + secret = ENV["WEBHOOK_SECRET"] + expected = "sha256=" + OpenSSL::HMAC.hexdigest("SHA256", secret, payload) + + if sig == expected + event.set("sig_valid", true) + else + event.set("sig_valid", false) + event.tag("invalid_signature") + end + ' + } + } + aggregate { + task_id => "%{[headers][x-forwarded-for]}" + code => ' + map["count"] ||= 0 + map["count"] += 1 + event.set("request_count", map["count"]) + ' + timeout => "60" + } + if [request_count] and [request_count] > 100 { + mutate { + add_tag => ["rate_limit_exceeded"] + } + } + } + if [message] =~ /=/ { + kv { + source => "message" + field_split => "&" + value_split => "=" + target => "parsed" + } + } + if [user_agent] and ![ua] { + useragent { + source => "user_agent" + target => "ua" + } + } + if [source_ip] { + dns { + reverse => ["source_ip"] + action => "append" + nameserver => ["8.8.8.8"] + hit_cache_size => "10000" + hit_cache_ttl => "3600" + } + } + prune { + whitelist_names => ["^@", "^_", "type", "tags", "message"] + } + mutate { + add_field => { + "env" => "${ENVIRONMENT:prod}" + "cluster" => "${CLUSTER:default}" + } + remove_field => ["@version"] + } + if [env] == "production" and ([tags] and "debug" in [tags]) { + drop { + } + } + throttle { + before_count => "3" + after_count => "1" + period => "60" + key => "%{fingerprint}" + add_tag => ["throttled"] + } + if "critical_threat" in [tags] { + clone { + clones => ["siem"] + add_field => { + "cloned" => "true" + } + } + } + metrics { + meter => ["events"] + add_tag => ["metric"] + flush_interval => "30" + rates => [1, 5, 15] + } + if ([severity] == "critical" or [severity] == "error") and ([status] >= 500 or [threat_score] >= 90) { + mutate { + add_field => { + "priority" => "P1" + "oncall" => "true" + } + add_tag => ["p1"] + } + } + else if ([severity] == "warning") and ([status] >= 400 or [threat_score] >= 70) { + mutate { + add_field => { + "priority" => "P2" + } + add_tag => ["p2"] + } + } + fingerprint { + source => "message" + target => "event_hash" + method => "MURMUR3" + } + uuid { + target => "event_id" + } + ruby { + code => ' + event.set("processed_at", Time.now.utc.iso8601) + event.set("pipeline_v", "2.0") + ' + } +} +output { + if "throttled" not in [tags] and "metric" not in [tags] { + elasticsearch { + hosts => ["https://es1:9200", "https://es2:9200"] + user => "${ES_USER}" + password => "${ES_PASS}" + ssl => "true" + cacert => "/etc/logstash/certs/ca.crt" + index => "%{type}-%{+YYYY.MM.dd}" + document_id => "%{event_id}" + pipeline => "enrich" + ilm_enabled => "true" + ilm_rollover_alias => "%{type}" + ilm_pattern => "{now/d}-000001" + ilm_policy => "logs-policy" + http_compression => "true" + } + } + if "siem" in [tags] { + elasticsearch { + hosts => ["https://siem-es:9200"] + user => "${SIEM_USER}" + password => "${SIEM_PASS}" + ssl => "true" + cacert => "/etc/logstash/certs/siem-ca.crt" + index => "security-%{+YYYY.MM.dd}" + } + } + if [env] == "production" { + s3 { + access_key_id => "${AWS_KEY}" + secret_access_key => "${AWS_SECRET}" + region => "us-east-1" + bucket => "logs-archive" + size_file => "104857600" + time_file => "15" + codec => json_lines + prefix => "logs/%{type}/year=%{+YYYY}/month=%{+MM}/day=%{+dd}" + encoding => "gzip" + server_side_encryption => "true" + } + } + kafka { + bootstrap_servers => "kafka1:9092,kafka2:9092" + topic_id => "processed-%{type}" + codec => json + compression_type => "snappy" + acks => "all" + security_protocol => "SASL_SSL" + sasl_mechanism => "SCRAM-SHA-512" + sasl_jaas_config => "org.apache.kafka.common.security.scram.ScramLoginModule required username='${KAFKA_USER}' password='${KAFKA_PASS}';" + } + if "p1" in [tags] or "critical" in [tags] { + redis { + host => ["redis1", "redis2"] + port => "26379" + password => "${REDIS_PASS}" + data_type => "list" + key => "alerts:critical" + } + http { + url => "https://alerts.example.com/api/events" + http_method => "post" + format => "json" + headers => { + "Authorization" => "Bearer ${ALERT_TOKEN}" + "Content-Type" => "application/json" + } + automatic_retries => "3" + } + } + if [env] != "production" { + file { + path => "/var/log/logstash/debug-%{type}.log" + codec => json_lines + } + } + if "metric" in [tags] { + graphite { + host => "graphite" + port => "2003" + metrics_format => "logstash.%{env}.%{type}.count" + fields_are_metrics => "true" + } + influxdb { + host => "influxdb" + port => "8086" + db => "metrics" + user => "${INFLUX_USER}" + password => "${INFLUX_PASS}" + measurement => "%{type}_metrics" + use_event_fields_for_data_points => "true" + } + } + if [type] == "rabbitmq" { + mongodb { + uri => "mongodb://${MONGO_USER}:${MONGO_PASS}@mongo:27017/logs" + database => "logs" + collection => "%{type}" + isodate => "true" + bulk => "true" + bulk_size => "100" + } + } + if "p1" in [tags] { + email { + to => "oncall@example.com" + from => "alerts@example.com" + subject => "P1 Alert: %{type}" + body => 'Event: %{event_id} + Time: %{@timestamp} + Severity: %{severity} + Message: %{message}' + address => "smtp.example.com" + port => "587" + use_tls => "true" + username => "${SMTP_USER}" + password => "${SMTP_PASS}" + } + } + tcp { + host => "logstash-secondary" + port => "5005" + codec => json_lines + } + stdout { + codec => rubydebug + } +} diff --git a/tests/Common/unit/conversion_data/pipelines/test-data-types.conf b/tests/Common/unit/conversion_data/pipelines/test-data-types.conf new file mode 100644 index 0000000..94180f6 --- /dev/null +++ b/tests/Common/unit/conversion_data/pipelines/test-data-types.conf @@ -0,0 +1,33 @@ +input { +} +filter { + grok { + match => { + "test" => "test" + } + pattern_definitions => { + "1" => "2" + "test" => "test" + "asd" => "asf" + } + patterns_dir => ["test"] + tag_on_failure => [] + } + grok { + match => { + "test" => [ + "test", + "test2" + ] + } + pattern_definitions => { + "test" => "test" + } + patterns_dir => "test" + } + # test + # multi + # row +} +output { +} diff --git a/tests/Common/unit/conversion_data/pipelines/test-datatypes.conf b/tests/Common/unit/conversion_data/pipelines/test-datatypes.conf new file mode 100644 index 0000000..94180f6 --- /dev/null +++ b/tests/Common/unit/conversion_data/pipelines/test-datatypes.conf @@ -0,0 +1,33 @@ +input { +} +filter { + grok { + match => { + "test" => "test" + } + pattern_definitions => { + "1" => "2" + "test" => "test" + "asd" => "asf" + } + patterns_dir => ["test"] + tag_on_failure => [] + } + grok { + match => { + "test" => [ + "test", + "test2" + ] + } + pattern_definitions => { + "test" => "test" + } + patterns_dir => "test" + } + # test + # multi + # row +} +output { +} diff --git a/tests/Common/unit/conversion_data/pipelines/test-devopsschool-1.conf b/tests/Common/unit/conversion_data/pipelines/test-devopsschool-1.conf new file mode 100644 index 0000000..6d068c5 --- /dev/null +++ b/tests/Common/unit/conversion_data/pipelines/test-devopsschool-1.conf @@ -0,0 +1,21 @@ +input { + beats { + port => "5044" + } +} +filter { + grok { + match => { + "message" => "%{COMBINEDAPACHELOG}" + } + } +} +output { + elasticsearch { + hosts => ["http://elasticsearch:9200"] + index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" + } + stdout { + codec => rubydebug + } +} diff --git a/tests/Common/unit/conversion_data/pipelines/test-devopsschool-2.conf b/tests/Common/unit/conversion_data/pipelines/test-devopsschool-2.conf new file mode 100644 index 0000000..bd9f369 --- /dev/null +++ b/tests/Common/unit/conversion_data/pipelines/test-devopsschool-2.conf @@ -0,0 +1,17 @@ +input { + beats { + port => "5044" + } +} +filter { + grok { + match => { + "message" => "%{SYSLOGLINE}" + } + } +} +output { + stdout { + codec => rubydebug + } +} diff --git a/tests/Common/unit/conversion_data/pipelines/test-devopsschool-4.conf b/tests/Common/unit/conversion_data/pipelines/test-devopsschool-4.conf new file mode 100644 index 0000000..4e32410 --- /dev/null +++ b/tests/Common/unit/conversion_data/pipelines/test-devopsschool-4.conf @@ -0,0 +1,25 @@ +input { + file { + path => "/var/log/apache2/access.log" + start_position => "beginning" + sincedb_path => "/dev/null" + } +} +filter { + grok { + match => { + "message" => "%{COMBINEDAPACHELOG}" + } + } + date { + match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"] + } + geoip { + source => "clientip" + } +} +output { + elasticsearch { + hosts => ["localhost:9200"] + } +} diff --git a/tests/Common/unit/conversion_data/pipelines/test-devopsschool-5.conf b/tests/Common/unit/conversion_data/pipelines/test-devopsschool-5.conf new file mode 100644 index 0000000..87350a1 --- /dev/null +++ b/tests/Common/unit/conversion_data/pipelines/test-devopsschool-5.conf @@ -0,0 +1,20 @@ +input { + beats { + port => "5044" + } +} +filter { + grok { + match => { + "message" => "%{COMBINEDAPACHELOG}" + } + } + geoip { + source => "clientip" + } +} +output { + elasticsearch { + hosts => ["localhost:9200"] + } +} diff --git a/tests/Common/unit/conversion_data/pipelines/test-elasticdocs-apache.conf b/tests/Common/unit/conversion_data/pipelines/test-elasticdocs-apache.conf new file mode 100644 index 0000000..9f87b11 --- /dev/null +++ b/tests/Common/unit/conversion_data/pipelines/test-elasticdocs-apache.conf @@ -0,0 +1,31 @@ +input { + file { + path => "/tmp/access_log" + start_position => "beginning" + } +} +filter { + if [path] =~ "access" { + mutate { + replace => { + "type" => "apache_access" + } + } + grok { + match => { + "message" => "%{COMBINEDAPACHELOG}" + } + } + } + date { + match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"] + } +} +output { + elasticsearch { + hosts => ["localhost:9200"] + } + stdout { + codec => rubydebug + } +} diff --git a/tests/Common/unit/conversion_data/pipelines/test-elasticdocs-configuring_filters.conf b/tests/Common/unit/conversion_data/pipelines/test-elasticdocs-configuring_filters.conf new file mode 100644 index 0000000..24d2f57 --- /dev/null +++ b/tests/Common/unit/conversion_data/pipelines/test-elasticdocs-configuring_filters.conf @@ -0,0 +1,22 @@ +input { + stdin { + } +} +filter { + grok { + match => { + "message" => "%{COMBINEDAPACHELOG}" + } + } + date { + match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"] + } +} +output { + elasticsearch { + hosts => ["localhost:9200"] + } + stdout { + codec => rubydebug + } +} diff --git a/tests/Common/unit/conversion_data/pipelines/test-elasticdocs-syslog.conf b/tests/Common/unit/conversion_data/pipelines/test-elasticdocs-syslog.conf new file mode 100644 index 0000000..761797d --- /dev/null +++ b/tests/Common/unit/conversion_data/pipelines/test-elasticdocs-syslog.conf @@ -0,0 +1,31 @@ +input { + tcp { + port => "5000" + type => "syslog" + } + udp { + port => "5000" + type => "syslog" + } +} +filter { + if [type] == "syslog" { + grok { + match => { + "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" + } + add_field => ["received_at", "%{@timestamp}", "received_from", "%{host}"] + } + date { + match => ["syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"] + } + } +} +output { + elasticsearch { + hosts => ["localhost:9200"] + } + stdout { + codec => rubydebug + } +} diff --git a/tests/Common/unit/conversion_data/pipelines/test-es-input.conf b/tests/Common/unit/conversion_data/pipelines/test-es-input.conf new file mode 100644 index 0000000..a0771af --- /dev/null +++ b/tests/Common/unit/conversion_data/pipelines/test-es-input.conf @@ -0,0 +1,26 @@ +input { + elasticsearch { + api_key => "test" + cloud_id => "test" + index => "kibana_sample_data_ecommerce" + query => '{"query":{"match_all":{}}}' + slices => "6" + ssl_enabled => "true" + connect_timeout_seconds => "120" + request_timeout_seconds => "600" + socket_timeout_seconds => "600" + } +} +filter { + mutate { + add_field => { + "test" => "test" + } + } +} +output { + csv { + fields => ["test"] + path => "/home/ubuntu/test.csv" + } +} diff --git a/tests/Common/unit/conversion_data/pipelines/test-ls-repo-mysql.conf b/tests/Common/unit/conversion_data/pipelines/test-ls-repo-mysql.conf new file mode 100644 index 0000000..8b3a968 --- /dev/null +++ b/tests/Common/unit/conversion_data/pipelines/test-ls-repo-mysql.conf @@ -0,0 +1,69 @@ +input { + beats { + port => 5044 + host => "0.0.0.0" + } +} +filter { + if [fileset][module] == "mysql" { + if [fileset][name] == "error" { + grok { + match => { + "message" => [ + "%{LOCALDATETIME:[mysql][error][timestamp]} (\[%{DATA:[mysql][error][level]}\] )?%{GREEDYDATA:[mysql][error][message]}", + "%{TIMESTAMP_ISO8601:[mysql][error][timestamp]} %{NUMBER:[mysql][error][thread_id]} \[%{DATA:[mysql][error][level]}\] %{GREEDYDATA:[mysql][error][message1]}", + "%{GREEDYDATA:[mysql][error][message2]}" + ] + } + pattern_definitions => { + "LOCALDATETIME" => "[0-9]+ %{TIME}" + } + remove_field => "message" + } + mutate { + rename => { + "[mysql][error][message1]" => "[mysql][error][message]" + } + } + mutate { + rename => { + "[mysql][error][message2]" => "[mysql][error][message]" + } + } + date { + match => ["[mysql][error][timestamp]", "ISO8601", "YYMMdd H:m:s"] + remove_field => "[mysql][error][time]" + } + } + else if [fileset][name] == "slowlog" { + grok { + match => { + "message" => [ + '^# User@Host: %{USER:[mysql][slowlog][user]}(\[[^\]]+\])? @ %{HOSTNAME:[mysql][slowlog][host]} \[(IP:[mysql][slowlog][ip])?\](\s*Id:\s* %{NUMBER:[mysql][slowlog][id]})? + # Query_time: %{NUMBER:[mysql][slowlog][query_time][sec]}\s* Lock_time: %{NUMBER:[mysql][slowlog][lock_time][sec]}\s* Rows_sent: %{NUMBER:[mysql][slowlog][rows_sent]}\s* Rows_examined: %{NUMBER:[mysql][slowlog][rows_examined]} + (SET timestamp=%{NUMBER:[mysql][slowlog][timestamp]}; + )?%{GREEDYMULTILINE:[mysql][slowlog][query]}' + ] + } + pattern_definitions => { + "GREEDYMULTILINE" => '(.| + )*' + } + remove_field => "message" + } + date { + match => ["[mysql][slowlog][timestamp]", "UNIX"] + } + mutate { + gsub => ["[mysql][slowlog][query]", "\n# Time: [0-9]+ [0-9][0-9]:[0-9][0-9]:[0-9][0-9](\\.[0-9]+)?$", ""] + } + } + } +} +output { + elasticsearch { + hosts => "localhost" + manage_template => "false" + index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" + } +} diff --git a/tests/Common/unit/conversion_data/pipelines/test-ls-repo-nginx.conf b/tests/Common/unit/conversion_data/pipelines/test-ls-repo-nginx.conf new file mode 100644 index 0000000..8956944 --- /dev/null +++ b/tests/Common/unit/conversion_data/pipelines/test-ls-repo-nginx.conf @@ -0,0 +1,64 @@ +input { + beats { + port => 5044 + host => "0.0.0.0" + } +} +filter { + if [fileset][module] == "nginx" { + if [fileset][name] == "access" { + grok { + match => { + "message" => [ + '%{IPORHOST:[nginx][access][remote_ip]} - %{DATA:[nginx][access][user_name]} \[%{HTTPDATE:[nginx][access][time]}\] "%{WORD:[nginx][access][method]} %{DATA:[nginx][access][url]} HTTP/%{NUMBER:[nginx][access][http_version]}" %{NUMBER:[nginx][access][response_code]} %{NUMBER:[nginx][access][body_sent][bytes]} "%{DATA:[nginx][access][referrer]}" "%{DATA:[nginx][access][agent]}"' + ] + } + remove_field => "message" + } + mutate { + add_field => { + "read_timestamp" => "%{@timestamp}" + } + } + date { + match => ["[nginx][access][time]", "dd/MMM/YYYY:H:m:s Z"] + remove_field => "[nginx][access][time]" + } + useragent { + source => "[nginx][access][agent]" + target => "[nginx][access][user_agent]" + remove_field => "[nginx][access][agent]" + } + geoip { + source => "[nginx][access][remote_ip]" + target => "[nginx][access][geoip]" + } + } + else if [fileset][name] == "error" { + grok { + match => { + "message" => [ + "%{DATA:[nginx][error][time]} \[%{DATA:[nginx][error][level]}\] %{NUMBER:[nginx][error][pid]}#%{NUMBER:[nginx][error][tid]}: (\*%{NUMBER:[nginx][error][connection_id]} )?%{GREEDYDATA:[nginx][error][message]}" + ] + } + remove_field => "message" + } + mutate { + rename => { + "@timestamp" => "read_timestamp" + } + } + date { + match => ["[nginx][error][time]", "YYYY/MM/dd H:m:s"] + remove_field => "[nginx][error][time]" + } + } + } +} +output { + elasticsearch { + hosts => "localhost" + manage_template => "false" + index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" + } +} diff --git a/tests/Common/unit/conversion_data/pipelines/test-ls-repo-system.conf b/tests/Common/unit/conversion_data/pipelines/test-ls-repo-system.conf new file mode 100644 index 0000000..ca865b2 --- /dev/null +++ b/tests/Common/unit/conversion_data/pipelines/test-ls-repo-system.conf @@ -0,0 +1,61 @@ +input { + beats { + port => 5044 + host => "0.0.0.0" + } +} +filter { + if [fileset][module] == "system" { + if [fileset][name] == "auth" { + grok { + match => { + "message" => [ + "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: %{DATA:[system][auth][ssh][event]} %{DATA:[system][auth][ssh][method]} for (invalid user )?%{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]} port %{NUMBER:[system][auth][ssh][port]} ssh2(: %{GREEDYDATA:[system][auth][ssh][signature]})?", + "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: %{DATA:[system][auth][ssh][event]} user %{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]}", + "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: Did not receive identification string from %{IPORHOST:[system][auth][ssh][dropped_ip]}", + "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sudo(?:\[%{POSINT:[system][auth][pid]}\])?: \s*%{DATA:[system][auth][user]} :( %{DATA:[system][auth][sudo][error]} ;)? TTY=%{DATA:[system][auth][sudo][tty]} ; PWD=%{DATA:[system][auth][sudo][pwd]} ; USER=%{DATA:[system][auth][sudo][user]} ; COMMAND=%{GREEDYDATA:[system][auth][sudo][command]}", + "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} groupadd(?:\[%{POSINT:[system][auth][pid]}\])?: new group: name=%{DATA:system.auth.groupadd.name}, GID=%{NUMBER:system.auth.groupadd.gid}", + "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} useradd(?:\[%{POSINT:[system][auth][pid]}\])?: new user: name=%{DATA:[system][auth][useradd][name]}, UID=%{NUMBER:[system][auth][useradd][uid]}, GID=%{NUMBER:[system][auth][useradd][gid]}, home=%{DATA:[system][auth][useradd][home]}, shell=%{DATA:[system][auth][useradd][shell]}$", + "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} %{DATA:[system][auth][program]}(?:\[%{POSINT:[system][auth][pid]}\])?: %{GREEDYMULTILINE:[system][auth][message]}" + ] + } + pattern_definitions => { + "GREEDYMULTILINE" => '(.| + )*' + } + remove_field => "message" + } + date { + match => ["[system][auth][timestamp]", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"] + } + geoip { + source => "[system][auth][ssh][ip]" + target => "[system][auth][ssh][geoip]" + } + } + else if [fileset][name] == "syslog" { + grok { + match => { + "message" => [ + "%{SYSLOGTIMESTAMP:[system][syslog][timestamp]} %{SYSLOGHOST:[system][syslog][hostname]} %{DATA:[system][syslog][program]}(?:\[%{POSINT:[system][syslog][pid]}\])?: %{GREEDYMULTILINE:[system][syslog][message]}" + ] + } + pattern_definitions => { + "GREEDYMULTILINE" => '(.| + )*' + } + remove_field => "message" + } + date { + match => ["[system][syslog][timestamp]", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"] + } + } + } +} +output { + elasticsearch { + hosts => "localhost" + manage_template => "false" + index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" + } +} diff --git a/tests/Common/unit/conversion_data/pipelines/test-multiline-ruby-with-hash.conf b/tests/Common/unit/conversion_data/pipelines/test-multiline-ruby-with-hash.conf new file mode 100644 index 0000000..ee59e51 --- /dev/null +++ b/tests/Common/unit/conversion_data/pipelines/test-multiline-ruby-with-hash.conf @@ -0,0 +1,49 @@ +input { + stdin { + codec => line + } +} +filter { + ruby { + code => ' + require "digest" + msg = event.get("message").to_s + event.set("hash", Digest::MD5.hexdigest(msg)) + # this # is NOT a comment — it is inside a single-quoted string + if event.get("level") == "ERROR" + event.set("alert", true) + event.set("severity", "high") + elsif event.get("level") == "WARN" + event.set("severity", "medium") + else + event.set("severity", "low") + end + # another hash # mark inside the string — still not a comment + ' + } + ruby { + init => ' + require "openssl" + require "base64" + # init comment inside single-quoted string + @secret = ENV["SIGNING_SECRET"] || "default" + ' + code => ' + payload = event.get("message").to_s + sig = Base64.strict_encode64( + OpenSSL::HMAC.digest("SHA256", @secret, payload) + ) + event.set("signature", sig) + ' + } + mutate { + add_field => { + "pipeline" => "ruby-test" + } + } +} +output { + stdout { + codec => rubydebug + } +} diff --git a/tests/Common/unit/conversion_data/pipelines/test-nested-conditionals-comments.conf b/tests/Common/unit/conversion_data/pipelines/test-nested-conditionals-comments.conf new file mode 100644 index 0000000..207efc1 --- /dev/null +++ b/tests/Common/unit/conversion_data/pipelines/test-nested-conditionals-comments.conf @@ -0,0 +1,71 @@ +input { + stdin { + } +} +filter { + # Top-level comment before any conditional + if [type] == "web" { + # Comment inside first if branch + if [status] >= 500 { + # Comment inside nested if + mutate { + add_tag => ["server_error"] + add_field => { + "severity" => "high" + } + } + # Comment after plugin inside nested if + if [status] == 503 { + mutate { + add_tag => ["service_unavailable"] + } + } + # Trailing comment inside nested if + } + else if [status] >= 400 { + # Comment inside else-if + mutate { + add_tag => ["client_error"] + add_field => { + "severity" => "medium" + } + } + # Trailing comment inside else-if + } + else { + # Comment inside else + mutate { + add_tag => ["success"] + add_field => { + "severity" => "low" + } + } + } + # Comment at end of outer if block + } + else if [type] == "db" { + # Comment at start of else-if block + mutate { + add_field => { + "source" => "database" + } + } + } + else { + # Comment in final else + drop { + } + } + # Comment between conditional and next plugin at section level + mutate { + add_field => { + "processed_by" => "logstash" + } + } + # Trailing section-level comment +} +output { + stdout { + codec => rubydebug + } +} diff --git a/tests/Common/unit/conversion_data/pipelines/test-regex-conditions.conf b/tests/Common/unit/conversion_data/pipelines/test-regex-conditions.conf new file mode 100644 index 0000000..b9e6306 --- /dev/null +++ b/tests/Common/unit/conversion_data/pipelines/test-regex-conditions.conf @@ -0,0 +1,62 @@ +input { + syslog { + port => 514 + } +} +filter { + if [message] =~ /^ERROR/ { + mutate { + add_tag => ["error"] + } + } + if [message] =~ /\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}/ { + date { + match => ["message", "ISO8601"] + target => "@timestamp" + } + } + if [host] =~ /^(web|app|db)-\d+\.example\.com$/ { + mutate { + add_field => { + "internal" => "true" + } + } + } + if [message] !~ /^$/ { + grok { + match => { + "message" => "%{GREEDYDATA:content}" + } + } + } + if [path] =~ /\/api\/v[12]\// and [method] == "POST" { + mutate { + add_tag => ["api_write"] + } + } + if [status] =~ /^5\d\d$/ { + mutate { + add_tag => ["server_error"] + } + } + else if [status] =~ /^4\d\d$/ { + mutate { + add_tag => ["client_error"] + } + } + if [user_agent] =~ /(?i)bot|crawler|spider/ { + drop { + } + } +} +output { + if "error" in [tags] { + file { + path => "/var/log/errors.log" + codec => json + } + } + stdout { + codec => rubydebug + } +} diff --git a/tests/Common/unit/conversion_data/pipelines/test-sample-nginx.conf b/tests/Common/unit/conversion_data/pipelines/test-sample-nginx.conf new file mode 100644 index 0000000..95ff7f9 --- /dev/null +++ b/tests/Common/unit/conversion_data/pipelines/test-sample-nginx.conf @@ -0,0 +1,67 @@ +input { + stdin { + codec => line + } +} +filter { + mutate { + add_field => { + "event.dataset" => "nginx.access" + "service.name" => "nginx" + } + } + grok { + match => { + "message" => [ + '%{IPORHOST:source.address} %{DATA:nginx.ident} %{DATA:user.name} \[%{HTTPDATE:nginx.access.time}\] "%{WORD:http.request.method} %{DATA:url.original} HTTP/%{NUMBER:http.version}" %{NUMBER:http.response.status_code:int} (?:%{NUMBER:http.response.body.bytes:int}|-) "%{DATA:http.request.referrer}" "%{DATA:user_agent.original}" %{NUMBER:nginx.access.request_time:float}', + '%{IPORHOST:source.address} %{DATA:nginx.ident} %{DATA:user.name} \[%{HTTPDATE:nginx.access.time}\] "%{WORD:http.request.method} %{DATA:url.original} HTTP/%{NUMBER:http.version}" %{NUMBER:http.response.status_code:int} (?:%{NUMBER:http.response.body.bytes:int}|-) "%{DATA:http.request.referrer}" "%{DATA:user_agent.original}"', + '%{IPORHOST:source.address} %{DATA:nginx.ident} %{DATA:user.name} \[%{HTTPDATE:nginx.access.time}\] "%{WORD:http.request.method} %{DATA:url.original} HTTP/%{NUMBER:http.version}" %{NUMBER:http.response.status_code:int} (?:%{NUMBER:http.response.body.bytes:int}|-) "%{DATA:http.request.referrer}" "%{DATA:user_agent.original}" (?:rt=%{NUMBER:nginx.access.request_time:float}\s+uct=%{NUMBER:nginx.access.upstream_connect_time:float}\s+uht=%{NUMBER:nginx.access.upstream_header_time:float}\s+urt=%{NUMBER:nginx.access.upstream_response_time:float})' + ] + } + tag_on_failure => ["_grok_nginx_access_fail"] + } + date { + match => ["nginx.access.time", "dd/MMM/yyyy:HH:mm:ss Z"] + target => "@timestamp" + } + urldecode { + field => "url.original" + } + dissect { + mapping => { + "url.original" => "%{url.path}?%{url.query}" + } + } + useragent { + source => "user_agent.original" + target => "user_agent" + } + mutate { + copy => { + "source.address" => "source.ip" + } + } + geoip { + source => "source.ip" + target => "source.geo" + tag_on_failure => ["_geoip_fail"] + } + mutate { + gsub => ["http.request.referrer", "^-$", "", "user.name", "^-$", ""] + } + if [http][response][status_code] and [http][response][status_code] >= 500 { + mutate { + add_tag => ["nginx_server_error"] + } + } + else if [http][response][status_code] and [http][response][status_code] >= 400 { + mutate { + add_tag => ["nginx_client_error"] + } + } +} +output { + stdout { + codec => rubydebug + } +} diff --git a/tests/Common/unit/conversion_data/pipelines/test-snmp-v0.2.conf b/tests/Common/unit/conversion_data/pipelines/test-snmp-v0.2.conf new file mode 100644 index 0000000..007f0b1 --- /dev/null +++ b/tests/Common/unit/conversion_data/pipelines/test-snmp-v0.2.conf @@ -0,0 +1,213 @@ +input { + snmp { + hosts => [ + { host => "udp:1.2.3.4/161" version => "3" timeout => 1000 retries => 2 } + ] + interval => "30" + security_name => "test" + security_level => "authPriv" + ecs_compatibility => "disabled" + oid_mapping_format => "dotted_string" + auth_protocol => "sha" + auth_pass => "test" + priv_protocol => "aes" + priv_pass => "test" + get => ["1.3.6.1.4.1.9.2.1.57.0", "1.3.6.1.4.1.9.9.48.1.1.1.5.1", "1.3.6.1.4.1.9.9.48.1.1.1.6.1", "1.3.6.1.2.1.1.1.0", "1.3.6.1.2.1.1.5.0", "1.3.6.1.2.1.1.2.0", "1.3.6.1.2.1.1.3.0"] + tables => [ + { name => "cdpCacheTable" columns => ["1.3.6.1.4.1.9.9.23.1.2.1.1.1", "1.3.6.1.4.1.9.9.23.1.2.1.1.6", "1.3.6.1.4.1.9.9.23.1.2.1.1.7", "1.3.6.1.4.1.9.9.23.1.2.1.1.8", "1.3.6.1.4.1.9.9.23.1.2.1.1.9", "1.3.6.1.4.1.9.9.23.1.2.1.1.5", "1.3.6.1.4.1.9.9.23.1.2.1.1.4"] }, + { name => "sensors" columns => ["1.3.6.1.4.1.9.9.13.1.3.1.2", "1.3.6.1.4.1.9.9.13.1.3.1.3", "1.3.6.1.4.1.9.9.13.1.3.1.4", "1.3.6.1.4.1.9.9.13.1.3.1.5", "1.3.6.1.4.1.9.9.13.1.3.1.6"] }, + { name => "fans" columns => ["1.3.6.1.4.1.9.9.13.1.4.1.2", "1.3.6.1.4.1.9.9.13.1.4.1.3"] }, + { name => "interfaces" columns => ["1.3.6.1.2.1.2.2.1.1", "1.3.6.1.2.1.2.2.1.2", "1.3.6.1.2.1.2.2.1.3", "1.3.6.1.2.1.2.2.1.7", "1.3.6.1.2.1.2.2.1.8", "1.3.6.1.2.1.31.1.1.1.1", "1.3.6.1.2.1.31.1.1.1.18", "1.3.6.1.2.1.31.1.1.1.15", "1.3.6.1.2.1.2.2.1.5", "1.3.6.1.2.1.2.2.1.6", "1.3.6.1.2.1.2.2.1.4", "1.3.6.1.2.1.31.1.1.1.6", "1.3.6.1.2.1.31.1.1.1.10", "1.3.6.1.2.1.31.1.1.1.9", "1.3.6.1.2.1.31.1.1.1.13", "1.3.6.1.2.1.31.1.1.1.7", "1.3.6.1.2.1.31.1.1.1.11", "1.3.6.1.2.1.31.1.1.1.8", "1.3.6.1.2.1.31.1.1.1.12", "1.3.6.1.2.1.2.2.1.9", "1.3.6.1.2.1.17.7.1.4.5.1.1", "1.3.6.1.2.1.2.2.1.14", "1.3.6.1.2.1.2.2.1.20", "1.3.6.1.2.1.2.2.1.13", "1.3.6.1.2.1.2.2.1.19"] } + ] + } +} +filter { + mutate { + rename => { + "host" => "[host][hostname]" + } + } + mutate { + rename => { + "1.3.6.1.4.1.9.2.1.57.0" => "[system][cpu][total][norm][pct]" + "1.3.6.1.4.1.9.9.48.1.1.1.5.1" => "[system][memory][actual][used][bytes]" + "1.3.6.1.4.1.9.9.48.1.1.1.6.1" => "[system][memory][actual][free][bytes]" + "1.3.6.1.2.1.1.1.0" => "[host][description]" + "1.3.6.1.2.1.1.5.0" => "[host][name]" + "1.3.6.1.2.1.1.2.0" => "[host][id]" + "1.3.6.1.2.1.1.3.0" => "[host][uptime]" + } + } + mutate { + add_field => { + "[network][name]" => "home-segment-1 (192.168.4.0/24)" + "[metricset][module]" => "system" + } + } + ruby { + code => ' v = event.get("[system][cpu][total][norm][pct]") + if v + event.set("[system][cpu][total][norm][pct]", v.to_f / 100.0) + end' + } + ruby { + code => ' + used = event.get("[system][memory][actual][used][bytes]") + free = event.get("[system][memory][actual][free][bytes]") + + if used && free + used_f = used.to_f + free_f = free.to_f + total_f = used_f + free_f + + if total_f > 0 + event.set("[system][memory][total]", total_f) + event.set("[system][memory][actual][used][pct]", (used_f / total_f)) + event.set("[system][memory][actual][free][pct]", (free_f / total_f)) + end + end + ' + } + ruby { + code => 'rows = event.get(\'[cdpCacheTable]\') + if rows.is_a?(Array) + host_name = event.get(\'[host][name]\') + host_hostname = event.get(\'[host][hostname]\') + network_name = event.get(\'[network][name]\') + timestamp = event.get(\'@timestamp\') + rows.each do |row| + next unless row.is_a?(Hash) + row[\'cdpCacheIfIndex\'] = row.delete(\'1.3.6.1.4.1.9.9.23.1.2.1.1.1\') + row[\'cdpCacheDeviceId\'] = row.delete(\'1.3.6.1.4.1.9.9.23.1.2.1.1.6\') + row[\'cdpCacheDevicePort\'] = row.delete(\'1.3.6.1.4.1.9.9.23.1.2.1.1.7\') + row[\'cdpCachePlatform\'] = row.delete(\'1.3.6.1.4.1.9.9.23.1.2.1.1.8\') + row[\'cdpCacheCapabilities\'] = row.delete(\'1.3.6.1.4.1.9.9.23.1.2.1.1.9\') + row[\'cdpCacheVersion\'] = row.delete(\'1.3.6.1.4.1.9.9.23.1.2.1.1.5\') + row[\'cdpCacheAddress\'] = row.delete(\'1.3.6.1.4.1.9.9.23.1.2.1.1.4\') + new_event = LogStash::Event.new({ + \'@timestamp\' => timestamp, + \'host\' => { \'name\' => host_name, \'hostname\' => host_hostname }, + \'network\' => { \'name\' => network_name }, + \'table\' => row, + \'metricset\' => { \'module\' => \'snmp\' }, + \'event\' => { \'kind\' => \'cdpcachetable\' } + }) + new_event_block.call(new_event) + end + event.remove(\'[cdpCacheTable]\') + event.set(\'[event][kind]\', \'metrics\') + end' + } + ruby { + code => 'rows = event.get(\'[sensors]\') + if rows.is_a?(Array) + host_name = event.get(\'[host][name]\') + host_hostname = event.get(\'[host][hostname]\') + network_name = event.get(\'[network][name]\') + timestamp = event.get(\'@timestamp\') + rows.each do |row| + next unless row.is_a?(Hash) + row[\'description\'] = row.delete(\'1.3.6.1.4.1.9.9.13.1.3.1.2\') + row[\'temp_celsius\'] = row.delete(\'1.3.6.1.4.1.9.9.13.1.3.1.3\') + row[\'temp_threshold\'] = row.delete(\'1.3.6.1.4.1.9.9.13.1.3.1.4\') + row[\'temp_last_shutdown\'] = row.delete(\'1.3.6.1.4.1.9.9.13.1.3.1.5\') + row[\'state\'] = row.delete(\'1.3.6.1.4.1.9.9.13.1.3.1.6\') + new_event = LogStash::Event.new({ + \'@timestamp\' => timestamp, + \'host\' => { \'name\' => host_name, \'hostname\' => host_hostname }, + \'network\' => { \'name\' => network_name }, + \'table\' => row, + \'metricset\' => { \'module\' => \'snmp\' }, + \'event\' => { \'kind\' => \'sensors\' } + }) + new_event_block.call(new_event) + end + event.remove(\'[sensors]\') + event.set(\'[event][kind]\', \'metrics\') + end' + } + ruby { + code => 'rows = event.get(\'[fans]\') + if rows.is_a?(Array) + host_name = event.get(\'[host][name]\') + host_hostname = event.get(\'[host][hostname]\') + network_name = event.get(\'[network][name]\') + timestamp = event.get(\'@timestamp\') + rows.each do |row| + next unless row.is_a?(Hash) + row[\'description\'] = row.delete(\'1.3.6.1.4.1.9.9.13.1.4.1.2\') + row[\'state\'] = row.delete(\'1.3.6.1.4.1.9.9.13.1.4.1.3\') + new_event = LogStash::Event.new({ + \'@timestamp\' => timestamp, + \'host\' => { \'name\' => host_name, \'hostname\' => host_hostname }, + \'network\' => { \'name\' => network_name }, + \'table\' => row, + \'metricset\' => { \'module\' => \'snmp\' }, + \'event\' => { \'kind\' => \'fans\' } + }) + new_event_block.call(new_event) + end + event.remove(\'[fans]\') + event.set(\'[event][kind]\', \'metrics\') + end' + } + ruby { + code => 'rows = event.get(\'[interfaces]\') + if rows.is_a?(Array) + host_name = event.get(\'[host][name]\') + host_hostname = event.get(\'[host][hostname]\') + network_name = event.get(\'[network][name]\') + timestamp = event.get(\'@timestamp\') + rows.each do |row| + next unless row.is_a?(Hash) + row[\'ifIndex\'] = row.delete(\'1.3.6.1.2.1.2.2.1.1\') + row[\'ifDescr\'] = row.delete(\'1.3.6.1.2.1.2.2.1.2\') + row[\'ifType\'] = row.delete(\'1.3.6.1.2.1.2.2.1.3\') + row[\'ifAdminStatus\'] = row.delete(\'1.3.6.1.2.1.2.2.1.7\') + row[\'ifOperStatus\'] = row.delete(\'1.3.6.1.2.1.2.2.1.8\') + row[\'ifName\'] = row.delete(\'1.3.6.1.2.1.31.1.1.1.1\') + row[\'ifAlias\'] = row.delete(\'1.3.6.1.2.1.31.1.1.1.18\') + row[\'ifHighSpeed\'] = row.delete(\'1.3.6.1.2.1.31.1.1.1.15\') + row[\'ifSpeed\'] = row.delete(\'1.3.6.1.2.1.2.2.1.5\') + row[\'ifPhysAddress\'] = row.delete(\'1.3.6.1.2.1.2.2.1.6\') + row[\'ifMtu\'] = row.delete(\'1.3.6.1.2.1.2.2.1.4\') + row[\'ifHCInOctets\'] = row.delete(\'1.3.6.1.2.1.31.1.1.1.6\') + row[\'ifHCOutOctets\'] = row.delete(\'1.3.6.1.2.1.31.1.1.1.10\') + row[\'ifHCInBroadcastPkts\'] = row.delete(\'1.3.6.1.2.1.31.1.1.1.9\') + row[\'ifHCOutBroadcastPkts\'] = row.delete(\'1.3.6.1.2.1.31.1.1.1.13\') + row[\'ifHCInUcastPkts\'] = row.delete(\'1.3.6.1.2.1.31.1.1.1.7\') + row[\'ifHCOutUcastPkts\'] = row.delete(\'1.3.6.1.2.1.31.1.1.1.11\') + row[\'ifHCInMulticastPkts\'] = row.delete(\'1.3.6.1.2.1.31.1.1.1.8\') + row[\'ifHCOutMulticastPkts\'] = row.delete(\'1.3.6.1.2.1.31.1.1.1.12\') + row[\'ifLastChange\'] = row.delete(\'1.3.6.1.2.1.2.2.1.9\') + row[\'dot1qPvid\'] = row.delete(\'1.3.6.1.2.1.17.7.1.4.5.1.1\') + row[\'ifInErrors\'] = row.delete(\'1.3.6.1.2.1.2.2.1.14\') + row[\'ifOutErrors\'] = row.delete(\'1.3.6.1.2.1.2.2.1.20\') + row[\'ifInDiscards\'] = row.delete(\'1.3.6.1.2.1.2.2.1.13\') + row[\'ifOutDiscards\'] = row.delete(\'1.3.6.1.2.1.2.2.1.19\') + new_event = LogStash::Event.new({ + \'@timestamp\' => timestamp, + \'host\' => { \'name\' => host_name, \'hostname\' => host_hostname }, + \'network\' => { \'name\' => network_name }, + \'table\' => row, + \'metricset\' => { \'module\' => \'snmp\' }, + \'event\' => { \'kind\' => \'interfaces\' } + }) + new_event_block.call(new_event) + end + event.remove(\'[interfaces]\') + event.set(\'[event][kind]\', \'metrics\') + end' + } +} +output { + elasticsearch { + data_stream => "true" + data_stream_type => "metrics" + data_stream_namespace => "default" + data_stream_dataset => "snmp.polling" + cloud_id => "test" + user => "test" + password => "test" + } +} diff --git a/tests/Common/unit/conversion_data/pipelines/test-string-escaping.conf b/tests/Common/unit/conversion_data/pipelines/test-string-escaping.conf new file mode 100644 index 0000000..d3de6f5 --- /dev/null +++ b/tests/Common/unit/conversion_data/pipelines/test-string-escaping.conf @@ -0,0 +1,33 @@ +input { +} +filter { + mutate { + add_field => { + "double_quoted_with_hash" => "value # not a comment" + "with_brackets" => "data [in] brackets {and} braces" + "with_arrow" => "key => value pattern" + "env_ref" => "${MY_VAR}" + "sprintf_ref" => "prefix-%{field_name}" + } + } + grok { + match => { + "message" => '%{IP:client} \[%{HTTPDATE:ts}\] "%{WORD:method} %{URIPATHPARAM:path}' + } + pattern_definitions => { + "CUSTOM_IP" => "\b(?:\d{1,3}\.){3}\d{1,3}\b" + } + } + mutate { + rename => { + "@timestamp" => "event_time" + "host" => "source_host" + } + } +} +output { + file { + path => "/var/log/output/%{type}/%{+YYYY}/%{+MM}/%{+dd}.log" + codec => json + } +} diff --git a/tests/Common/unit/conversion_data/pipelines/test-twitter.conf b/tests/Common/unit/conversion_data/pipelines/test-twitter.conf new file mode 100644 index 0000000..13a8860 --- /dev/null +++ b/tests/Common/unit/conversion_data/pipelines/test-twitter.conf @@ -0,0 +1,41 @@ +input { + # This is the sample pipeline whose screenshots are used in + # the Pipeline Viewer documentation (../pipeline-viewer.asciidoc) + # + # Whenever the Pipeline Viewer UI changes, run this pipeline and + # open in the new UI to take updated screenshots. + # + # Note: you will have to setup the environment variables used + # below. Refer to the Twitter Logstash Input plugin documentation + # for their expected values + twitter { + id => "tweet harvester" + consumer_key => "${TWITTER_API_CONSUMER_KEY}" + consumer_secret => "${TWITTER_API_CONSUMER_SECRET}" + keywords => ["rain", "monsoon", "shower", "drizzle"] + oauth_token => "${TWITTER_API_OAUTH_TOKEN}" + oauth_token_secret => "${TWITTER_API_OAUTH_TOKEN_SECRET}" + } +} +filter { + grok { + match => { + "message" => "%{WORD:is_rt}" + } + } + if [is_rt] == "RT" { + drop { + id => "drop_all_RTs" + } + } +} +output { + stdout { + codec => dots + } + elasticsearch { + user => "elastic" + password => "changeme" + index => "tweets" + } +} diff --git a/tests/Common/unit/conversion_data/pipelines/test_complex1.conf b/tests/Common/unit/conversion_data/pipelines/test_complex1.conf new file mode 100644 index 0000000..18570e7 --- /dev/null +++ b/tests/Common/unit/conversion_data/pipelines/test_complex1.conf @@ -0,0 +1,329 @@ +input { + # + # "LogstashUI kitchen sink" pipeline + # Goal: be extremely feature-rich while staying within known-valid plugin options. + # + # Beats / Elastic Agent style shippers + beats { + id => "in_beats_5044" + port => "5044" + add_field => { + "ingest_transport" => "beats" + } + tags => ["from_beats"] + } + # JSON-over-TCP (common for app logs) + tcp { + id => "in_tcp_json_5514" + port => "5514" + mode => "server" + codec => json + add_field => { + "ingest_transport" => "tcp" + } + tags => ["from_tcp"] + } + # Syslog-ish UDP + udp { + id => "in_udp_5515" + port => "5515" + codec => plain + add_field => { + "ingest_transport" => "udp" + } + tags => ["from_udp"] + } + # HTTP event intake (webhooks, apps posting JSON, etc.) + http { + id => "in_http_8080" + port => "8080" + codec => json + add_field => { + "ingest_transport" => "http" + } + tags => ["from_http"] + } + # Local dev/testing input + stdin { + id => "in_stdin" + codec => line + add_field => { + "ingest_transport" => "stdin" + } + tags => ["from_stdin"] + } + # Synthetic test data (makes it easy to validate end-to-end quickly) + generator { + id => "in_generator" + lines => ["Feb 21 09:12:01 host1 sshd[123]: Failed password for invalid user admin from 10.1.2.3 port 51234 ssh2", "{\"@timestamp\":\"2026-02-21T14:12:02Z\",\"message\":\"GET /health 200\",\"source_ip\":\"8.8.8.8\",\"user_agent\":\"Mozilla/5.0\"}", "level=info service=api latency_ms=42 source_ip=192.168.1.50 msg=\"request completed\""] + count => "1" + add_field => { + "ingest_transport" => "generator" + } + tags => ["from_generator"] + } +} +filter { + # + # Normalize a few shared fields + # + mutate { + id => "f_mutate_bootstrap" + add_field => { + "[@metadata][pipeline]" => "logstashui_kitchen_sink" + "event.module" => "logstashui" + } + } + # Keep a canonical message field + if ![message] and [event][original] { + mutate { + id => "f_mutate_event_original_to_message" + copy => { + "[event][original]" => "message" + } + } + } + # + # Try to parse JSON *if* message looks like JSON (common when tcp/udp/plain feed JSON strings) + # + if [message] =~ "^[[:space:]]*\\{" { + json { + id => "f_json_from_message" + source => "message" + target => "json" + tag_on_failure => ["_jsonparsefailure_message"] + } + # If json parsed, promote a few expected keys (only if present) + if [json][@timestamp] { + mutate { + id => "f_promote_json_ts" + copy => { + "[json][@timestamp]" => "@timestamp" + } + } + } + if [json][source_ip] { + mutate { + id => "f_promote_json_source_ip" + copy => { + "[json][source_ip]" => "source_ip" + } + } + } + if [json][user_agent] { + mutate { + id => "f_promote_json_ua" + copy => { + "[json][user_agent]" => "user_agent" + } + } + } + } + # + # Syslog-ish parsing (UDP and some TCP) + # + if "from_udp" in [tags] or "from_tcp" in [tags] { + # Try dissect first (fast) and fall back to grok + dissect { + id => "f_dissect_syslogish" + mapping => { + "message" => "%{syslog_timestamp} %{syslog_host} %{syslog_program}[%{syslog_pid}]: %{syslog_message}" + } + tag_on_failure => ["_dissectfailure_syslogish"] + } + if "_dissectfailure_syslogish" in [tags] { + grok { + id => "f_grok_syslogish" + match => { + "message" => [ + "%{SYSLOGTIMESTAMP:syslog_timestamp} %{HOSTNAME:syslog_host} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" + ] + } + tag_on_failure => ["_grokparsefailure_syslogish"] + } + } + # If we extracted a syslog timestamp, use it + if [syslog_timestamp] { + date { + id => "f_date_syslog" + match => ["syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"] + tag_on_failure => ["_dateparsefailure_syslog"] + } + } + } + # + # key=value parsing for “flat” log lines + # + if [message] =~ "([A-Za-z0-9_.-]+)=([^\"]\\S+|\"[^\"]*\")" { + kv { + id => "f_kv_message" + source => "message" + trim_key => " " + trim_value => " " + value_split => "=" + field_split_pattern => "\s+" + tag_on_failure => ["_kvfailure_message"] + } + } + # + # Basic typing / normalization + # + mutate { + id => "f_mutate_normalize" + rename => { + "msg" => "message_short" + } + convert => { + "latency_ms" => "integer" + } + lowercase => ["level"] + } + # + # Enrichments: useragent, geoip, cidr, dns + # + if [user_agent] { + useragent { + id => "f_useragent" + source => "user_agent" + target => "user_agent_parsed" + } + } + # Canonicalize IP into source_ip if it exists elsewhere + if ![source_ip] and [source][ip] { + mutate { + id => "f_copy_source_ip" + copy => { + "[source][ip]" => "source_ip" + } + } + } + if [source_ip] { + # Tag private vs public + cidr { + id => "f_cidr_private" + address => ["%{source_ip}"] + network => ["10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"] + add_tag => ["src_private"] + } + # GeoIP typically only makes sense for public IPs, so do it only if not private-tagged + if "src_private" not in [tags] { + geoip { + id => "f_geoip" + source => "source_ip" + target => "source_geo" + } + } + # Reverse DNS lookup; replace source_ip with hostname when possible (or leave as-is) + dns { + id => "f_dns_reverse" + reverse => ["source_ip"] + action => "replace" + } + } + # + # Translate severity/level into a normalized numeric + # + translate { + id => "f_translate_level_to_severity" + source => "level" + target => "severity" + dictionary => { + "trace" => "0" + "debug" => "1" + "info" => "2" + "warn" => "3" + "error" => "4" + "fatal" => "5" + } + fallback => "2" + } + mutate { + id => "f_convert_severity_int" + convert => { + "severity" => "integer" + } + } + # + # Stable fingerprint for dedup / correlation + # + fingerprint { + id => "f_fingerprint_message" + source => ["message"] + method => "MURMUR3" + target => "[@metadata][fingerprint]" + } + # + # Example branching: treat auth-ish messages specially + # + if [syslog_program] == "sshd" or [message] =~ "(?i)failed password|authentication failure|invalid user" { + mutate { + id => "f_tag_auth" + add_tag => ["category_auth"] + add_field => { + "event.category" => "authentication" + } + } + } + else if [message] =~ "(?i)GET\\s+/health|/ready|/live" { + mutate { + id => "f_tag_health" + add_tag => ["category_healthcheck"] + add_field => { + "event.category" => "availability" + } + } + } + else { + mutate { + id => "f_tag_generic" + add_tag => ["category_generic"] + } + } + # + # Prune down noisy fields (keeps top-level essentials) + # + prune { + id => "f_prune" + whitelist_names => ["^@timestamp$", "^message$", "^message_short$", "^host$", "^source_ip$", "^source_geo$", "^severity$", "^level$", "^tags$", "^event\\..*$", "^user_agent.*$", "^syslog_.*$", "^ingest_transport$"] + } +} +output { + # Always see something in console during dev + stdout { + id => "out_stdout_rubydebug" + codec => rubydebug { + metadata => "true" + } + } + # Write to disk (great for debugging replay) + file { + id => "out_file_jsonl" + path => "/tmp/logstashui-%{+YYYY.MM.dd}.jsonl" + codec => json_lines + } + # Elasticsearch (local default) + elasticsearch { + id => "out_es_local" + hosts => ["http://localhost:9200"] + index => "logstashui-%{+YYYY.MM.dd}" + ilm_enabled => "false" + } + # Webhook back to your UI/API (example) + http { + id => "out_http_callback" + url => "http://localhost:9000/logstash/callback" + http_method => "post" + format => "json" + } + # Kafka (example) + kafka { + id => "out_kafka" + bootstrap_servers => "localhost:9092" + topic_id => "logstashui-events" + } + # Pipeline-to-pipeline (requires another pipeline with pipeline input address => "downstream") + pipeline { + id => "out_pipeline_downstream" + send_to => ["downstream"] + } +} diff --git a/tests/Common/unit/conversion_data/pipelines/test_elasticdocs-conditional.conf b/tests/Common/unit/conversion_data/pipelines/test_elasticdocs-conditional.conf new file mode 100644 index 0000000..7f58662 --- /dev/null +++ b/tests/Common/unit/conversion_data/pipelines/test_elasticdocs-conditional.conf @@ -0,0 +1,44 @@ +input { + file { + path => "/tmp/*_log" + } +} +filter { + if [path] =~ "access" { + mutate { + replace => { + "type" => "apache_access" + } + } + grok { + match => { + "message" => "%{COMBINEDAPACHELOG}" + } + } + date { + match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"] + } + } + else if [path] =~ "error" { + mutate { + replace => { + "type" => "apache_error" + } + } + } + else { + mutate { + replace => { + "type" => "random_logs" + } + } + } +} +output { + elasticsearch { + hosts => ["localhost:9200"] + } + stdout { + codec => rubydebug + } +} diff --git a/tests/Common/unit/conversion_data/pipelines/text-complex4.conf b/tests/Common/unit/conversion_data/pipelines/text-complex4.conf new file mode 100644 index 0000000..76b3692 --- /dev/null +++ b/tests/Common/unit/conversion_data/pipelines/text-complex4.conf @@ -0,0 +1,123 @@ +input { + generator { + id => "gen_edgeA" + count => 1 + lines => ["2026-02-22T01:23:45Z level=INFO service=api trace.id=abc123 method=GET path=\"/api/v2/items/42\" ip=8.8.8.8 ua=\"Mozilla/5.0 (Windows NT 10.0; Win64; x64)\" msg=\"hello\\world\""] + add_field => { + "[@metadata][source]" => "generator" + "event.original" => "%{message}" + } + } +} +filter { + # Fast split: timestamp + remainder + dissect { + id => "dissect_ts_rest" + mapping => { + "message" => "%{ts} %{rest}" + } + tag_on_failure => ["_dissectfailure_ts_rest"] + } + date { + id => "date_ts" + match => ["ts", "ISO8601"] + tag_on_failure => ["_dateparsefailure_ts"] + } + # Parse key=value in rest + kv { + id => "kv_rest" + source => "rest" + trim_key => " " + trim_value => " " + value_split => "=" + field_split_pattern => "\s+" + include_brackets => "false" + tag_on_failure => ["_kvfailure_rest"] + } + # Normalize: remove surrounding quotes on selected fields (common log format) + mutate { + id => "mutate_strip_quotes" + gsub => ["path", "^\"|\"$", "", "ua", "^\"|\"$", "", "msg", "^\"|\"$", ""] + } + # Promote a few fields into ECS-ish places + mutate { + id => "mutate_promote" + rename => { + "ip" => "[source][ip]" + "ua" => "[user_agent][original]" + "method" => "[http][request][method]" + "path" => "[url][path]" + "trace.id" => "[trace][id]" + } + lowercase => ["level"] + } + # Route key uses nested refs in sprintf (great UI test) + mutate { + id => "mutate_route_key" + add_field => { + "route_key" => "%{[@metadata][source]}::%{[service]}::%{[http][request][method]}::%{[url][path]}" + } + } + # Regex literals (escaped slashes) + if [url][path] =~ /^\/api\/v2\/items\/[0-9]+$/ { + mutate { + id => "tag_items" + add_tag => ["route_items"] + } + } + else if [url][path] =~ /^\/api\/v2\/[A-Za-z0-9._-]+$/ { + mutate { + id => "tag_api_generic" + add_tag => ["route_api_generic"] + } + } + else { + mutate { + id => "tag_other" + add_tag => ["route_other"] + } + } + # useragent parsing + if [user_agent][original] { + useragent { + id => "ua_parse" + source => "[user_agent][original]" + target => "[user_agent][parsed]" + } + } + # geoip on public source.ip + if [source][ip] { + geoip { + id => "geoip_source" + source => "[source][ip]" + target => "[source][geo]" + } + } + # fingerprint based on nested refs + message + fingerprint { + id => "fp_event" + source => ["route_key", "message"] + method => "MURMUR3" + target => "[@metadata][fp]" + } + # Replace literal backslash with slash in msg (escape-heavy but valid) + mutate { + id => "mutate_gsub_backslash" + gsub => ["msg", "\\\\", "/"] + } + prune { + id => "prune_edgeA" + whitelist_names => ["^@timestamp$", "^message$", "^tags$", "^level$", "^service$", "^route_key$", "^trace\\..*$", "^http\\..*$", "^url\\..*$", "^source\\..*$", "^user_agent\\..*$", "^msg$", "^@metadata\\..*$"] + } +} +output { + stdout { + codec => rubydebug { + metadata => "true" + } + } + file { + path => "/tmp/edgeA-%{+YYYY.MM.dd}.jsonl" + codec => json_lines + } +} diff --git a/tests/Common/unit/conversion_data/pipelines/text-complex5.conf b/tests/Common/unit/conversion_data/pipelines/text-complex5.conf new file mode 100644 index 0000000..e407bbb --- /dev/null +++ b/tests/Common/unit/conversion_data/pipelines/text-complex5.conf @@ -0,0 +1,40 @@ +input { + generator { + id => "gen_edgecase_3" + count => 1 + lines => ["path=\"C:\\Program Files\\App\\\" msg=\"quote:\" and backslash:\\\\\""] + } +} +filter { + # This ruby code contains both quote styles and backslashes. + ruby { + id => "ruby_edgecase_3" + code => ' + # Double quotes inside single-quoted LSCL string + event.set("[edge][note]", "He said: "hello"") + # Single quote inside Ruby string + event.set("[edge][apostrophe]", "it\'s fine") + # Trailing backslash in a field value (nasty for serializers) + event.set("[edge][trail]", "C: emp") + ' + } + # Parse key=val + kv { + id => "kv_edgecase_3" + source => "message" + value_split => "=" + field_split_pattern => "\s+" + } + # Replace literal backslash "\" with "/" + mutate { + id => "gsub_edgecase_3" + gsub => ["path", "\\\\", "/"] + } +} +output { + stdout { + codec => rubydebug { + metadata => "true" + } + } +} diff --git a/tests/Common/unit/conversion_data/pipelines/text-complex6.conf b/tests/Common/unit/conversion_data/pipelines/text-complex6.conf new file mode 100644 index 0000000..bd67ec0 --- /dev/null +++ b/tests/Common/unit/conversion_data/pipelines/text-complex6.conf @@ -0,0 +1,192 @@ +input { + kafka { + id => "input_kafka_1" + topics => ["critical_business_events", "low_latency_metrics"] + bootstrap_servers => "kafka1:9092,kafka2:9092" + group_id => "logstash_critical_group" + codec => json_lines + type => "business_event" + tags => ["kafka_input", "critical"] + max_poll_records => "500" + } + redis { + id => "input_redis_1" + host => "redis-cache.example.com" + port => "6379" + data_type => "list" + key => "service_log_queue" + type => "service_log" + tags => ["redis_input", "service_data"] + codec => plain + } + tcp { + id => "input_tcp_1" + port => "9999" + type => "audit_log" + ssl_enable => "true" + ssl_cert => "/etc/logstash/certs/logstash.crt" + ssl_key => "/etc/logstash/certs/logstash.key" + ssl_verify => "true" + codec => json { + delimiter => ' + ' + } + tags => ["tcp_input", "sensitive"] + } +} +filter { + if "service_data" in [tags] { + json { + id => "filter_json_1" + source => "message" + target => "parsed_service_log" + remove_field => ["message"] + add_tag => ["json_attempt"] + } + if "_jsonparsefailure" in [tags] { + grok { + id => "filter_grok_1" + match => { + "message" => "(?%{TIMESTAMP_ISO8601}) %{DATA:service_id} \[%{LOGLEVEL:level}] %{NUMBER:req_id:int} - %{GREEDYDATA:log_msg}" + } + add_tag => ["grok_fallback_success"] + remove_tag => ["_jsonparsefailure"] + } + } + if [parsed_service_log][sensitive_data] == true or [tags] =~ /_grokparsefailure/ { + mutate { + id => "filter_mutate_1" + gsub => ["message", "(\\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Z|a-z]{2,}\\b)", "email_masked"] + } + } + mutate { + id => "filter_mutate_2" + rename => { + "[parsed_service_log][level]" => "log_level" + } + add_field => { + "correlation_id" => "%{[parsed_service_log][request_id]}" + } + } + } + if [correlation_id] and [log_level] { + aggregate { + id => "filter_aggregate_1" + task_id => "%{correlation_id}" + code => ' + if event.get(\'log_level\') == \'START\' + map[\'start_time\'] = event.get(\'@timestamp\').time.to_f + map[\'service\'] = event.get(\'service_id\') + event.cancel + elsif event.get(\'log_level\') == \'END\' and map[\'start_time\'] + end_time = event.get(\'@timestamp\').time.to_f + duration = (end_time - map[\'start_time\']) * 1000 # Duration in ms + event.set(\'request_duration_ms\', duration.round(3)) + event.set(\'service_name\', map[\'service\']) + event.set(\'type\', \'request_summary\') + end + ' + map_action => "create_or_update" + push_map_as_event_on_timeout => "true" + timeout => "60" + timeout_code => "event.set('error_reason', 'Unmatched_START_Event')" + timeout_task_id_field => "unmatched_correlation_id" + } + } + if "critical" in [tags] or [type] == "audit_log" { + translate { + id => "filter_translate_1" + field => "tenant_id" + destination => "tenant_name" + dictionary_path => "/etc/logstash/dicts/tenant_map.yml" + fallback => "Unknown_Tenant" + refresh_interval => "600" + } + mutate { + id => "filter_mutate_3" + convert => { + "transaction_amount" => "float" + } + remove_field => ["host", "port"] + } + date { + id => "filter_date_1" + match => ["[event_time]", "ISO8601", "UNIX_MS"] + target => "@timestamp" + remove_tag => ["_dateparsefailure"] + } + } + if ("_grokparsefailure" in [tags] or "_jsonparsefailure" in [tags]) and [type] != "audit_log" { + mutate { + id => "filter_mutate_4" + add_tag => ["dlq_candidate", "parsing_error"] + add_field => { + "dlq_reason" => "Parsing_Failed" + } + } + } + mutate { + id => "filter_mutate_5" + remove_tag => ["_jsonparsefailure", "_grokparsefailure", "_dateparsefailure"] + } +} +output { + if [tenant_name] =~ /^PRIORITY_/ { + elasticsearch { + id => "output_elasticsearch_1" + hosts => ["https://es-priority:9200"] + index => "tenant_priority-%{tenant_name}-%{+YYYY.MM}" + workers => "1" + manage_template => "false" + } + } + else if [tenant_name] { + elasticsearch { + id => "output_elasticsearch_2" + hosts => ["https://es-main:9200"] + index => "tenant_general-%{+YYYY.MM.dd}" + dlq_enabled => "true" + dlq_path => "/var/lib/logstash/dlq" + } + } + if "dlq_candidate" in [tags] { + file { + id => "output_file_1" + path => "/var/log/logstash/error_logs/dlq_parsing_failures.log" + codec => json_lines { + target => "original_event" + } + add_tag => ["s3_backup"] + } + } + if [type] == "audit_log" or [type] == "request_summary" or "s3_backup" in [tags] { + s3 { + id => "output_s3_1" + bucket => "logstash-archive-bucket" + region => "us-west-2" + time_file => "15" + size_file => "50" + codec => json_lines + temporary_directory => "/tmp/logstash_s3_tmp" + } + } + if "unmatched_correlation_id" in [tags] { + tcp { + id => "output_tcp_1" + host => "graylog-server.example.com" + port => "12201" + codec => gelf { + level => 1 + short_message => "Log Aggregation Timeout/Error: %{unmatched_correlation_id}" + } + } + } + if [log_level] =~ /(START|END|FATAL)/ { + stdout { + id => "output_stdout_1" + codec => rubydebug { + metadata => "true" + } + } + } +} diff --git a/tests/Common/unit/conversion_data/pipelines/text-complex7.conf b/tests/Common/unit/conversion_data/pipelines/text-complex7.conf new file mode 100644 index 0000000..18f018e --- /dev/null +++ b/tests/Common/unit/conversion_data/pipelines/text-complex7.conf @@ -0,0 +1,222 @@ +input { + beats { + id => "input_beats_1" + port => "5044" + ssl => "true" + ssl_certificate => "/etc/logstash/certs/logstash.crt" + ssl_key => "/etc/logstash/certs/logstash.key" + codec => json + tags => ["beats_input", "app_log"] + } + udp { + id => "input_udp_1" + port => "5140" + buffer_size => "8192" + codec => plain { + charset => "UTF-8" + } + type => "network_flow" + tags => ["udp_input", "unstructured"] + } + jdbc { + id => "input_jdbc_1" + jdbc_driver_library => "/usr/share/logstash/logstash-core/lib/jars/postgresql-42.2.8.jar" + jdbc_driver_class => "org.postgresql.Driver" + jdbc_connection_string => "jdbc:postgresql://db.example.com:5432/config_db" + jdbc_user => "logstash_user" + jdbc_password => "${JDBC_PASSWORD}" + schedule => "0 * * * *" + statement => "SELECT id, user_name, config_item, change_timestamp FROM config_changes WHERE change_timestamp > :sql_last_value ORDER BY change_timestamp ASC" + use_column_value => "true" + tracking_column => "change_timestamp" + tracking_column_type => "timestamp" + last_run_metadata_path => "/var/lib/logstash/.jdbc_last_run_config_db" + type => "config_audit" + tags => ["jdbc_input", "audit"] + } +} +filter { + mutate { + id => "filter_mutate_1" + rename => { + "@timestamp" => "log_recv_time" + } + add_field => { + "severity" => "INFO" + } + } + if "app_log" in [tags] { + if "_jsonparsefailure" in [tags] { + grok { + id => "filter_grok_1" + match => { + "message" => "(?\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3}) \[%{DATA:thread}] %{LOGLEVEL:log_level} %{DATA:logger} - %{GREEDYDATA:log_message}" + } + add_tag => ["grok_fallback_success"] + remove_tag => ["_jsonparsefailure"] + } + } + if [log_level] { + mutate { + id => "filter_mutate_2" + uppercase => ["log_level"] + copy => { + "log_level" => "severity" + } + } + } + if [severity] =~ /(WARNING|ERROR|FATAL)/ { + geoip { + id => "filter_geoip_1" + source => "[fields][source_ip]" + target => "geo" + database => "/etc/logstash/geoip/GeoLite2-City.mmdb" + remove_field => ["continent_code", "location"] + } + translate { + id => "filter_translate_1" + field => "[service_code]" + destination => "service_name" + dictionary_path => "/etc/logstash/dictionaries/service_codes.csv" + fallback => "Unknown Service" + refresh_interval => "300" + } + } + mutate { + id => "filter_mutate_3" + remove_field => ["message", "agent"] + } + } + else if [type] == "network_flow" { + grok { + id => "filter_grok_2" + match => { + "message" => "%{NETFLOW_V9}" + } + on_failure => ["_netflowparsefailure"] + } + if "_netflowparsefailure" in [tags] { + mutate { + id => "filter_mutate_4" + add_tag => ["unparsed_flow"] + remove_tag => ["_grokparsefailure", "_netflowparsefailure"] + copy => { + "message" => "unparsed_data" + } + replace => { + "message" => "Truncated unparsed flow data." + } + } + } + else { + aggregate { + id => "filter_aggregate_1" + task_id => "%{source_ip}_%{destination_ip}_%{protocol}" + code => "map['total_packets'] ||= 0; map['total_packets'] += event.get('packets').to_i; map['total_bytes'] ||= 0; map['total_bytes'] += event.get('bytes').to_i" + map_action => "create_or_update" + push_map_as_event_on_timeout => "true" + timeout => "120" + timeout_task_id_field => "aggregated_flow_id" + timeout_tags => ["_aggregate_timeout"] + } + } + } + else if [type] == "config_audit" { + if [change_timestamp] { + date { + id => "filter_date_1" + match => ["change_timestamp", "YYYY-MM-dd HH:mm:ss.SSSSSS"] + target => "@timestamp" + remove_field => ["change_timestamp"] + } + if [user_name] != "system" { + ruby { + id => "filter_ruby_1" + code => "event.set('user_hash', Digest::MD5.hexdigest(event.get('user_name')))" + } + mutate { + id => "filter_mutate_5" + remove_field => ["user_name"] + } + } + } + } + if "_grokparsefailure" in [tags] or "_jsonparsefailure" in [tags] { + mutate { + id => "filter_mutate_6" + add_field => { + "log_status" => "FAILED_TO_PARSE" + } + } + } + else { + mutate { + id => "filter_mutate_7" + add_field => { + "log_status" => "PROCESSED" + } + } + } + if [log_recv_time] < now() - 86400000 { + drop { + id => "filter_drop_1" + } + } +} +output { + if [log_status] == "PROCESSED" and [severity] =~ /(ERROR|FATAL)/ { + elasticsearch { + id => "output_elasticsearch_1" + hosts => ["https://es-hot.example.com:9200"] + index => "high-priority-%{+YYYY.MM.dd}" + user => "logstash_writer" + password => "secure_password" + ssl => "true" + cacert => "/etc/logstash/certs/ca.crt" + action => "index" + } + } + else if [log_status] == "PROCESSED" { + elasticsearch { + id => "output_elasticsearch_2" + hosts => ["https://es-warm.example.com:9200"] + index => "general-logs-%{+YYYY.MM.dd}" + user => "logstash_writer" + password => "secure_password" + ssl => "true" + cacert => "/etc/logstash/certs/ca.crt" + workers => "4" + ilm_enabled => "false" + } + } + if [log_status] == "FAILED_TO_PARSE" { + file { + id => "output_file_1" + path => "/var/log/logstash/dlq_failures.json" + codec => json { + pretty => "true" + } + add_tag => ["dlq_routed"] + } + } + if "_aggregate_timeout" in [tags] { + tcp { + id => "output_tcp_1" + host => "alert-sys.example.com" + port => "6514" + codec => gelf { + protocol => "TCP" + short_message => "Aggregated Flow Timeout: %{aggregated_flow_id}" + } + socket_timeout => "5" + } + } + if rand(100) < 1 { + stdout { + id => "output_stdout_1" + codec => rubydebug { + metadata => "true" + } + } + } +} diff --git a/tests/Common/unit/conversion_data/pipelines/text-ls-repo-nginx-error.conf b/tests/Common/unit/conversion_data/pipelines/text-ls-repo-nginx-error.conf new file mode 100644 index 0000000..0d5f148 --- /dev/null +++ b/tests/Common/unit/conversion_data/pipelines/text-ls-repo-nginx-error.conf @@ -0,0 +1,67 @@ +input { + stdin { + codec => line + } +} +filter { + mutate { + add_field => { + "event.dataset" => "nginx.access" + "service.name" => "nginx" + } + } + grok { + match => { + "message" => [ + '%{IPORHOST:source.address} %{DATA:nginx.ident} %{DATA:user.name} \[%{HTTPDATE:nginx.access.time}\] "%{WORD:http.request.method} %{DATA:url.original} HTTP/%{NUMBER:http.version}" %{NUMBER:http.response.status_code:int} (?:%{NUMBER:http.response.body.bytes:int}|-) "%{DATA:http.request.referrer}" "%{DATA:user_agent.original}" %{NUMBER:nginx.access.request_time:float}', + '%{IPORHOST:source.address} %{DATA:nginx.ident} %{DATA:user.name} \[%{HTTPDATE:nginx.access.time}\] "%{WORD:http.request.method} %{DATA:url.original} HTTP/%{NUMBER:http.version}" %{NUMBER:http.response.status_code:int} (?:%{NUMBER:http.response.body.bytes:int}|-) "%{DATA:http.request.referrer}" "%{DATA:user_agent.original}"', + '%{IPORHOST:source.address} %{DATA:nginx.ident} %{DATA:user.name} \[%{HTTPDATE:nginx.access.time}\] "%{WORD:http.request.method} %{DATA:url.original} HTTP/%{NUMBER:http.version}" %{NUMBER:http.response.status_code:int} (?:%{NUMBER:http.response.body.bytes:int}|-) "%{DATA:http.request.referrer}" "%{DATA:user_agent.original}" (?:rt=%{NUMBER:nginx.access.request_time:float}\s+uct=%{NUMBER:nginx.access.upstream_connect_time:float}\s+uht=%{NUMBER:nginx.access.upstream_header_time:float}\s+urt=%{NUMBER:nginx.access.upstream_response_time:float})' + ] + } + tag_on_failure => ["_grok_nginx_access_fail"] + } + date { + match => ["nginx.access.time", "dd/MMM/yyyy:HH:mm:ss Z"] + target => "@timestamp" + } + urldecode { + field => "url.original" + } + dissect { + mapping => { + "url.original" => "%{url.path}?%{url.query}" + } + } + useragent { + source => "user_agent.original" + target => "user_agent" + } + mutate { + copy => { + "source.address" => "source.ip" + } + } + geoip { + source => "source.ip" + target => "source.geo" + tag_on_failure => ["_geoip_fail"] + } + mutate { + gsub => ["http.request.referrer", "^-$", "", "user.name", "^-$", ""] + } + if [http][response][status_code] >= 500 { + mutate { + add_tag => ["nginx_server_error"] + } + } + else if [http][response][status_code] >= 400 { + mutate { + add_tag => ["nginx_client_error"] + } + } +} +output { + stdout { + codec => rubydebug + } +} diff --git a/tests/Common/unit/test_components_to_pipeline.py b/tests/Common/unit/test_components_to_pipeline.py new file mode 100644 index 0000000..06422a0 --- /dev/null +++ b/tests/Common/unit/test_components_to_pipeline.py @@ -0,0 +1,57 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +from Common.logstash_config_parse import ComponentToPipeline, logstash_config_to_components +import pytest +import json +import os + +# Load test cases from external files +def load_test_cases(): + """Load test cases from conversion_data directory.""" + base_dir = os.path.dirname(os.path.abspath(__file__)) + pipelines_dir = os.path.join(base_dir, "conversion_data", "pipelines") + components_dir = os.path.join(base_dir, "conversion_data", "components") + + test_cases = [] + + # Get all .conf files + for filename in sorted(os.listdir(pipelines_dir)): + if filename.endswith('.conf'): + name = filename[:-5] # Remove .conf extension + + # Load pipeline config + pipeline_file = os.path.join(pipelines_dir, filename) + with open(pipeline_file, 'r', encoding='utf-8') as f: + pipeline = f.read() + + # Load components JSON + components_file = os.path.join(components_dir, f"{name}.json") + with open(components_file, 'r', encoding='utf-8') as f: + components = f.read() + + test_cases.append((name, pipeline, components)) + + return test_cases + +test_cases = load_test_cases() + + + +@pytest.mark.parametrize( + "name, pipeline, components", + test_cases, + ids=[case[0] for case in test_cases] +) +def test_components_to_config(name, pipeline, components): + """ + Test that ComponentToPipeline can generate a pipeline config from components. + Compares the original pipeline with the generated pipeline from stored components. + """ + # Load the stored components and convert to pipeline + parser = ComponentToPipeline(json.loads(components)) + generated_pipeline = parser.components_to_logstash_config() + + # Compare the original pipeline with the generated one + assert pipeline == generated_pipeline diff --git a/tests/Common/unit/test_context_processors.py b/tests/Common/unit/test_context_processors.py new file mode 100644 index 0000000..65b484b --- /dev/null +++ b/tests/Common/unit/test_context_processors.py @@ -0,0 +1,224 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +import pytest +from django.test import RequestFactory +from unittest.mock import patch, Mock + +from Common.context_processors import ( + version_update_info, + navigation_highlight +) +from PipelineManager.models import Connection + + +@pytest.fixture +def request_factory(): + """Django RequestFactory for creating mock requests""" + return RequestFactory() + + +@pytest.fixture +def mock_request(request_factory): + """Create a basic mock request""" + return request_factory.get('/') + + +class TestVersionUpdateInfo: + """Test version_update_info context processor""" + + @patch('Common.context_processors.check_for_update') + def test_version_update_info_returns_context(self, mock_check_update, mock_request): + """Test that version_update_info returns correct context""" + mock_update_data = { + 'update_available': True, + 'latest_version': '2.0.0', + 'current_version': '1.0.0' + } + mock_check_update.return_value = mock_update_data + + context = version_update_info(mock_request) + + assert 'version_update' in context + assert context['version_update'] == mock_update_data + mock_check_update.assert_called_once() + + @patch('Common.context_processors.check_for_update') + def test_version_update_info_no_update(self, mock_check_update, mock_request): + """Test version_update_info when no update is available""" + mock_update_data = { + 'update_available': False, + 'latest_version': '1.0.0', + 'current_version': '1.0.0' + } + mock_check_update.return_value = mock_update_data + + context = version_update_info(mock_request) + + assert context['version_update']['update_available'] is False + + @patch('Common.context_processors.check_for_update') + def test_version_update_info_none_response(self, mock_check_update, mock_request): + """Test version_update_info when check_for_update returns None""" + mock_check_update.return_value = None + + context = version_update_info(mock_request) + + assert 'version_update' in context + assert context['version_update'] is None + + @patch('Common.context_processors.check_for_update') + def test_version_update_info_error_handling(self, mock_check_update, mock_request): + """Test version_update_info handles errors gracefully""" + mock_check_update.side_effect = Exception("Network error") + + # Should raise the exception (no error handling in the function) + with pytest.raises(Exception): + version_update_info(mock_request) + + +class TestNavigationHighlight: + """Test navigation_highlight context processor. + + highlight_snmp_devices was removed from the server-side context processor; + that logic now lives in client-side localStorage. The processor only tracks + whether any Connection exists and exposes: + - highlight_connection_manager (bool) + - has_connections (bool) + """ + + def test_no_connections_highlights_connection_manager(self, mock_request, db): + """Connection Manager is highlighted when no connections exist""" + Connection.objects.all().delete() + + context = navigation_highlight(mock_request) + + assert context['highlight_connection_manager'] is True + assert context['has_connections'] is False + + def test_connections_exist_does_not_highlight_connection_manager(self, mock_request, db): + """Connection Manager is NOT highlighted when at least one connection exists""" + Connection.objects.create( + name='Test Connection', + connection_type='CENTRALIZED', + host='https://localhost:9200', + username='elastic', + password='changeme', + port=None, + ) + + context = navigation_highlight(mock_request) + + assert context['highlight_connection_manager'] is False + assert context['has_connections'] is True + + def test_multiple_connections(self, mock_request, db): + """has_connections is True when multiple connections exist""" + Connection.objects.create( + name='Connection 1', + connection_type='CENTRALIZED', + host='https://localhost:9200', + username='elastic', + password='changeme', + port=None, + ) + Connection.objects.create( + name='Connection 2', + connection_type='CENTRALIZED', + cloud_id='test-id', + api_key='test-api-key', + ) + + context = navigation_highlight(mock_request) + + assert context['highlight_connection_manager'] is False + assert context['has_connections'] is True + + def test_context_keys_always_present(self, mock_request, db): + """highlight_connection_manager and has_connections are always in context""" + Connection.objects.all().delete() + + context = navigation_highlight(mock_request) + + assert 'highlight_connection_manager' in context + assert 'has_connections' in context + assert isinstance(context['highlight_connection_manager'], bool) + assert isinstance(context['has_connections'], bool) + + def test_navigation_highlight_with_different_request_types(self, request_factory, db): + """navigation_highlight works regardless of HTTP method""" + Connection.objects.all().delete() + + for method in ('get', 'post', 'put'): + request = getattr(request_factory, method)('/test/') + context = navigation_highlight(request) + assert context['highlight_connection_manager'] is True + + def test_navigation_highlight_database_queries(self, mock_request, db): + """navigation_highlight queries Connection.objects.exists() exactly once""" + Connection.objects.all().delete() + + with patch.object(Connection.objects, 'exists', return_value=False) as mock_conn_exists: + context = navigation_highlight(mock_request) + + mock_conn_exists.assert_called_once() + assert context['highlight_connection_manager'] is True + assert context['has_connections'] is False + + def test_navigation_highlight_logic_flow(self, mock_request, db): + """Complete logic: no connections → highlight; connection added → no highlight""" + Connection.objects.all().delete() + + # State 1: No connections + context = navigation_highlight(mock_request) + assert context == { + 'highlight_connection_manager': True, + 'has_connections': False, + } + + # State 2: Connection added + Connection.objects.create( + name='Test', + connection_type='CENTRALIZED', + host='https://localhost:9200', + username='elastic', + password='changeme', + port=None, + ) + context = navigation_highlight(mock_request) + assert context == { + 'highlight_connection_manager': False, + 'has_connections': True, + } + + +class TestContextProcessorsIntegration: + """Integration tests for context processors""" + + @patch('Common.context_processors.check_for_update') + def test_both_context_processors_together(self, mock_check_update, mock_request, db): + """Both context processors can be used together without key collisions""" + mock_check_update.return_value = {'update_available': True} + Connection.objects.all().delete() + + version_context = version_update_info(mock_request) + navigation_context = navigation_highlight(mock_request) + + combined_context = {**version_context, **navigation_context} + + assert 'version_update' in combined_context + assert 'highlight_connection_manager' in combined_context + assert 'has_connections' in combined_context + # version_update + highlight_connection_manager + has_connections + assert len(combined_context) == 3 + + def test_context_processors_dont_interfere(self, mock_request, db): + """Context processors return disjoint key sets""" + with patch('Common.context_processors.check_for_update') as mock_check: + mock_check.return_value = {'test': 'data'} + + version_context = version_update_info(mock_request) + navigation_context = navigation_highlight(mock_request) + + assert set(version_context.keys()).isdisjoint(set(navigation_context.keys())) diff --git a/tests/Common/unit/test_decorators.py b/tests/Common/unit/test_decorators.py new file mode 100644 index 0000000..00bd5d8 --- /dev/null +++ b/tests/Common/unit/test_decorators.py @@ -0,0 +1,269 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +import pytest +from django.contrib.auth.models import User +from django.http import HttpRequest, HttpResponse +from django.test import RequestFactory +from unittest.mock import Mock + +from Common.decorators import require_admin_role +from Management.models import UserProfile + + +@pytest.fixture +def request_factory(): + """Django RequestFactory for creating mock requests""" + return RequestFactory() + + +@pytest.fixture +def admin_user(db): + """Create a user with admin profile""" + user = User.objects.create_user( + username='admin_user', + password='testpass123', + email='admin@example.com' + ) + # Signal creates profile automatically, just ensure it's admin + profile, created = UserProfile.objects.get_or_create(user=user, defaults={'role': 'admin'}) + if not created: + profile.role = 'admin' + profile.save() + return user + + +@pytest.fixture +def readonly_user(db): + """Create a user with readonly profile""" + user = User.objects.create_user( + username='readonly_user', + password='testpass123', + email='readonly@example.com' + ) + # Signal creates profile automatically, update to readonly + profile = UserProfile.objects.get(user=user) + profile.role = 'readonly' + profile.save() + # Refresh user from database to get updated profile relationship + user.refresh_from_db() + return user + + +@pytest.fixture +def user_without_profile(db): + """Create a user without a profile (edge case for bug 2a)""" + from django.db.models.signals import post_save + from Management.models import create_user_profile + + # Temporarily disconnect the signal to prevent auto-creation + post_save.disconnect(create_user_profile, sender=User) + + try: + user = User.objects.create_user( + username='no_profile_user', + password='testpass123', + email='noprofile@example.com' + ) + # Explicitly ensure no profile exists + UserProfile.objects.filter(user=user).delete() + finally: + # Reconnect the signal + post_save.connect(create_user_profile, sender=User) + + return user + + +@pytest.fixture +def mock_view(): + """Create a mock view function""" + def view_func(request, *args, **kwargs): + return HttpResponse("Success", status=200) + view_func.__name__ = "mock_view_function" + return view_func + + +class TestRequireAdminRoleDecorator: + """Test require_admin_role decorator""" + + def test_unauthenticated_request_denied(self, request_factory, mock_view): + """Test that unauthenticated requests are denied""" + request = request_factory.get('/test/') + request.user = Mock() + request.user.is_authenticated = False + + decorated_view = require_admin_role(mock_view) + response = decorated_view(request) + + assert response.status_code == 403 + assert b'You must be logged in to perform this action' in response.content + assert 'HX-Trigger' in response + assert 'showToastEvent' in response['HX-Trigger'] + + def test_admin_user_allowed(self, request_factory, mock_view, admin_user): + """Test that admin users are allowed access""" + request = request_factory.get('/test/') + request.user = admin_user + + decorated_view = require_admin_role(mock_view) + response = decorated_view(request) + + assert response.status_code == 200 + assert b'Success' in response.content + + def test_readonly_user_denied(self, request_factory, mock_view, readonly_user): + """Test that readonly users are denied access""" + request = request_factory.get('/test/') + request.user = readonly_user + + decorated_view = require_admin_role(mock_view) + response = decorated_view(request) + + assert response.status_code == 403 + assert b'Access denied: Admin role required' in response.content + assert 'HX-Trigger' in response + assert 'showToastEvent' in response['HX-Trigger'] + + def test_user_without_profile_denied(self, request_factory, mock_view, user_without_profile): + """ + CRITICAL TEST for bug 2a: Test that users without profiles are denied access. + This is the missing-profile edge case that was a security vulnerability. + """ + request = request_factory.get('/test/') + request.user = user_without_profile + + # Verify user has no profile + assert not hasattr(user_without_profile, 'profile') or not UserProfile.objects.filter(user=user_without_profile).exists() + + decorated_view = require_admin_role(mock_view) + response = decorated_view(request) + + # User should be DENIED, not allowed + assert response.status_code == 403 + assert b'Access denied: Admin role required' in response.content + assert 'HX-Trigger' in response + assert 'showToastEvent' in response['HX-Trigger'] + + def test_superuser_without_profile_denied(self, request_factory, mock_view, db): + """ + Test that even superusers without profiles are denied. + This simulates a superuser created via createsuperuser before signal fires. + """ + from django.db.models.signals import post_save + from Management.models import create_user_profile + + # Temporarily disconnect the signal + post_save.disconnect(create_user_profile, sender=User) + + try: + superuser = User.objects.create_superuser( + username='superuser', + password='testpass123', + email='super@example.com' + ) + # Ensure no profile exists + UserProfile.objects.filter(user=superuser).delete() + finally: + # Reconnect the signal + post_save.connect(create_user_profile, sender=User) + + request = request_factory.get('/test/') + request.user = superuser + + decorated_view = require_admin_role(mock_view) + response = decorated_view(request) + + # Even superuser should be denied without profile + assert response.status_code == 403 + assert b'Access denied: Admin role required' in response.content + + def test_decorator_preserves_view_metadata(self, mock_view): + """Test that decorator preserves original view function metadata""" + decorated_view = require_admin_role(mock_view) + + # functools.wraps should preserve __name__ + assert decorated_view.__name__ == mock_view.__name__ + + def test_decorator_with_view_args_and_kwargs(self, request_factory, admin_user): + """Test that decorator properly passes args and kwargs to view""" + def view_with_args(request, arg1, arg2, kwarg1=None): + return HttpResponse(f"{arg1}-{arg2}-{kwarg1}", status=200) + + view_with_args.__name__ = "view_with_args" + + request = request_factory.get('/test/') + request.user = admin_user + + decorated_view = require_admin_role(view_with_args) + response = decorated_view(request, "val1", "val2", kwarg1="val3") + + assert response.status_code == 200 + assert b'val1-val2-val3' in response.content + + def test_logging_for_readonly_user(self, request_factory, mock_view, readonly_user, caplog): + """Test that readonly user access attempts are logged""" + request = request_factory.get('/test/') + request.user = readonly_user + + decorated_view = require_admin_role(mock_view) + response = decorated_view(request) + + # Check that warning was logged with role information + assert "readonly_user" in caplog.text + assert "'readonly'" in caplog.text + assert "mock_view_function" in caplog.text + + def test_logging_for_user_without_profile(self, request_factory, mock_view, user_without_profile, caplog): + """Test that users without profiles have 'no profile' logged""" + request = request_factory.get('/test/') + request.user = user_without_profile + + decorated_view = require_admin_role(mock_view) + response = decorated_view(request) + + # Check that warning was logged with 'no profile' information + assert "no_profile_user" in caplog.text + assert "no profile" in caplog.text + assert "mock_view_function" in caplog.text + + def test_htmx_trigger_header_format(self, request_factory, mock_view, readonly_user): + """Test that HX-Trigger header is properly formatted JSON""" + import json + + request = request_factory.get('/test/') + request.user = readonly_user + + decorated_view = require_admin_role(mock_view) + response = decorated_view(request) + + # Verify HX-Trigger is valid JSON + trigger_data = json.loads(response['HX-Trigger']) + assert 'showToastEvent' in trigger_data + assert trigger_data['showToastEvent']['type'] == 'error' + assert 'Admin role required' in trigger_data['showToastEvent']['message'] + + def test_multiple_decorators_stacking(self, request_factory, admin_user): + """Test that decorator can be stacked with other decorators""" + def another_decorator(view_func): + def wrapper(request, *args, **kwargs): + response = view_func(request, *args, **kwargs) + response['X-Custom-Header'] = 'test' + return response + wrapper.__name__ = view_func.__name__ + return wrapper + + def view_func(request): + return HttpResponse("Success", status=200) + view_func.__name__ = "stacked_view" + + # Stack decorators + decorated_view = require_admin_role(another_decorator(view_func)) + + request = request_factory.get('/test/') + request.user = admin_user + + response = decorated_view(request) + + assert response.status_code == 200 + assert 'X-Custom-Header' in response diff --git a/tests/Common/unit/test_elastic_utils.py b/tests/Common/unit/test_elastic_utils.py new file mode 100644 index 0000000..c79af6d --- /dev/null +++ b/tests/Common/unit/test_elastic_utils.py @@ -0,0 +1,654 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +import pytest +from unittest.mock import Mock, patch, MagicMock +from elasticsearch import Elasticsearch + +from Common.elastic_utils import ( + test_elastic_connectivity as es_test_connectivity, + get_elastic_connections_from_list, + get_elastic_connection, + _get_creds, + get_elasticsearch_indices, + get_elasticsearch_field_mappings, + _extract_field_names, + query_elasticsearch_documents, + normalize_kibana_url, +) +from PipelineManager.models import Connection + + +@pytest.fixture +def mock_connection(db): + """Create a mock connection for testing""" + connection = Connection.objects.create( + name='Test Connection', + connection_type='CENTRALIZED', + host='https://localhost:9200', + username='elastic', + password='changeme', + port=None, # host already contains the port; avoid double-appending + ) + return connection + + +@pytest.fixture +def mock_cloud_connection(db): + """Create a mock cloud connection for testing""" + connection = Connection.objects.create( + name='Cloud Connection', + connection_type='CENTRALIZED', + cloud_id='test-cloud-id:dGVzdA==', + api_key='test-api-key' + ) + return connection + + +class TestGetCreds: + """Test _get_creds function""" + + def test_get_creds_with_host_and_password(self, mock_connection): + """Test getting credentials with host and password auth""" + creds = _get_creds(mock_connection.id) + + assert 'hosts' in creds + assert creds['hosts'] == 'https://localhost:9200' + assert 'http_auth' in creds + assert creds['http_auth'][0] == 'elastic' + + def test_get_creds_with_cloud_id_and_api_key(self, mock_cloud_connection): + """Test getting credentials with cloud_id and api_key""" + creds = _get_creds(mock_cloud_connection.id) + + assert 'cloud_id' in creds + assert creds['cloud_id'] == 'test-cloud-id:dGVzdA==' + assert 'api_key' in creds + assert 'http_auth' not in creds + assert 'hosts' not in creds + + +class TestGetElasticConnection: + """Test get_elastic_connection function""" + + @patch('Common.elastic_utils.Elasticsearch') + def test_get_elastic_connection(self, mock_es_class, mock_connection): + """Test getting Elasticsearch connection""" + mock_es_instance = Mock() + mock_es_class.return_value = mock_es_instance + + result = get_elastic_connection(mock_connection.id) + + assert result == mock_es_instance + mock_es_class.assert_called_once() + call_kwargs = mock_es_class.call_args[1] + assert 'hosts' in call_kwargs or 'cloud_id' in call_kwargs + + +class TestGetElasticConnectionsFromList: + """Test get_elastic_connections_from_list function""" + + @patch('Common.elastic_utils.get_elastic_connection') + def test_get_connections_from_list(self, mock_get_connection, mock_connection): + """Test getting list of connections""" + mock_es = Mock() + mock_get_connection.return_value = mock_es + + connections = get_elastic_connections_from_list() + + assert len(connections) == 1 + assert connections[0]['name'] == 'Test Connection' + assert connections[0]['es'] == mock_es + assert connections[0]['id'] == mock_connection.id + assert connections[0]['connection_type'] == 'CENTRALIZED' + + @patch('Common.elastic_utils.get_elastic_connection') + def test_get_multiple_connections(self, mock_get_connection, db): + """Test getting multiple connections""" + Connection.objects.create( + name='Connection 1', + connection_type='CENTRALIZED', + host='https://localhost:9200', + username='elastic', + password='changeme' + ) + Connection.objects.create( + name='Connection 2', + connection_type='CENTRALIZED', + cloud_id='test-id', + api_key='test-api-key' + ) + + mock_get_connection.return_value = Mock() + + connections = get_elastic_connections_from_list() + + assert len(connections) == 2 + assert connections[0]['name'] == 'Connection 1' + assert connections[1]['name'] == 'Connection 2' + + +class TestGetElasticsearchIndices: + """Test get_elasticsearch_indices function""" + + @patch('Common.elastic_utils.get_elastic_connection') + def test_get_indices_default_pattern(self, mock_get_connection, mock_connection): + """Test getting indices with default pattern""" + mock_es = Mock() + mock_es.cat.indices.return_value = [ + {'index': 'index-1'}, + {'index': 'index-2'}, + {'index': 'index-3'} + ] + mock_get_connection.return_value = mock_es + + indices = get_elasticsearch_indices(mock_connection.id) + + assert len(indices) == 3 + assert 'index-1' in indices + assert 'index-2' in indices + assert 'index-3' in indices + mock_es.cat.indices.assert_called_once_with(index='*', format='json', h='index') + + @patch('Common.elastic_utils.get_elastic_connection') + def test_get_indices_custom_pattern(self, mock_get_connection, mock_connection): + """Test getting indices with custom pattern""" + mock_es = Mock() + mock_es.cat.indices.return_value = [ + {'index': 'logs-2024-01'}, + {'index': 'logs-2024-02'} + ] + mock_get_connection.return_value = mock_es + + indices = get_elasticsearch_indices(mock_connection.id, pattern='logs-*') + + assert len(indices) == 2 + mock_es.cat.indices.assert_called_once_with(index='logs-*', format='json', h='index') + + @patch('Common.elastic_utils.get_elastic_connection') + def test_get_indices_sorted(self, mock_get_connection, mock_connection): + """Test that indices are returned sorted""" + mock_es = Mock() + mock_es.cat.indices.return_value = [ + {'index': 'zebra'}, + {'index': 'alpha'}, + {'index': 'beta'} + ] + mock_get_connection.return_value = mock_es + + indices = get_elasticsearch_indices(mock_connection.id) + + assert indices == ['alpha', 'beta', 'zebra'] + + @patch('Common.elastic_utils.get_elastic_connection') + def test_get_indices_limited_to_50(self, mock_get_connection, mock_connection): + """Test that only top 50 indices are returned""" + mock_es = Mock() + mock_es.cat.indices.return_value = [ + {'index': f'index-{i:03d}'} for i in range(100) + ] + mock_get_connection.return_value = mock_es + + indices = get_elasticsearch_indices(mock_connection.id) + + assert len(indices) == 50 + + @patch('Common.elastic_utils.get_elastic_connection') + def test_get_indices_error_handling(self, mock_get_connection, mock_connection): + """Test error handling when fetching indices fails""" + mock_es = Mock() + mock_es.cat.indices.side_effect = Exception("Connection error") + mock_get_connection.return_value = mock_es + + indices = get_elasticsearch_indices(mock_connection.id) + + assert indices == [] + + +class TestGetElasticsearchFieldMappings: + """Test get_elasticsearch_field_mappings function""" + + @patch('Common.elastic_utils.get_elastic_connection') + def test_get_field_mappings(self, mock_get_connection, mock_connection): + """Test getting field mappings from index""" + mock_es = Mock() + mock_es.indices.get_mapping.return_value = { + 'test-index': { + 'mappings': { + 'properties': { + 'field1': {'type': 'text'}, + 'field2': {'type': 'keyword'}, + 'nested_field': { + 'properties': { + 'subfield1': {'type': 'long'} + } + } + } + } + } + } + mock_get_connection.return_value = mock_es + + fields = get_elasticsearch_field_mappings(mock_connection.id, 'test-index') + + assert 'field1' in fields + assert 'field2' in fields + assert 'nested_field' in fields + assert 'nested_field.subfield1' in fields + assert len(fields) == 4 + + @patch('Common.elastic_utils.get_elastic_connection') + def test_get_field_mappings_sorted(self, mock_get_connection, mock_connection): + """Test that field mappings are sorted""" + mock_es = Mock() + mock_es.indices.get_mapping.return_value = { + 'test-index': { + 'mappings': { + 'properties': { + 'zebra': {'type': 'text'}, + 'alpha': {'type': 'keyword'}, + 'beta': {'type': 'long'} + } + } + } + } + mock_get_connection.return_value = mock_es + + fields = get_elasticsearch_field_mappings(mock_connection.id, 'test-index') + + assert fields == ['alpha', 'beta', 'zebra'] + + @patch('Common.elastic_utils.get_elastic_connection') + def test_get_field_mappings_deduplication(self, mock_get_connection, mock_connection): + """Test that duplicate fields are removed""" + mock_es = Mock() + mock_es.indices.get_mapping.return_value = { + 'test-index-1': { + 'mappings': { + 'properties': { + 'field1': {'type': 'text'} + } + } + }, + 'test-index-2': { + 'mappings': { + 'properties': { + 'field1': {'type': 'text'} + } + } + } + } + mock_get_connection.return_value = mock_es + + fields = get_elasticsearch_field_mappings(mock_connection.id, 'test-index-*') + + assert fields.count('field1') == 1 + + @patch('Common.elastic_utils.get_elastic_connection') + def test_get_field_mappings_error_handling(self, mock_get_connection, mock_connection): + """Test error handling when fetching mappings fails""" + mock_es = Mock() + mock_es.indices.get_mapping.side_effect = Exception("Index not found") + mock_get_connection.return_value = mock_es + + fields = get_elasticsearch_field_mappings(mock_connection.id, 'nonexistent-index') + + assert fields == [] + + +class TestExtractFieldNames: + """Test _extract_field_names function""" + + def test_extract_simple_fields(self): + """Test extracting simple field names""" + properties = { + 'field1': {'type': 'text'}, + 'field2': {'type': 'keyword'} + } + + fields = _extract_field_names(properties) + + assert 'field1' in fields + assert 'field2' in fields + assert len(fields) == 2 + + def test_extract_nested_fields(self): + """Test extracting nested field names""" + properties = { + 'parent': { + 'properties': { + 'child1': {'type': 'text'}, + 'child2': {'type': 'keyword'} + } + } + } + + fields = _extract_field_names(properties) + + assert 'parent' in fields + assert 'parent.child1' in fields + assert 'parent.child2' in fields + assert len(fields) == 3 + + def test_extract_deeply_nested_fields(self): + """Test extracting deeply nested field names""" + properties = { + 'level1': { + 'properties': { + 'level2': { + 'properties': { + 'level3': {'type': 'text'} + } + } + } + } + } + + fields = _extract_field_names(properties) + + assert 'level1' in fields + assert 'level1.level2' in fields + assert 'level1.level2.level3' in fields + + def test_extract_with_prefix(self): + """Test extracting field names with prefix""" + properties = { + 'field1': {'type': 'text'} + } + + fields = _extract_field_names(properties, prefix='parent') + + assert 'parent.field1' in fields + + def test_extract_empty_properties(self): + """Test extracting from empty properties""" + fields = _extract_field_names({}) + + assert fields == [] + + +class TestQueryElasticsearchDocuments: + """Test query_elasticsearch_documents function""" + + @patch('Common.elastic_utils.get_elastic_connection') + def test_query_by_document_ids(self, mock_get_connection, mock_connection): + """Test querying documents by IDs""" + mock_es = Mock() + mock_es.mget.return_value = { + 'docs': [ + {'_source': {'field': 'value1'}, 'found': True}, + {'_source': {'field': 'value2'}, 'found': True} + ] + } + mock_get_connection.return_value = mock_es + + docs = query_elasticsearch_documents( + mock_connection.id, + 'test-index', + doc_ids=['id1', 'id2'] + ) + + assert len(docs) == 2 + assert docs[0] == {'field': 'value1'} + assert docs[1] == {'field': 'value2'} + mock_es.mget.assert_called_once_with(index='test-index', ids=['id1', 'id2']) + + @patch('Common.elastic_utils.get_elastic_connection') + def test_query_by_document_ids_not_found(self, mock_get_connection, mock_connection): + """Test querying documents by IDs with some not found""" + mock_es = Mock() + mock_es.mget.return_value = { + 'docs': [ + {'_source': {'field': 'value1'}, 'found': True}, + {'found': False} + ] + } + mock_get_connection.return_value = mock_es + + docs = query_elasticsearch_documents( + mock_connection.id, + 'test-index', + doc_ids=['id1', 'id2'] + ) + + assert len(docs) == 1 + assert docs[0] == {'field': 'value1'} + + @patch('Common.elastic_utils.get_elastic_connection') + def test_query_by_field(self, mock_get_connection, mock_connection): + """Test querying documents by field""" + mock_es = Mock() + mock_es.search.return_value = { + 'hits': { + 'hits': [ + {'_source': {'field1': 'value1'}}, + {'_source': {'field1': 'value2'}} + ] + } + } + mock_get_connection.return_value = mock_es + + docs = query_elasticsearch_documents( + mock_connection.id, + 'test-index', + field='field1', + size=10 + ) + + assert len(docs) == 2 + mock_es.search.assert_called_once() + + @patch('Common.elastic_utils.get_elastic_connection') + def test_query_with_query_string(self, mock_get_connection, mock_connection): + """Test querying documents with query string""" + mock_es = Mock() + mock_es.search.return_value = { + 'hits': { + 'hits': [ + {'_source': {'field': 'value'}} + ] + } + } + mock_get_connection.return_value = mock_es + + docs = query_elasticsearch_documents( + mock_connection.id, + 'test-index', + query_string='field:value', + size=5 + ) + + assert len(docs) == 1 + call_kwargs = mock_es.search.call_args[1] + assert call_kwargs['query']['query_string']['query'] == 'field:value' + + @patch('Common.elastic_utils.get_elastic_connection') + def test_query_match_all(self, mock_get_connection, mock_connection): + """Test querying documents with match_all""" + mock_es = Mock() + mock_es.search.return_value = { + 'hits': { + 'hits': [ + {'_source': {'field': 'value'}} + ] + } + } + mock_get_connection.return_value = mock_es + + docs = query_elasticsearch_documents( + mock_connection.id, + 'test-index', + size=10 + ) + + call_kwargs = mock_es.search.call_args[1] + assert 'match_all' in call_kwargs['query'] + + @patch('Common.elastic_utils.get_elastic_connection') + def test_query_error_handling(self, mock_get_connection, mock_connection): + """Test error handling when query fails""" + mock_es = Mock() + mock_es.search.side_effect = Exception("Query error") + mock_get_connection.return_value = mock_es + + docs = query_elasticsearch_documents( + mock_connection.id, + 'test-index' + ) + + assert docs == [] + + @patch('Common.elastic_utils.get_elastic_connection') + def test_query_with_specific_field_source(self, mock_get_connection, mock_connection): + """Test querying with specific field in _source""" + mock_es = Mock() + mock_es.search.return_value = { + 'hits': { + 'hits': [ + {'_source': {'field1': 'value1'}} + ] + } + } + mock_get_connection.return_value = mock_es + + docs = query_elasticsearch_documents( + mock_connection.id, + 'test-index', + field='field1', + size=10 + ) + + call_kwargs = mock_es.search.call_args[1] + assert call_kwargs['source'] == ['field1'] + + +class TestTestElasticConnectivity: + """Tests for test_elastic_connectivity function""" + + def test_returns_json_string(self): + """Test that test_elastic_connectivity returns a JSON-formatted string""" + import json + + mock_connection = Mock() + mock_connection.info.return_value = { + 'name': 'node-1', + 'cluster_name': 'my-cluster', + 'version': {'number': '8.0.0'} + } + + result = es_test_connectivity(mock_connection) + + assert isinstance(result, str) + # Should be valid JSON + parsed = json.loads(result) + assert parsed['name'] == 'node-1' + assert parsed['cluster_name'] == 'my-cluster' + + def test_json_is_pretty_printed(self): + """Test that the JSON output is indented (pretty-printed)""" + mock_connection = Mock() + mock_connection.info.return_value = {'name': 'node-1'} + + result = es_test_connectivity(mock_connection) + + # Pretty-printed JSON contains newlines and spaces for indentation + assert '\n' in result + assert ' ' in result # 4-space indent + + def test_calls_info_on_connection(self): + """Test that .info() is called on the provided connection object""" + mock_connection = Mock() + mock_connection.info.return_value = {'name': 'node-1'} + + es_test_connectivity(mock_connection) + + mock_connection.info.assert_called_once() + + def test_returns_all_cluster_info_fields(self): + """Test that all fields from cluster info are returned in JSON""" + import json + + cluster_info = { + 'name': 'node-1', + 'cluster_name': 'test-cluster', + 'cluster_uuid': 'abc-123', + 'version': { + 'number': '8.12.0', + 'build_flavor': 'default' + }, + 'tagline': 'You Know, for Search' + } + mock_connection = Mock() + mock_connection.info.return_value = cluster_info + + result = es_test_connectivity(mock_connection) + parsed = json.loads(result) + + # All top-level keys should be present + for key in cluster_info: + assert key in parsed + + def test_propagates_exception_from_info(self): + """Test that exceptions from .info() are propagated (not swallowed)""" + mock_connection = Mock() + mock_connection.info.side_effect = ConnectionError("Cannot reach Elasticsearch") + + with pytest.raises(ConnectionError): + es_test_connectivity(mock_connection) + + def test_empty_info_response(self): + """Test behavior when info() returns an empty dict""" + import json + + mock_connection = Mock() + mock_connection.info.return_value = {} + + result = es_test_connectivity(mock_connection) + parsed = json.loads(result) + assert parsed == {} + + +class TestNormalizeKibanaUrl: + """URL-based connections often store the ES endpoint; Agent Builder needs Kibana.""" + + def test_rewrites_es_infix_to_kb(self): + assert normalize_kibana_url( + 'https://proj.es.us-east-1.aws.elastic.cloud' + ) == 'https://proj.kb.us-east-1.aws.elastic.cloud' + + def test_inserts_kb_for_alias_without_es_or_kb(self): + assert normalize_kibana_url( + 'https://logstashuiserverless-ae1d5d3b.us-east-1.aws.elastic.cloud' + ) == 'https://logstashuiserverless-ae1d5d3b.kb.us-east-1.aws.elastic.cloud' + + def test_leaves_kibana_host_unchanged(self): + url = 'https://proj.kb.us-east-1.aws.elastic.cloud' + assert normalize_kibana_url(url) == url + + def test_strips_kibana_app_path(self): + assert normalize_kibana_url( + 'https://proj.kb.us-east-1.aws.elastic.cloud/app/home' + ) == 'https://proj.kb.us-east-1.aws.elastic.cloud' + + def test_drops_es_port_when_rewriting_elastic_cloud(self): + assert normalize_kibana_url( + 'https://proj.es.us-east-1.aws.elastic.cloud:9243' + ) == 'https://proj.kb.us-east-1.aws.elastic.cloud' + + def test_preserves_self_managed_kibana_port(self): + assert normalize_kibana_url('https://localhost:5601') == 'https://localhost:5601' + + def test_adds_https_when_scheme_missing(self): + assert normalize_kibana_url( + 'proj.es.eu-west-1.gcp.elastic.cloud' + ) == 'https://proj.kb.eu-west-1.gcp.elastic.cloud' + + def test_cloud_id_found_io_host_passthrough(self): + url = 'https://abc123.us-east-1.aws.found.io' + assert normalize_kibana_url(url) == url + + def test_empty_and_none(self): + assert normalize_kibana_url('') == '' + assert normalize_kibana_url(None) is None + diff --git a/tests/Common/unit/test_encryption.py b/tests/Common/unit/test_encryption.py new file mode 100644 index 0000000..9885ea7 --- /dev/null +++ b/tests/Common/unit/test_encryption.py @@ -0,0 +1,267 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +import pytest +import os +import tempfile +from pathlib import Path +from unittest.mock import patch, mock_open, Mock +from cryptography.fernet import Fernet, InvalidToken + +from Common.encryption import ( + get_encryption_key, + encrypt_credential, + decrypt_credential, + get_django_secret_key +) + + +@pytest.fixture +def temp_data_dir(tmp_path, monkeypatch): + """Isolated data directory (LOGSTASHUI_DATA_DIR).""" + data_dir = tmp_path / "data" + data_dir.mkdir() + monkeypatch.setenv("LOGSTASHUI_DATA_DIR", str(data_dir)) + return data_dir + + +class TestGetEncryptionKey: + """Test get_encryption_key function""" + + def test_key_from_environment_variable(self, monkeypatch, temp_data_dir): + """Test loading key from CREDENTIAL_KEY environment variable""" + valid_key = Fernet.generate_key() + monkeypatch.setenv('CREDENTIAL_KEY', valid_key.decode()) + + with patch('Common.encryption.Path') as mock_path: + mock_path.return_value.resolve.return_value.parent.parent = temp_data_dir.parent + key = get_encryption_key() + + assert key == valid_key + assert isinstance(key, bytes) + + def test_invalid_key_in_environment_variable(self, monkeypatch, temp_data_dir): + """Test that invalid CREDENTIAL_KEY raises RuntimeError""" + monkeypatch.setenv('CREDENTIAL_KEY', 'invalid-key-format') + + with patch('Common.encryption.Path') as mock_path: + mock_path.return_value.resolve.return_value.parent.parent = temp_data_dir.parent + with pytest.raises(RuntimeError, match="Invalid CREDENTIAL_KEY format"): + get_encryption_key() + + def test_key_from_file(self, temp_data_dir): + """Test loading key from file""" + key_file = temp_data_dir / ".secret_key" + valid_key = Fernet.generate_key() + key_file.write_bytes(valid_key) + + with patch('Common.encryption.Path') as mock_path: + mock_path.return_value.resolve.return_value.parent.parent = temp_data_dir.parent + key = get_encryption_key() + + assert key == valid_key + + def test_invalid_key_in_file(self, temp_data_dir): + """Test that invalid key in file raises RuntimeError""" + key_file = temp_data_dir / ".secret_key" + key_file.write_bytes(b'invalid-key-data') + + with patch('Common.encryption.Path') as mock_path: + mock_path.return_value.resolve.return_value.parent.parent = temp_data_dir.parent + with pytest.raises(RuntimeError, match="Invalid encryption key in file"): + get_encryption_key() + + def test_generate_new_key_and_persist(self, temp_data_dir): + """Test generating new key and persisting to file""" + key_file = temp_data_dir / ".secret_key" + + with patch('Common.encryption.Path') as mock_path: + mock_path.return_value.resolve.return_value.parent.parent = temp_data_dir.parent + key = get_encryption_key() + + # Verify key was generated + assert isinstance(key, bytes) + assert len(key) > 0 + + # Verify key was saved to file + assert key_file.exists() + saved_key = key_file.read_bytes() + assert saved_key == key + + # Verify key is valid Fernet key + fernet = Fernet(key) + assert fernet is not None + + def test_permission_error_reading_key_file(self, temp_data_dir): + """Test handling of permission errors when reading key file""" + key_file = temp_data_dir / ".secret_key" + key_file.write_bytes(Fernet.generate_key()) + + with patch('Common.encryption.Path') as mock_path: + mock_path.return_value.resolve.return_value.parent.parent = temp_data_dir.parent + with patch('builtins.open', side_effect=PermissionError("Access denied")): + with pytest.raises(RuntimeError, match="Cannot read encryption key file: Permission denied"): + get_encryption_key() + + +class TestEncryptDecryptCredential: + """Test encrypt_credential and decrypt_credential functions""" + + def test_encrypt_decrypt_round_trip(self, temp_data_dir): + """Test that encryption and decryption work correctly""" + plaintext = "my-secret-password" + + with patch('Common.encryption.Path') as mock_path: + mock_path.return_value.resolve.return_value.parent.parent = temp_data_dir.parent + + encrypted = encrypt_credential(plaintext) + assert encrypted != plaintext + assert isinstance(encrypted, str) + + decrypted = decrypt_credential(encrypted) + assert decrypted == plaintext + + def test_encrypt_empty_string_passthrough(self): + """Test that empty string is passed through without encryption""" + assert encrypt_credential("") == "" + assert encrypt_credential(None) is None + + def test_decrypt_empty_string_passthrough(self): + """Test that empty string is passed through without decryption""" + assert decrypt_credential("") == "" + assert decrypt_credential(None) is None + + def test_encrypt_non_string_raises_error(self): + """Test that encrypting non-string raises ValueError""" + with pytest.raises(ValueError, match="plaintext must be a string"): + encrypt_credential(123) + + with pytest.raises(ValueError, match="plaintext must be a string"): + encrypt_credential(['list']) + + def test_decrypt_non_string_raises_error(self): + """Test that decrypting non-string raises ValueError""" + with pytest.raises(ValueError, match="encrypted_text must be a string"): + decrypt_credential(123) + + with pytest.raises(ValueError, match="encrypted_text must be a string"): + decrypt_credential(['list']) + + def test_decrypt_invalid_token(self, temp_data_dir): + """Test that decrypting with invalid token raises ValueError""" + with patch('Common.encryption.Path') as mock_path: + mock_path.return_value.resolve.return_value.parent.parent = temp_data_dir.parent + + with pytest.raises(ValueError, match="Cannot decrypt credential: Invalid token"): + decrypt_credential("invalid-encrypted-data") + + def test_decrypt_with_wrong_key(self, temp_data_dir): + """Test that decrypting with wrong key raises ValueError""" + # Encrypt with one key + key1 = Fernet.generate_key() + fernet1 = Fernet(key1) + encrypted = fernet1.encrypt(b"secret").decode() + + # Try to decrypt with different key + key_file = temp_data_dir / ".secret_key" + key2 = Fernet.generate_key() + key_file.write_bytes(key2) + + with patch('Common.encryption.Path') as mock_path: + mock_path.return_value.resolve.return_value.parent.parent = temp_data_dir.parent + + with pytest.raises(ValueError, match="Cannot decrypt credential: Invalid token"): + decrypt_credential(encrypted) + + def test_encrypt_unicode_characters(self, temp_data_dir): + """Test encrypting and decrypting unicode characters""" + plaintext = "🔒 Secret with émojis and spëcial çhars 中文" + + with patch('Common.encryption.Path') as mock_path: + mock_path.return_value.resolve.return_value.parent.parent = temp_data_dir.parent + + encrypted = encrypt_credential(plaintext) + decrypted = decrypt_credential(encrypted) + assert decrypted == plaintext + + +class TestGetDjangoSecretKey: + """Test get_django_secret_key function""" + + def test_key_from_environment_variable(self, monkeypatch, temp_data_dir): + """Test loading Django secret key from environment variable""" + secret_key = "test-secret-key-from-environment-variable-long-enough" + monkeypatch.setenv('SECRET_KEY', secret_key) + + with patch('Common.encryption.Path') as mock_path: + mock_path.return_value.resolve.return_value.parent.parent = temp_data_dir.parent + key = get_django_secret_key() + + assert key == secret_key + + def test_short_key_warning(self, monkeypatch, temp_data_dir, caplog): + """Test that short SECRET_KEY generates warning""" + short_key = "short" + monkeypatch.setenv('SECRET_KEY', short_key) + + with patch('Common.encryption.Path') as mock_path: + mock_path.return_value.resolve.return_value.parent.parent = temp_data_dir.parent + key = get_django_secret_key() + + assert key == short_key + assert "SECRET_KEY from environment is short" in caplog.text + + def test_key_from_file(self, temp_data_dir): + """Test loading Django secret key from file""" + key_file = temp_data_dir / ".django_secret_key" + secret_key = "test-secret-key-from-file-should-be-long-enough-now" + key_file.write_text(secret_key) + + with patch('Common.encryption.Path') as mock_path: + mock_path.return_value.resolve.return_value.parent.parent = temp_data_dir.parent + key = get_django_secret_key() + + assert key == secret_key + + def test_empty_key_file_raises_error(self, temp_data_dir): + """Test that empty key file raises RuntimeError""" + key_file = temp_data_dir / ".django_secret_key" + key_file.write_text("") + + with patch('Common.encryption.Path') as mock_path: + mock_path.return_value.resolve.return_value.parent.parent = temp_data_dir.parent + with pytest.raises(RuntimeError, match="Django secret key file is empty"): + get_django_secret_key() + + def test_generate_new_key_and_persist(self, temp_data_dir): + """Test generating new Django secret key and persisting to file""" + key_file = temp_data_dir / ".django_secret_key" + + with patch('Common.encryption.Path') as mock_path: + mock_path.return_value.resolve.return_value.parent.parent = temp_data_dir.parent + key = get_django_secret_key() + + # Verify key was generated + assert isinstance(key, str) + assert len(key) == 50 + + # Verify key contains expected characters + valid_chars = set('abcdefghijklmnopqrstuvwxyz0123456789!@#$%^&*(-_=+)') + assert all(c in valid_chars for c in key) + + # Verify key was saved to file + assert key_file.exists() + saved_key = key_file.read_text().strip() + assert saved_key == key + + def test_permission_error_reading_key_file(self, temp_data_dir): + """Test handling of permission errors when reading Django secret key file""" + key_file = temp_data_dir / ".django_secret_key" + key_file.write_text("test-key") + + with patch('Common.encryption.Path') as mock_path: + mock_path.return_value.resolve.return_value.parent.parent = temp_data_dir.parent + with patch('builtins.open', side_effect=PermissionError("Access denied")): + with pytest.raises(RuntimeError, match="Cannot read Django secret key: Permission denied"): + get_django_secret_key() diff --git a/tests/Common/unit/test_error_handlers.py b/tests/Common/unit/test_error_handlers.py new file mode 100644 index 0000000..bf80652 --- /dev/null +++ b/tests/Common/unit/test_error_handlers.py @@ -0,0 +1,263 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +import pytest +from django.test import RequestFactory + +from Common.error_handlers import handler400, handler403, handler404, handler500 + +# All tests in this file require DB access because Django's render() triggers +# context processors (navigation_highlight) that query Connection/Device models. +pytestmark = pytest.mark.django_db + + +@pytest.fixture +def request_factory(): + """Django RequestFactory for creating mock requests""" + return RequestFactory() + + +@pytest.fixture +def mock_request(request_factory): + """Create a basic GET mock request""" + return request_factory.get('/some/path/') + + +class TestHandler400: + """Tests for handler400 (Bad Request)""" + + def test_returns_400_status(self, mock_request): + """Test that handler400 returns a 400 status code""" + response = handler400(mock_request) + assert response.status_code == 400 + + def test_correct_error_code_in_context(self, mock_request): + """Test that context contains correct error code""" + response = handler400(mock_request) + assert response.context_data['error_code'] == '400' + + def test_correct_error_title(self, mock_request): + """Test that context contains correct error title""" + response = handler400(mock_request) + assert response.context_data['error_title'] == 'Bad Request' + + def test_correct_error_message(self, mock_request): + """Test that context contains appropriate error message""" + response = handler400(mock_request) + assert 'could not understand' in response.context_data['error_message'] + + def test_with_exception(self, mock_request): + """Test that exception class name is included when exception is provided""" + exc = ValueError("bad input") + response = handler400(mock_request, exception=exc) + assert response.context_data['exception'] == 'ValueError' + + def test_without_exception(self, mock_request): + """Test that exception is None when no exception is provided""" + response = handler400(mock_request) + assert response.context_data['exception'] is None + + def test_with_none_exception(self, mock_request): + """Test that explicitly passing None exception gives None in context""" + response = handler400(mock_request, exception=None) + assert response.context_data['exception'] is None + + def test_uses_error_template(self, mock_request): + """Test that the error.html template is used""" + response = handler400(mock_request) + assert response.template_name == 'error.html' + + +class TestHandler403: + """Tests for handler403 (Access Denied)""" + + def test_returns_403_status(self, mock_request): + """Test that handler403 returns a 403 status code""" + response = handler403(mock_request) + assert response.status_code == 403 + + def test_correct_error_code(self, mock_request): + """Test correct error code in context""" + response = handler403(mock_request) + assert response.context_data['error_code'] == '403' + + def test_correct_error_title(self, mock_request): + """Test correct error title""" + response = handler403(mock_request) + assert response.context_data['error_title'] == 'Access Denied' + + def test_correct_error_message(self, mock_request): + """Test permission-related error message""" + response = handler403(mock_request) + assert 'permission' in response.context_data['error_message'].lower() + + def test_with_exception(self, mock_request): + """Test exception class name in context""" + exc = PermissionError("Access denied") + response = handler403(mock_request, exception=exc) + assert response.context_data['exception'] == 'PermissionError' + + def test_without_exception(self, mock_request): + """Test that exception is None when not provided""" + response = handler403(mock_request) + assert response.context_data['exception'] is None + + def test_uses_error_template(self, mock_request): + """Test that the error.html template is used""" + response = handler403(mock_request) + assert response.template_name == 'error.html' + + +class TestHandler404: + """Tests for handler404 (Page Not Found)""" + + def test_returns_404_status(self, mock_request): + """Test that handler404 returns a 404 status code""" + response = handler404(mock_request) + assert response.status_code == 404 + + def test_correct_error_code(self, mock_request): + """Test correct error code in context""" + response = handler404(mock_request) + assert response.context_data['error_code'] == '404' + + def test_correct_error_title(self, mock_request): + """Test correct error title""" + response = handler404(mock_request) + assert response.context_data['error_title'] == 'Page Not Found' + + def test_correct_error_message(self, mock_request): + """Test not-found error message""" + response = handler404(mock_request) + assert 'does not exist' in response.context_data['error_message'] + + def test_path_included_in_context(self, request_factory): + """Test that request path is included in context for 404""" + request = request_factory.get('/missing/page/') + response = handler404(request) + assert response.context_data['path'] == '/missing/page/' + + def test_with_exception(self, mock_request): + """Test exception class name in context""" + exc = LookupError("Not found") + response = handler404(mock_request, exception=exc) + assert response.context_data['exception'] == 'LookupError' + + def test_without_exception(self, mock_request): + """Test that exception is None when not provided""" + response = handler404(mock_request) + assert response.context_data['exception'] is None + + def test_uses_error_template(self, mock_request): + """Test that the error.html template is used""" + response = handler404(mock_request) + assert response.template_name == 'error.html' + + def test_path_reflects_actual_request(self, request_factory): + """Test that path value matches the actual request path""" + request = request_factory.get('/admin/nonexistent/') + response = handler404(request) + assert response.context_data['path'] == '/admin/nonexistent/' + + +class TestHandler500: + """Tests for handler500 (Server Error)""" + + def test_returns_500_status(self, mock_request): + """Test that handler500 returns a 500 status code""" + response = handler500(mock_request) + assert response.status_code == 500 + + def test_correct_error_code(self, mock_request): + """Test correct error code in context""" + response = handler500(mock_request) + assert response.context_data['error_code'] == '500' + + def test_correct_error_title(self, mock_request): + """Test correct error title""" + response = handler500(mock_request) + assert response.context_data['error_title'] == 'Server Error' + + def test_correct_error_message(self, mock_request): + """Test server error message""" + response = handler500(mock_request) + assert 'Something went wrong' in response.context_data['error_message'] + + def test_path_included_in_context(self, request_factory): + """Test that request path is included in context for 500""" + request = request_factory.get('/api/some-endpoint/') + response = handler500(request) + assert response.context_data['path'] == '/api/some-endpoint/' + + def test_with_exception(self, mock_request): + """Test exception class name in context""" + exc = RuntimeError("Something blew up") + response = handler500(mock_request, exception=exc) + assert response.context_data['exception'] == 'RuntimeError' + + def test_without_exception(self, mock_request): + """Test that exception is None when not provided""" + response = handler500(mock_request) + assert response.context_data['exception'] is None + + def test_uses_error_template(self, mock_request): + """Test that the error.html template is used""" + response = handler500(mock_request) + assert response.template_name == 'error.html' + + def test_path_reflects_actual_request(self, request_factory): + """Test that path value matches the actual request path""" + request = request_factory.get('/pipeline/1/deploy/') + response = handler500(request) + assert response.context_data['path'] == '/pipeline/1/deploy/' + + +class TestErrorHandlerEdgeCases: + """Edge case tests across all error handlers""" + + def test_all_handlers_use_same_template(self, mock_request): + """Test that all four handlers use the same error.html template""" + for handler in [handler400, handler403, handler404, handler500]: + response = handler(mock_request) + assert response.template_name == 'error.html', \ + f"{handler.__name__} should use 'error.html' template" + + def test_exception_class_name_not_message(self, mock_request): + """Test that context stores class name, not exception message""" + exc = ValueError("This is the message, not the class name") + response = handler400(mock_request, exception=exc) + # Should be class name, not the message + assert response.context_data['exception'] == 'ValueError' + assert 'message' not in response.context_data['exception'] + + def test_handler404_and_500_include_path_not_400_403(self, mock_request): + """Test that only 404 and 500 include path in context""" + # 400 and 403 should NOT have path + assert 'path' not in handler400(mock_request).context_data + assert 'path' not in handler403(mock_request).context_data + # 404 and 500 SHOULD have path + assert 'path' in handler404(mock_request).context_data + assert 'path' in handler500(mock_request).context_data + + @pytest.mark.parametrize("handler,expected_status", [ + (handler400, 400), + (handler403, 403), + (handler404, 404), + (handler500, 500), + ]) + def test_status_codes(self, mock_request, handler, expected_status): + """Parametrized test verifying each handler returns the correct status code""" + response = handler(mock_request) + assert response.status_code == expected_status + + @pytest.mark.parametrize("handler,expected_code", [ + (handler400, '400'), + (handler403, '403'), + (handler404, '404'), + (handler500, '500'), + ]) + def test_error_codes_in_context(self, mock_request, handler, expected_code): + """Parametrized test verifying error_code in context matches HTTP status""" + response = handler(mock_request) + assert response.context_data['error_code'] == expected_code diff --git a/tests/Common/unit/test_formatters.py b/tests/Common/unit/test_formatters.py new file mode 100644 index 0000000..c0c2a4f --- /dev/null +++ b/tests/Common/unit/test_formatters.py @@ -0,0 +1,407 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +import pytest +from Common.formatters import ( + _safe_get_numeric, + _safe_extract_value, + _format_uptime, + _sanitize_pipeline_name_component +) + + +class TestSafeGetNumeric: + """Test _safe_get_numeric function with edge cases""" + + def test_integer_value(self): + """Test integer value is returned as-is""" + assert _safe_get_numeric(42) == 42 + assert isinstance(_safe_get_numeric(42), int) + + def test_float_value(self): + """Test float value is returned as-is""" + assert _safe_get_numeric(3.14) == 3.14 + assert isinstance(_safe_get_numeric(3.14), float) + + def test_string_integer(self): + """Test string integer is converted to int""" + assert _safe_get_numeric("123") == 123 + assert isinstance(_safe_get_numeric("123"), int) + + def test_string_float(self): + """Test string float is converted to float""" + assert _safe_get_numeric("3.14") == 3.14 + assert isinstance(_safe_get_numeric("3.14"), float) + + def test_none_returns_default(self): + """Test None returns default value""" + assert _safe_get_numeric(None) == 0 + assert _safe_get_numeric(None, default=100) == 100 + + def test_empty_list_returns_default(self): + """Test empty list returns default value""" + assert _safe_get_numeric([]) == 0 + assert _safe_get_numeric([], default=50) == 50 + + def test_list_with_integer(self): + """Test list with integer returns first element""" + assert _safe_get_numeric([42]) == 42 + assert _safe_get_numeric([42, 100]) == 42 + + def test_list_with_float(self): + """Test list with float returns first element""" + assert _safe_get_numeric([3.14]) == 3.14 + assert _safe_get_numeric([3.14, 2.71]) == 3.14 + + def test_list_with_string_number(self): + """Test list with string number converts first element""" + assert _safe_get_numeric(["123"]) == 123 + assert _safe_get_numeric(["3.14"]) == 3.14 + + def test_invalid_string_returns_default(self): + """Test invalid string returns default value""" + assert _safe_get_numeric("not a number") == 0 + assert _safe_get_numeric("abc", default=99) == 99 + + def test_list_with_invalid_string_returns_default(self): + """Test list with invalid string returns default""" + assert _safe_get_numeric(["invalid"]) == 0 + assert _safe_get_numeric(["invalid"], default=77) == 77 + + def test_boolean_value(self): + """Test boolean values (True=1, False=0)""" + assert _safe_get_numeric(True) == 1 + assert _safe_get_numeric(False) == 0 + + def test_zero_value(self): + """Test zero is returned correctly""" + assert _safe_get_numeric(0) == 0 + assert _safe_get_numeric("0") == 0 + assert _safe_get_numeric([0]) == 0 + + def test_negative_numbers(self): + """Test negative numbers are handled correctly""" + assert _safe_get_numeric(-42) == -42 + assert _safe_get_numeric("-42") == -42 + assert _safe_get_numeric([-3.14]) == -3.14 + + def test_custom_default_value(self): + """Test custom default values work correctly""" + assert _safe_get_numeric(None, default=-1) == -1 + assert _safe_get_numeric([], default=999) == 999 + assert _safe_get_numeric("invalid", default=42) == 42 + + def test_list_with_none_returns_default(self): + """Test list containing None returns default""" + assert _safe_get_numeric([None]) == 0 + assert _safe_get_numeric([None], default=10) == 10 + + def test_scientific_notation(self): + """Test scientific notation strings - not supported, returns default""" + # The function uses '.' check for float detection, so "1e3" is treated as invalid + assert _safe_get_numeric("1e3") == 0 + # "1.5e2" has a period so it tries float() which works + assert _safe_get_numeric("1.5e2") == 150.0 + + def test_whitespace_in_string(self): + """Test strings with whitespace""" + assert _safe_get_numeric(" 123 ") == 123 + assert _safe_get_numeric(" 3.14 ") == 3.14 + + def test_dict_returns_default(self): + """Test dict returns default value""" + assert _safe_get_numeric({"value": 123}) == 0 + assert _safe_get_numeric({"value": 123}, default=5) == 5 + + +class TestSafeExtractValue: + """Test _safe_extract_value function with edge cases""" + + def test_simple_value(self): + """Test simple values are returned as-is""" + assert _safe_extract_value("test") == "test" + assert _safe_extract_value(123) == 123 + assert _safe_extract_value(3.14) == 3.14 + + def test_none_returns_default(self): + """Test None returns default value""" + assert _safe_extract_value(None) == 0 + assert _safe_extract_value(None, default="default") == "default" + + def test_empty_list_returns_default(self): + """Test empty list returns default value""" + assert _safe_extract_value([]) == 0 + assert _safe_extract_value([], default="empty") == "empty" + + def test_list_with_value(self): + """Test list with value returns first element""" + assert _safe_extract_value(["test"]) == "test" + assert _safe_extract_value([123]) == 123 + assert _safe_extract_value(["first", "second"]) == "first" + + def test_list_with_none_returns_default(self): + """Test list with None values returns default""" + assert _safe_extract_value([None]) == 0 + assert _safe_extract_value([None, None]) == 0 + assert _safe_extract_value([None], default="none") == "none" + + def test_list_with_empty_string_returns_default(self): + """Test list with empty strings returns default""" + assert _safe_extract_value([""]) == 0 + assert _safe_extract_value(["", ""]) == 0 + assert _safe_extract_value([""], default="empty") == "empty" + + def test_list_with_mixed_none_and_empty(self): + """Test list with mix of None and empty strings returns default""" + assert _safe_extract_value([None, "", None]) == 0 + assert _safe_extract_value(["", None, ""], default=99) == 99 + + def test_list_with_valid_after_invalid(self): + """Test list returns first non-null, non-empty value""" + assert _safe_extract_value([None, "valid"]) == "valid" + assert _safe_extract_value(["", "valid"]) == "valid" + assert _safe_extract_value([None, "", "valid"]) == "valid" + + def test_list_with_zero(self): + """Test list with zero value (zero is valid, not empty)""" + assert _safe_extract_value([0]) == 0 + assert _safe_extract_value([None, 0]) == 0 + + def test_list_with_false(self): + """Test list with False value (False is valid, not empty)""" + assert _safe_extract_value([False]) is False + assert _safe_extract_value([None, False]) is False + + def test_custom_default_value(self): + """Test custom default values""" + assert _safe_extract_value(None, default="custom") == "custom" + assert _safe_extract_value([], default=999) == 999 + + def test_dict_value(self): + """Test dict values are returned as-is""" + test_dict = {"key": "value"} + assert _safe_extract_value(test_dict) == test_dict + + def test_boolean_values(self): + """Test boolean values are returned correctly""" + assert _safe_extract_value(True) is True + assert _safe_extract_value(False) is False + + def test_list_with_whitespace_string(self): + """Test list with whitespace-only string (treated as non-empty)""" + assert _safe_extract_value([" "]) == " " + assert _safe_extract_value([" "]) == " " + + +class TestFormatUptime: + """Test _format_uptime function with edge cases""" + + def test_zero_milliseconds(self): + """Test zero milliseconds""" + assert _format_uptime(0) == "0s" + + def test_seconds_only(self): + """Test uptime in seconds only""" + assert _format_uptime(5000) == "5s" + assert _format_uptime(59000) == "59s" + + def test_minutes_and_seconds(self): + """Test uptime in minutes and seconds""" + assert _format_uptime(60000) == "1m 0s" + assert _format_uptime(90000) == "1m 30s" + assert _format_uptime(3599000) == "59m 59s" + + def test_hours_and_minutes(self): + """Test uptime in hours and minutes""" + assert _format_uptime(3600000) == "1h 0m" + assert _format_uptime(3660000) == "1h 1m" + assert _format_uptime(7200000) == "2h 0m" + assert _format_uptime(86399000) == "23h 59m" + + def test_days_and_hours(self): + """Test uptime in days and hours""" + assert _format_uptime(86400000) == "1d 0h" + assert _format_uptime(90000000) == "1d 1h" + assert _format_uptime(172800000) == "2d 0h" + assert _format_uptime(176400000) == "2d 1h" + + def test_one_millisecond(self): + """Test one millisecond rounds to 0 seconds""" + assert _format_uptime(1) == "0s" + + def test_999_milliseconds(self): + """Test 999 milliseconds rounds to 0 seconds""" + assert _format_uptime(999) == "0s" + + def test_exactly_one_minute(self): + """Test exactly one minute""" + assert _format_uptime(60000) == "1m 0s" + + def test_exactly_one_hour(self): + """Test exactly one hour""" + assert _format_uptime(3600000) == "1h 0m" + + def test_exactly_one_day(self): + """Test exactly one day""" + assert _format_uptime(86400000) == "1d 0h" + + def test_large_uptime(self): + """Test large uptime values""" + # 30 days + assert _format_uptime(2592000000) == "30d 0h" + # 365 days + assert _format_uptime(31536000000) == "365d 0h" + + def test_complex_uptime(self): + """Test complex uptime with all components""" + # 1 day, 2 hours, 3 minutes, 4 seconds, 500 milliseconds + ms = (1 * 86400000) + (2 * 3600000) + (3 * 60000) + (4 * 1000) + 500 + assert _format_uptime(ms) == "1d 2h" + + def test_uptime_priority_days_over_hours(self): + """Test that days format takes priority over hours""" + # 1 day, 23 hours + ms = (1 * 86400000) + (23 * 3600000) + assert _format_uptime(ms) == "1d 23h" + + def test_uptime_priority_hours_over_minutes(self): + """Test that hours format takes priority over minutes""" + # 1 hour, 59 minutes + ms = (1 * 3600000) + (59 * 60000) + assert _format_uptime(ms) == "1h 59m" + + def test_uptime_priority_minutes_over_seconds(self): + """Test that minutes format takes priority over seconds""" + # 1 minute, 59 seconds + ms = (1 * 60000) + (59 * 1000) + assert _format_uptime(ms) == "1m 59s" + + def test_negative_uptime(self): + """Test negative uptime (edge case, should handle gracefully)""" + # Negative values will result in negative calculations + result = _format_uptime(-1000) + assert "s" in result + + def test_fractional_seconds(self): + """Test that fractional seconds are truncated""" + # 1500ms = 1.5 seconds, should show as 1s + assert _format_uptime(1500) == "1s" + # 2999ms = 2.999 seconds, should show as 2s + assert _format_uptime(2999) == "2s" + + @pytest.mark.parametrize("milliseconds,expected", [ + (0, "0s"), + (1000, "1s"), + (60000, "1m 0s"), + (3600000, "1h 0m"), + (86400000, "1d 0h"), + (90061000, "1d 1h"), # 1 day, 1 hour, 1 minute, 1 second + (5000, "5s"), + (125000, "2m 5s"), + (7325000, "2h 2m"), + (90000000, "1d 1h"), + ]) + def test_parametrized_uptime_formats(self, milliseconds, expected): + """Test various uptime formats with parametrized inputs""" + assert _format_uptime(milliseconds) == expected + + +class TestSanitizePipelineNameComponent: + """Test _sanitize_pipeline_name_component function""" + + def test_simple_valid_name(self): + """Test simple valid name is returned as-is (lowercased)""" + assert _sanitize_pipeline_name_component('myname') == 'myname' + + def test_uppercase_is_lowercased(self): + """Test that uppercase letters are lowercased""" + assert _sanitize_pipeline_name_component('MyName') == 'myname' + assert _sanitize_pipeline_name_component('UPPERCASE') == 'uppercase' + + def test_numbers_are_preserved(self): + """Test that numbers are kept""" + assert _sanitize_pipeline_name_component('name123') == 'name123' + assert _sanitize_pipeline_name_component('abc456def') == 'abc456def' + + def test_underscores_are_preserved(self): + """Test that underscores are kept""" + assert _sanitize_pipeline_name_component('my_name') == 'my_name' + assert _sanitize_pipeline_name_component('a_b_c') == 'a_b_c' + + def test_hyphens_are_preserved(self): + """Test that hyphens are kept""" + assert _sanitize_pipeline_name_component('my-name') == 'my-name' + assert _sanitize_pipeline_name_component('a-b-c') == 'a-b-c' + + def test_spaces_replaced_with_underscore(self): + """Test that spaces are replaced with underscores""" + assert _sanitize_pipeline_name_component('my name') == 'my_name' + assert _sanitize_pipeline_name_component('hello world foo') == 'hello_world_foo' + + def test_special_characters_replaced_with_underscore(self): + """Test that special characters are replaced with underscores""" + assert _sanitize_pipeline_name_component('name@host') == 'name_host' + assert _sanitize_pipeline_name_component('name.value') == 'name_value' + assert _sanitize_pipeline_name_component('name/path') == 'name_path' + assert _sanitize_pipeline_name_component('name:port') == 'name_port' + + def test_consecutive_underscores_collapsed(self): + """Test that consecutive underscores are collapsed to a single one""" + assert _sanitize_pipeline_name_component('a__b') == 'a_b' + assert _sanitize_pipeline_name_component('a___b') == 'a_b' + # Multiple special chars in a row → multiple underscores → collapsed + assert _sanitize_pipeline_name_component('a@#b') == 'a_b' + + def test_leading_underscores_stripped(self): + """Test that leading underscores are stripped""" + assert _sanitize_pipeline_name_component('_name') == 'name' + assert _sanitize_pipeline_name_component('__name') == 'name' + + def test_trailing_underscores_stripped(self): + """Test that trailing underscores are stripped""" + assert _sanitize_pipeline_name_component('name_') == 'name' + assert _sanitize_pipeline_name_component('name__') == 'name' + + def test_leading_special_chars_stripped(self): + """Test that leading special characters (→ underscores) are stripped""" + # '@name' → '_name' (special char → underscore) → 'name' (stripped leading) + assert _sanitize_pipeline_name_component('@name') == 'name' + + def test_all_allowed_chars(self): + """Test a name using all allowed character types""" + result = _sanitize_pipeline_name_component('MyName-123_test') + assert result == 'myname-123_test' + + def test_empty_string(self): + """Test that empty string returns empty string""" + assert _sanitize_pipeline_name_component('') == '' + + def test_only_special_chars(self): + """Test that a string of only special chars results in empty string""" + result = _sanitize_pipeline_name_component('@@@@') + # All replaced with underscores, collapsed, then stripped → empty + assert result == '' + + def test_unicode_replaced_with_underscore(self): + """Test that unicode/emoji chars are replaced with underscores""" + result = _sanitize_pipeline_name_component('name_中文') + # Non-ASCII chars each become '_', consecutive collapsed, trailing stripped + assert result == 'name' + + def test_already_clean_name(self): + """Test that a clean name passes through unchanged (except lowercasing)""" + assert _sanitize_pipeline_name_component('good-name_123') == 'good-name_123' + + @pytest.mark.parametrize('input_name,expected', [ + ('Hello World', 'hello_world'), + ('My-Pipeline_Name', 'my-pipeline_name'), + (' spaces ', 'spaces'), + ('a.b.c', 'a_b_c'), + ('test!@#$%', 'test'), + ('UPPER_LOWER', 'upper_lower'), + ('123abc', '123abc'), + ]) + def test_parametrized_sanitization(self, input_name, expected): + """Parametrized test for various sanitization scenarios""" + assert _sanitize_pipeline_name_component(input_name) == expected diff --git a/tests/Common/unit/test_logstash_config_parse.py b/tests/Common/unit/test_logstash_config_parse.py new file mode 100644 index 0000000..9f279b2 --- /dev/null +++ b/tests/Common/unit/test_logstash_config_parse.py @@ -0,0 +1,542 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +import pytest +import json + +from Common.logstash_config_parse import ( + _extract_error_context, + _strip_inline_comments, + parse_logstash_config, + logstash_config_to_components, + ComponentToPipeline, +) + + +# ───────────────────────────────────────────────────────────── +# Helpers / shared fixtures +# ───────────────────────────────────────────────────────────── + +MINIMAL_INPUT = 'input {\n stdin {}\n}\n' +MINIMAL_FILTER = 'filter {\n mutate {}\n}\n' +MINIMAL_OUTPUT = 'output {\n stdout {}\n}\n' +FULL_PIPELINE = MINIMAL_INPUT + MINIMAL_FILTER + MINIMAL_OUTPUT + + +# ───────────────────────────────────────────────────────────── +# _extract_error_context +# ───────────────────────────────────────────────────────────── + +class TestExtractErrorContext: + """Unit tests for _extract_error_context""" + + def test_contains_separator_lines(self): + """Result includes the === separator lines""" + config = "input {\n stdin {}\n}\n" + result = _extract_error_context(config, line=1, column=1) + assert '=' * 60 in result + + def test_contains_problematic_code_header(self): + """Result contains the PROBLEMATIC CODE label""" + config = "input {\n stdin {}\n}\n" + result = _extract_error_context(config, line=1, column=1) + assert 'PROBLEMATIC CODE' in result + + def test_error_line_marked_with_arrow(self): + """The error line is prefixed with '>>>'""" + config = "input {\n bad_line\n}\n" + result = _extract_error_context(config, line=2, column=1) + assert '>>>' in result + + def test_non_error_lines_not_marked(self): + """Non-error lines have normal indentation, not '>>>'""" + config = "input {\n bad_line\n}\n" + result = _extract_error_context(config, line=2, column=1) + lines = result.split('\n') + # Lines that aren't the error line should start with spaces, not >>> + non_arrow_lines = [l for l in lines if l.strip() and not l.strip().startswith(('=', 'P', '>')) and '|' in l] + for line in non_arrow_lines: + assert not line.startswith('>>>') + + def test_column_pointer_added(self): + """A '^' pointer is added at the correct column position""" + config = "input {\n bad_line\n}\n" + result = _extract_error_context(config, line=2, column=3) + assert '^' in result + + def test_no_pointer_for_column_zero(self): + """No '^' pointer when column is 0""" + config = "input {\n bad_line\n}\n" + result = _extract_error_context(config, line=2, column=0) + assert '^' not in result + + def test_line_number_appears_in_output(self): + """The error line number appears in the formatted output""" + config = "line1\nline2\nline3\n" + result = _extract_error_context(config, line=2, column=1) + assert '2' in result + + def test_context_lines_parameter_controls_range(self): + """context_lines parameter controls how many surrounding lines are shown""" + config = "\n".join([f"line{i}" for i in range(1, 21)]) + # With 1 context line around line 10, we should see lines 9, 10, 11 + result_1 = _extract_error_context(config, line=10, column=1, context_lines=1) + result_5 = _extract_error_context(config, line=10, column=1, context_lines=5) + # Larger context means more lines shown + assert len(result_5) > len(result_1) + + def test_first_line_error_no_index_error(self): + """Error at line 1 should not cause an IndexError""" + config = "bad config\nmore\n" + result = _extract_error_context(config, line=1, column=1) + assert '>>>' in result + + def test_last_line_error_no_index_error(self): + """Error at last line should not cause an IndexError""" + config = "good\nbad" + result = _extract_error_context(config, line=2, column=1) + assert '>>>' in result + + def test_returns_string(self): + """Function always returns a string""" + config = "input {}" + result = _extract_error_context(config, line=1, column=1) + assert isinstance(result, str) + + +# ───────────────────────────────────────────────────────────── +# _strip_inline_comments +# ───────────────────────────────────────────────────────────── + +class TestStripInlineComments: + """Unit tests for _strip_inline_comments""" + + def test_removes_inline_comment_after_value(self): + """Inline comments inside plugin blocks are stripped from the value line + and re-injected as standalone comment lines before the plugin's closing }""" + config = 'input {\n beats {\n port => 5044 # this is a comment\n }\n}\n' + result = _strip_inline_comments(config) + # The comment is moved, not discarded — it should appear somewhere in the output + assert '# this is a comment' in result + # The original value line must have the comment stripped + lines = result.split('\n') + port_line = next(l for l in lines if 'port' in l) + assert '# this is a comment' not in port_line + assert 'port => 5044' in port_line + + def test_removes_inline_comment_after_closing_brace(self): + """Inline comments on a plugin's closing } line are emitted as a + standalone comment line at the outer scope after the }""" + config = 'input {\n beats {\n port => 5044\n } # end beats\n}\n' + result = _strip_inline_comments(config) + # Comment is re-emitted after the closing }, not discarded + assert '# end beats' in result + # The } line itself must be clean + lines = result.split('\n') + close_line = next(l for l in lines if l.strip() == '}') + assert '# end beats' not in close_line + + def test_preserves_hash_in_string_value(self): + """# inside a quoted string is NOT treated as a comment""" + config = 'input {\n beats {\n path => "/var/log/#file"\n }\n}\n' + result = _strip_inline_comments(config) + assert '/var/log/#file' in result + + def test_preserves_standalone_comment_at_section_level(self): + """Standalone comment lines at the top/section level are preserved""" + config = '# This is a top-level comment\ninput {\n stdin {}\n}\n' + result = _strip_inline_comments(config) + assert '# This is a top-level comment' in result + + def test_preserves_standalone_comment_inside_plugin_block(self): + """Standalone comment lines INSIDE a plugin block are preserved in place + (the grammar now supports them and attaches them as plugin.comments[])""" + config = 'input {\n beats {\n # comment inside plugin\n port => 5044\n }\n}\n' + result = _strip_inline_comments(config) + assert '# comment inside plugin' in result + assert 'port => 5044' in result + + def test_preserves_standalone_comment_inside_conditional(self): + """Standalone comment lines inside an if/else block are preserved""" + config = ( + 'filter {\n' + ' if [type] == "syslog" {\n' + ' # comment in conditional\n' + ' mutate {}\n' + ' }\n' + '}\n' + ) + result = _strip_inline_comments(config) + assert '# comment in conditional' in result + + def test_returns_string(self): + """Function returns a string""" + assert isinstance(_strip_inline_comments('input { stdin {} }'), str) + + def test_no_comments_unchanged(self): + """Config with no comments is returned unchanged""" + config = 'input {\n stdin {}\n}\n' + result = _strip_inline_comments(config) + assert result == config + + def test_trailing_whitespace_before_comment_removed(self): + """Trailing whitespace before the inline comment is also stripped""" + config = 'input {\n beats {\n port => 5044 # trailing spaces before comment\n }\n}\n' + result = _strip_inline_comments(config) + # The value should be present without trailing spaces + lines_with_port = [l for l in result.split('\n') if 'port' in l] + assert len(lines_with_port) == 1 + assert lines_with_port[0].endswith('5044') + + +# ───────────────────────────────────────────────────────────── +# parse_logstash_config +# ───────────────────────────────────────────────────────────── + +class TestParseLogstashConfig: + """Unit tests for parse_logstash_config + + Note on return type: Lark's LALR transformer with a single section returns + a bare dict (not a list). With multiple sections it returns a list of dicts. + This matches the defensive check inside logstash_config_to_components: + if not isinstance(parsed, list): parsed = [parsed] + The helper _as_list() below mirrors that normalization. + """ + + @staticmethod + def _as_list(result): + """Normalize to list regardless of whether one or multiple sections came back.""" + if isinstance(result, list): + return result + return [result] if result else [] + + def test_parses_single_section_as_dict_or_list(self): + """Single-section configs are returned as a dict (Lark unwraps single items)""" + result = parse_logstash_config(MINIMAL_INPUT) + assert isinstance(result, (dict, list)) + + def test_parses_all_three_sections(self): + """All three sections (input, filter, output) are parsed into a list""" + result = parse_logstash_config(FULL_PIPELINE) + assert isinstance(result, list) + types = [s['type'] for s in result] + assert 'input' in types + assert 'filter' in types + assert 'output' in types + + def test_single_section_has_correct_type(self): + """A single input section has type == 'input'""" + result = self._as_list(parse_logstash_config(MINIMAL_INPUT)) + assert len(result) == 1 + assert result[0]['type'] == 'input' + + def test_each_section_has_statements(self): + """Each parsed section has a 'statements' key""" + result = self._as_list(parse_logstash_config(MINIMAL_INPUT)) + for section in result: + assert 'statements' in section + + def test_plugin_name_captured(self): + """The plugin name ('stdin') is captured correctly""" + result = self._as_list(parse_logstash_config(MINIMAL_INPUT)) + statements = result[0]['statements'] + plugin_names = [str(s.get('name', '')) for s in statements] + assert 'stdin' in plugin_names + + def test_invalid_config_raises_value_error(self): + """Invalid config raises a ValueError""" + bad_config = "this is not valid logstash config {{{" + with pytest.raises(ValueError): + parse_logstash_config(bad_config) + + def test_error_message_includes_line_info(self): + """ValueError message contains line number information""" + bad_config = "input {\n ??? invalid\n}\n" + with pytest.raises(ValueError, match=r'line'): + parse_logstash_config(bad_config) + + def test_plugin_with_settings(self): + """Settings on a plugin are parsed into the settings dict""" + config = 'input {\n beats {\n port => 5044\n }\n}\n' + result = self._as_list(parse_logstash_config(config)) + statements = result[0]['statements'] + beats = next(s for s in statements if str(s.get('name', '')) == 'beats') + assert beats['settings']['port'] == 5044 + + def test_plugin_with_string_setting(self): + """String settings are unquoted during parsing""" + config = 'input {\n file {\n path => "/var/log/syslog"\n }\n}\n' + result = self._as_list(parse_logstash_config(config)) + statements = result[0]['statements'] + file_plugin = next(s for s in statements if str(s.get('name', '')) == 'file') + assert file_plugin['settings']['path'] == '/var/log/syslog' + + def test_plugin_with_numeric_setting(self): + """Numeric settings are parsed correctly (not checked as list here)""" + config = 'input {\n file {\n sincedb_clean_after => 0\n start_position => "beginning"\n }\n}\n' + result = self._as_list(parse_logstash_config(config)) + assert len(result) == 1 + assert result[0]['type'] == 'input' + + def test_empty_plugin_block(self): + """A plugin with no settings produces an empty settings dict""" + result = self._as_list(parse_logstash_config(MINIMAL_INPUT)) + statements = result[0]['statements'] + stdin = next(s for s in statements if str(s.get('name', '')) == 'stdin') + assert stdin['settings'] == {} + + def test_strips_inline_comments_before_parsing(self): + """Inline comments that would break parsing are stripped first""" + config = 'input {\n beats {\n port => 5044 # inline comment\n }\n}\n' + # Should not raise — comments are stripped before parsing + result = self._as_list(parse_logstash_config(config)) + assert len(result) == 1 + + +# ───────────────────────────────────────────────────────────── +# logstash_config_to_components +# ───────────────────────────────────────────────────────────── + +class TestLogstashConfigToComponents: + """Unit tests for logstash_config_to_components""" + + def test_returns_json_string(self): + """Returns a JSON string""" + result = logstash_config_to_components(FULL_PIPELINE) + assert isinstance(result, str) + # Should be valid JSON + parsed = json.loads(result) + assert isinstance(parsed, dict) + + def test_output_has_three_sections(self): + """Output JSON has input, filter, output keys""" + result = json.loads(logstash_config_to_components(FULL_PIPELINE)) + assert 'input' in result + assert 'filter' in result + assert 'output' in result + + def test_all_sections_present_even_when_missing(self): + """All three sections are present even when only some exist in config""" + result = json.loads(logstash_config_to_components(MINIMAL_INPUT)) + assert 'input' in result + assert 'filter' in result + assert 'output' in result + assert result['filter'] == [] + assert result['output'] == [] + + def test_plugin_in_correct_section(self): + """Plugins end up in their correct section""" + result = json.loads(logstash_config_to_components(FULL_PIPELINE)) + input_plugins = [c['plugin'] for c in result['input']] + filter_plugins = [c['plugin'] for c in result['filter']] + output_plugins = [c['plugin'] for c in result['output']] + assert 'stdin' in input_plugins + assert 'mutate' in filter_plugins + assert 'stdout' in output_plugins + + def test_component_has_required_keys(self): + """Each component has id, type, plugin, and config keys""" + result = json.loads(logstash_config_to_components(FULL_PIPELINE)) + for section_components in result.values(): + for component in section_components: + assert 'id' in component + assert 'type' in component + assert 'plugin' in component + assert 'config' in component + + def test_component_type_matches_section(self): + """Each component's 'type' matches the section it belongs to""" + result = json.loads(logstash_config_to_components(FULL_PIPELINE)) + for section_name, section_components in result.items(): + for component in section_components: + assert component['type'] == section_name + + def test_component_id_includes_plugin_name(self): + """Component IDs include the plugin name""" + result = json.loads(logstash_config_to_components(MINIMAL_INPUT)) + stdin_component = result['input'][0] + assert 'stdin' in stdin_component['id'] + + def test_plugin_settings_in_config(self): + """Plugin settings appear in the component's config dict""" + config = 'input {\n beats {\n port => 5044\n }\n}\n' + result = json.loads(logstash_config_to_components(config)) + beats = result['input'][0] + assert beats['plugin'] == 'beats' + assert beats['config']['port'] == 5044 + + def test_invalid_config_raises_exception(self): + """Invalid config raises an Exception""" + with pytest.raises(Exception): + logstash_config_to_components("this is complete garbage {{{}}") + + def test_multiple_plugins_same_section(self): + """Multiple plugins in the same section are all captured""" + config = ( + 'input {\n' + ' stdin {}\n' + ' beats { port => 5044 }\n' + '}\n' + 'output { stdout {} }\n' + ) + result = json.loads(logstash_config_to_components(config)) + assert len(result['input']) == 2 + plugin_names = [c['plugin'] for c in result['input']] + assert 'stdin' in plugin_names + assert 'beats' in plugin_names + + def test_components_have_unique_ids(self): + """All component IDs are unique within the output""" + config = ( + 'input { stdin {} }\n' + 'filter { mutate {} grok {} }\n' + 'output { stdout {} }\n' + ) + result = json.loads(logstash_config_to_components(config)) + all_ids = [] + for section_components in result.values(): + all_ids.extend(c['id'] for c in section_components) + assert len(all_ids) == len(set(all_ids)), "Component IDs should be unique" + + def test_output_is_pretty_printed_json(self): + """Output JSON is indented (pretty-printed)""" + result = logstash_config_to_components(MINIMAL_INPUT) + assert '\n' in result + assert ' ' in result # 4-space indent + + +# ───────────────────────────────────────────────────────────── +# ComponentToPipeline +# ───────────────────────────────────────────────────────────── + +class TestComponentToPipeline: + """Unit tests for ComponentToPipeline helper methods""" + + def _make_parser(self, components=None): + if components is None: + components = {'input': [], 'filter': [], 'output': []} + return ComponentToPipeline(components) + + # _format_string_value tests + + def test_format_string_simple(self): + """Simple strings are quoted with double quotes""" + parser = self._make_parser() + result = parser._format_string_value('hello') + assert result == '"hello"' + + def test_format_string_non_string_passthrough(self): + """Non-string values are returned as-is""" + parser = self._make_parser() + assert parser._format_string_value(42) == 42 + assert parser._format_string_value(True) is True + assert parser._format_string_value(3.14) == 3.14 + + def test_format_string_with_double_quotes_uses_single(self): + """String containing double quotes is wrapped in single quotes""" + parser = self._make_parser() + result = parser._format_string_value('say "hello"') + assert result.startswith("'") + assert result.endswith("'") + + def test_format_string_with_single_quotes_uses_double(self): + """String containing single quotes is wrapped in double quotes""" + parser = self._make_parser() + result = parser._format_string_value("it's fine") + assert result.startswith('"') + assert result.endswith('"') + + def test_format_multiline_string_uses_single_quotes(self): + """Multiline string uses single quotes""" + parser = self._make_parser() + result = parser._format_string_value('line1\nline2') + assert result.startswith("'") + + # _generate_plugin_id tests + + def test_generate_plugin_id_format(self): + """Generated ID follows section_plugin_count format""" + parser = self._make_parser() + plugin_id = parser._generate_plugin_id('beats', 'input') + assert plugin_id == 'input_beats_1' + + def test_generate_plugin_id_increments(self): + """Each call increments the counter for the same plugin type""" + parser = self._make_parser() + id1 = parser._generate_plugin_id('beats', 'input') + id2 = parser._generate_plugin_id('beats', 'input') + assert id1 == 'input_beats_1' + assert id2 == 'input_beats_2' + + def test_generate_plugin_id_separate_counters_per_section(self): + """Different sections have independent counters""" + parser = self._make_parser() + input_id = parser._generate_plugin_id('stdout', 'input') + output_id = parser._generate_plugin_id('stdout', 'output') + assert input_id == 'input_stdout_1' + assert output_id == 'output_stdout_1' + + # components_to_logstash_config tests + + def test_empty_components_returns_empty_sections(self): + """Empty component lists produce an empty pipeline string""" + parser = self._make_parser({'input': [], 'filter': [], 'output': []}) + result = parser.components_to_logstash_config() + assert isinstance(result, str) + + def test_simple_stdin_plugin(self): + """A simple stdin plugin is rendered correctly""" + components = { + 'input': [ + {'id': 'input_stdin_0', 'type': 'input', 'plugin': 'stdin', 'config': {}} + ], + 'filter': [], + 'output': [] + } + parser = ComponentToPipeline(components) + result = parser.components_to_logstash_config() + assert 'input {' in result + assert 'stdin {' in result + + def test_plugin_with_string_setting(self): + """String settings are rendered with quotes""" + components = { + 'input': [ + {'id': 'input_file_0', 'type': 'input', 'plugin': 'file', + 'config': {'path': '/var/log/syslog'}} + ], + 'filter': [], + 'output': [] + } + parser = ComponentToPipeline(components) + result = parser.components_to_logstash_config() + assert 'path => "/var/log/syslog"' in result + + def test_plugin_with_numeric_setting(self): + """Numeric settings are rendered without quotes""" + components = { + 'input': [ + {'id': 'input_beats_0', 'type': 'input', 'plugin': 'beats', + 'config': {'port': 5044}} + ], + 'filter': [], + 'output': [] + } + parser = ComponentToPipeline(components) + result = parser.components_to_logstash_config() + assert 'port => 5044' in result + assert 'port => "5044"' not in result + + def test_all_three_sections_rendered(self): + """All three sections (input, filter, output) appear in the output""" + components = { + 'input': [{'id': 'i', 'type': 'input', 'plugin': 'stdin', 'config': {}}], + 'filter': [{'id': 'f', 'type': 'filter', 'plugin': 'mutate', 'config': {}}], + 'output': [{'id': 'o', 'type': 'output', 'plugin': 'stdout', 'config': {}}] + } + parser = ComponentToPipeline(components) + result = parser.components_to_logstash_config() + assert 'input {' in result + assert 'filter {' in result + assert 'output {' in result diff --git a/tests/Common/unit/test_logstash_utils.py b/tests/Common/unit/test_logstash_utils.py new file mode 100644 index 0000000..b3a86c7 --- /dev/null +++ b/tests/Common/unit/test_logstash_utils.py @@ -0,0 +1,163 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +import pytest +from unittest.mock import Mock, patch, MagicMock + +from Common.logstash_utils import get_logstash_pipeline + + +class TestGetLogstashPipeline: + """Tests for get_logstash_pipeline function""" + + @patch('Common.logstash_utils.get_elastic_connection') + def test_returns_pipeline_document_on_success(self, mock_get_connection): + """Test that the pipeline document is returned when found""" + pipeline_name = 'my_pipeline' + expected_doc = { + 'pipeline': 'input { stdin {} } output { stdout {} }', + 'last_modified': '2024-01-01T00:00:00Z', + 'username': 'elastic' + } + + mock_es = Mock() + mock_es.logstash.get_pipeline.return_value = {pipeline_name: expected_doc} + mock_get_connection.return_value = mock_es + + result = get_logstash_pipeline(es_id=1, pipeline_name=pipeline_name) + + assert result == expected_doc + + @patch('Common.logstash_utils.get_elastic_connection') + def test_calls_get_elastic_connection_with_correct_id(self, mock_get_connection): + """Test that get_elastic_connection is called with the provided es_id""" + mock_es = Mock() + mock_es.logstash.get_pipeline.return_value = {'pipe': {'pipeline': 'input {}'}} + mock_get_connection.return_value = mock_es + + get_logstash_pipeline(es_id=42, pipeline_name='pipe') + + mock_get_connection.assert_called_once_with(42) + + @patch('Common.logstash_utils.get_elastic_connection') + def test_calls_get_pipeline_with_correct_name(self, mock_get_connection): + """Test that logstash.get_pipeline is called with the correct pipeline name""" + pipeline_name = 'target_pipeline' + mock_es = Mock() + mock_es.logstash.get_pipeline.return_value = {pipeline_name: {'pipeline': ''}} + mock_get_connection.return_value = mock_es + + get_logstash_pipeline(es_id=1, pipeline_name=pipeline_name) + + mock_es.logstash.get_pipeline.assert_called_once_with(id=pipeline_name) + + @patch('Common.logstash_utils.get_elastic_connection') + def test_returns_none_on_key_error(self, mock_get_connection): + """Test that None is returned when pipeline name not found in response (KeyError)""" + mock_es = Mock() + # Pipeline name not in response dict → KeyError + mock_es.logstash.get_pipeline.return_value = {'other_pipeline': {}} + mock_get_connection.return_value = mock_es + + result = get_logstash_pipeline(es_id=1, pipeline_name='nonexistent_pipeline') + + assert result is None + + @patch('Common.logstash_utils.get_elastic_connection') + def test_returns_none_on_connection_error(self, mock_get_connection): + """Test that None is returned when an exception occurs during connection""" + mock_get_connection.side_effect = Exception("Connection refused") + + result = get_logstash_pipeline(es_id=1, pipeline_name='my_pipeline') + + assert result is None + + @patch('Common.logstash_utils.get_elastic_connection') + def test_returns_none_on_api_error(self, mock_get_connection): + """Test that None is returned when the ES API call raises an exception""" + mock_es = Mock() + mock_es.logstash.get_pipeline.side_effect = Exception("API Error: 500") + mock_get_connection.return_value = mock_es + + result = get_logstash_pipeline(es_id=1, pipeline_name='my_pipeline') + + assert result is None + + @patch('Common.logstash_utils.get_elastic_connection') + def test_logs_error_on_key_error(self, mock_get_connection, caplog): + """Test that an error is logged when pipeline is not found (KeyError)""" + pipeline_name = 'missing_pipeline' + mock_es = Mock() + mock_es.logstash.get_pipeline.return_value = {} # Missing key + mock_get_connection.return_value = mock_es + + get_logstash_pipeline(es_id=5, pipeline_name=pipeline_name) + + assert pipeline_name in caplog.text + + @patch('Common.logstash_utils.get_elastic_connection') + def test_logs_error_on_generic_exception(self, mock_get_connection, caplog): + """Test that an error is logged when a generic exception occurs""" + pipeline_name = 'error_pipeline' + mock_es = Mock() + mock_es.logstash.get_pipeline.side_effect = ConnectionError("Timeout") + mock_get_connection.return_value = mock_es + + get_logstash_pipeline(es_id=3, pipeline_name=pipeline_name) + + assert pipeline_name in caplog.text + + @patch('Common.logstash_utils.get_elastic_connection') + def test_returns_full_pipeline_document_structure(self, mock_get_connection): + """Test that the full pipeline document structure is preserved""" + pipeline_name = 'complex_pipeline' + expected_doc = { + 'pipeline': 'input { beats { port => 5044 } } output { elasticsearch {} }', + 'last_modified': '2024-03-01T12:00:00Z', + 'username': 'kibana_system', + 'metadata': { + 'type': 'logstash_pipeline', + 'version': 1 + } + } + + mock_es = Mock() + mock_es.logstash.get_pipeline.return_value = {pipeline_name: expected_doc} + mock_get_connection.return_value = mock_es + + result = get_logstash_pipeline(es_id=1, pipeline_name=pipeline_name) + + assert result == expected_doc + assert result['pipeline'] == expected_doc['pipeline'] + assert result['username'] == 'kibana_system' + + @patch('Common.logstash_utils.get_elastic_connection') + def test_handles_multiple_pipelines_in_response(self, mock_get_connection): + """Test correct pipeline is returned when response contains multiple pipelines""" + target_name = 'target' + target_doc = {'pipeline': 'input { stdin {} }'} + other_doc = {'pipeline': 'input { beats {} }'} + + mock_es = Mock() + mock_es.logstash.get_pipeline.return_value = { + target_name: target_doc, + 'other': other_doc + } + mock_get_connection.return_value = mock_es + + result = get_logstash_pipeline(es_id=1, pipeline_name=target_name) + + assert result == target_doc + assert result != other_doc + + @patch('Common.logstash_utils.get_elastic_connection') + def test_handles_empty_pipeline_response(self, mock_get_connection): + """Test returns None when response is completely empty dict""" + mock_es = Mock() + mock_es.logstash.get_pipeline.return_value = {} + mock_get_connection.return_value = mock_es + + result = get_logstash_pipeline(es_id=1, pipeline_name='any_pipeline') + + assert result is None diff --git a/tests/Common/unit/test_middleware.py b/tests/Common/unit/test_middleware.py new file mode 100644 index 0000000..81e5fd0 --- /dev/null +++ b/tests/Common/unit/test_middleware.py @@ -0,0 +1,242 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +import pytest +from django.test import RequestFactory +from django.http import HttpResponse + +from Common.middleware import SecurityHeadersMiddleware + + +@pytest.fixture +def request_factory(): + """Django RequestFactory for creating mock requests""" + return RequestFactory() + + +@pytest.fixture +def mock_request(request_factory): + """Create a basic GET mock request""" + return request_factory.get('/') + + +@pytest.fixture +def middleware(): + """Create middleware instance with a simple get_response callable""" + def get_response(request): + return HttpResponse("OK", status=200) + + return SecurityHeadersMiddleware(get_response) + + +@pytest.fixture +def middleware_with_custom_response(): + """Factory fixture to create middleware with a custom response""" + def factory(response): + return SecurityHeadersMiddleware(lambda r: response) + return factory + + +class TestSecurityHeadersMiddlewareInit: + """Tests for SecurityHeadersMiddleware initialization""" + + def test_middleware_stores_get_response(self): + """Test that middleware stores the get_response callable""" + def get_response(request): + return HttpResponse("OK") + + mw = SecurityHeadersMiddleware(get_response) + assert mw.get_response is get_response + + def test_middleware_callable(self, middleware, mock_request): + """Test that middleware is callable and returns a response""" + response = middleware(mock_request) + assert response is not None + assert response.status_code == 200 + + +class TestContentSecurityPolicy: + """Tests for the Content-Security-Policy header""" + + def test_csp_header_is_set(self, middleware, mock_request): + """Test that CSP header is present in the response""" + response = middleware(mock_request) + assert 'Content-Security-Policy' in response + + def test_csp_default_src_self(self, middleware, mock_request): + """Test that default-src is restricted to self""" + response = middleware(mock_request) + csp = response['Content-Security-Policy'] + assert "default-src 'self'" in csp + + def test_csp_script_src_includes_self(self, middleware, mock_request): + """Test that script-src includes self""" + response = middleware(mock_request) + csp = response['Content-Security-Policy'] + assert "script-src 'self'" in csp + + def test_csp_script_src_allows_unsafe_inline(self, middleware, mock_request): + """Test that script-src allows unsafe-inline (required for htmx)""" + response = middleware(mock_request) + csp = response['Content-Security-Policy'] + # Need unsafe-inline for htmx and dynamic JS + assert "'unsafe-inline'" in csp + + def test_csp_style_src_allows_unsafe_inline(self, middleware, mock_request): + """Test that style-src allows unsafe-inline (required for Tailwind)""" + response = middleware(mock_request) + csp = response['Content-Security-Policy'] + assert "style-src 'self' 'unsafe-inline'" in csp + + def test_csp_img_src_allows_data_and_https(self, middleware, mock_request): + """Test that img-src allows data URIs and HTTPS sources""" + response = middleware(mock_request) + csp = response['Content-Security-Policy'] + assert "img-src 'self' data: https:" in csp + + def test_csp_font_src_allows_data(self, middleware, mock_request): + """Test that font-src allows data URIs""" + response = middleware(mock_request) + csp = response['Content-Security-Policy'] + assert "font-src 'self' data:" in csp + + def test_csp_connect_src_self_only(self, middleware, mock_request): + """Test that connect-src is restricted to self""" + response = middleware(mock_request) + csp = response['Content-Security-Policy'] + assert "connect-src 'self'" in csp + + def test_csp_frame_src_includes_elastic(self, middleware, mock_request): + """Test that frame-src includes elastic.co documentation domains""" + response = middleware(mock_request) + csp = response['Content-Security-Policy'] + assert "https://www.elastic.co" in csp + assert "https://elastic.co" in csp + + def test_csp_frame_src_includes_github(self, middleware, mock_request): + """Test that frame-src includes github.com for plugin docs""" + response = middleware(mock_request) + csp = response['Content-Security-Policy'] + assert "https://github.com" in csp + + def test_csp_frame_src_includes_rubydoc(self, middleware, mock_request): + """Test that frame-src includes rubydoc.info for plugin docs""" + response = middleware(mock_request) + csp = response['Content-Security-Policy'] + assert "https://rubydoc.info" in csp + + def test_csp_frame_ancestors_self_only(self, middleware, mock_request): + """Test that frame-ancestors restricts framing to same origin (anti-clickjacking)""" + response = middleware(mock_request) + csp = response['Content-Security-Policy'] + assert "frame-ancestors 'self'" in csp + + def test_csp_directives_separated_by_semicolons(self, middleware, mock_request): + """Test that CSP directives are joined with semicolons""" + response = middleware(mock_request) + csp = response['Content-Security-Policy'] + assert '; ' in csp + + +class TestAdditionalSecurityHeaders: + """Tests for X-Content-Type-Options and Referrer-Policy headers""" + + def test_x_content_type_options_nosniff(self, middleware, mock_request): + """Test that X-Content-Type-Options is set to nosniff""" + response = middleware(mock_request) + assert response['X-Content-Type-Options'] == 'nosniff' + + def test_referrer_policy_set(self, middleware, mock_request): + """Test that Referrer-Policy header is set""" + response = middleware(mock_request) + assert 'Referrer-Policy' in response + + def test_referrer_policy_value(self, middleware, mock_request): + """Test Referrer-Policy has the correct value""" + response = middleware(mock_request) + assert response['Referrer-Policy'] == 'no-referrer-when-downgrade' + + +class TestMiddlewarePassthrough: + """Tests ensuring middleware doesn't break normal response behavior""" + + def test_middleware_preserves_response_status(self, request_factory): + """Test that middleware preserves the original response status code""" + def get_response(request): + return HttpResponse("Created", status=201) + + mw = SecurityHeadersMiddleware(get_response) + request = request_factory.get('/') + response = mw(request) + assert response.status_code == 201 + + def test_middleware_preserves_response_body(self, request_factory): + """Test that middleware preserves the original response body""" + def get_response(request): + return HttpResponse("Hello World") + + mw = SecurityHeadersMiddleware(get_response) + request = request_factory.get('/') + response = mw(request) + assert b'Hello World' in response.content + + def test_middleware_does_not_remove_existing_headers(self, request_factory): + """Test that middleware does not remove headers already set by the view""" + def get_response(request): + resp = HttpResponse("OK") + resp['X-Custom-Header'] = 'custom-value' + return resp + + mw = SecurityHeadersMiddleware(get_response) + request = request_factory.get('/') + response = mw(request) + assert response['X-Custom-Header'] == 'custom-value' + + def test_middleware_works_with_post_request(self, request_factory): + """Test that middleware applies headers to POST requests too""" + def get_response(request): + return HttpResponse("Posted", status=200) + + mw = SecurityHeadersMiddleware(get_response) + request = request_factory.post('/submit/') + response = mw(request) + assert 'Content-Security-Policy' in response + assert response['X-Content-Type-Options'] == 'nosniff' + + def test_middleware_works_with_json_response(self, request_factory): + """Test that middleware works with JSON responses""" + from django.http import JsonResponse + + def get_response(request): + return JsonResponse({'key': 'value'}) + + mw = SecurityHeadersMiddleware(get_response) + request = request_factory.get('/api/data/') + response = mw(request) + assert 'Content-Security-Policy' in response + assert response['X-Content-Type-Options'] == 'nosniff' + + def test_middleware_applies_to_error_responses(self, request_factory): + """Test that security headers are added even to error responses""" + def get_response(request): + return HttpResponse("Not Found", status=404) + + mw = SecurityHeadersMiddleware(get_response) + request = request_factory.get('/missing/') + response = mw(request) + assert response.status_code == 404 + assert 'Content-Security-Policy' in response + + def test_headers_applied_on_every_request(self, request_factory): + """Test that headers are applied on every call (not just first)""" + def get_response(request): + return HttpResponse("OK") + + mw = SecurityHeadersMiddleware(get_response) + + for _ in range(3): + request = request_factory.get('/') + response = mw(request) + assert 'Content-Security-Policy' in response + assert response['X-Content-Type-Options'] == 'nosniff' diff --git a/tests/Common/unit/test_pipeline_to_components.py b/tests/Common/unit/test_pipeline_to_components.py new file mode 100644 index 0000000..7b61c58 --- /dev/null +++ b/tests/Common/unit/test_pipeline_to_components.py @@ -0,0 +1,51 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +from Common.logstash_config_parse import logstash_config_to_components +import pytest +import json +import os + +# Load test cases from external files +def load_test_cases(): + """Load test cases from conversion_data directory.""" + base_dir = os.path.dirname(os.path.abspath(__file__)) + pipelines_dir = os.path.join(base_dir, "conversion_data", "pipelines") + components_dir = os.path.join(base_dir, "conversion_data", "components") + + test_cases = [] + + # Get all .conf files + for filename in sorted(os.listdir(pipelines_dir)): + if filename.endswith('.conf'): + name = filename[:-5] # Remove .conf extension + + # Load pipeline config + pipeline_file = os.path.join(pipelines_dir, filename) + with open(pipeline_file, 'r', encoding='utf-8') as f: + pipeline = f.read() + + # Load components JSON + components_file = os.path.join(components_dir, f"{name}.json") + with open(components_file, 'r', encoding='utf-8') as f: + components_json = json.load(f) + # Convert back to JSON string for comparison + components = json.dumps(components_json, indent=4) + + test_cases.append((name, pipeline, components)) + + return test_cases + +test_cases = load_test_cases() + +@pytest.mark.parametrize( + "name, pipeline, components", + test_cases, + ids=[case[0] for case in test_cases] +) +def test_pipeline_to_components(name, pipeline, components): + assert logstash_config_to_components(pipeline) == components + + + diff --git a/tests/Common/unit/test_product_ca.py b/tests/Common/unit/test_product_ca.py new file mode 100644 index 0000000..933ff36 --- /dev/null +++ b/tests/Common/unit/test_product_ca.py @@ -0,0 +1,334 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +"""Tests for product CA generation and enrollment token payload.""" + +import hashlib +import ipaddress +import socket + +import pytest +from cryptography import x509 +from cryptography.hazmat.primitives import serialization +from django.test import override_settings + + +@pytest.mark.django_db +def test_ensure_product_ca_generates_and_fingerprints(tmp_path, settings): + from Common import product_ca + + # Isolate CA storage + product_ca._cached_cert_pem = None + product_ca._cached_fingerprint = None + settings.BASE_DIR = tmp_path + settings.DATA_DIR = tmp_path / "data" + settings.DATA_DIR.mkdir(exist_ok=True) + + pem1, fp1 = product_ca.ensure_product_ca() + assert b"BEGIN CERTIFICATE" in pem1 + assert len(fp1) == 64 + cert = x509.load_pem_x509_certificate(pem1) + der = cert.public_bytes(serialization.Encoding.DER) + assert hashlib.sha256(der).hexdigest() == fp1 + + # Second call is cached / reloads same files + product_ca._cached_cert_pem = None + product_ca._cached_fingerprint = None + pem2, fp2 = product_ca.ensure_product_ca() + assert fp1 == fp2 + assert pem1 == pem2 + + +@pytest.mark.django_db +def test_build_enrollment_token_payload_includes_fingerprint(tmp_path, settings): + from Common import product_ca + + product_ca._cached_cert_pem = None + product_ca._cached_fingerprint = None + settings.BASE_DIR = tmp_path + settings.DATA_DIR = tmp_path / "data" + settings.DATA_DIR.mkdir(exist_ok=True) + settings.LOGSTASHUI_CONFIG = { + "agent": {"include_ca_fingerprint": True}, + } + + payload = product_ca.build_enrollment_token_payload("secret-token") + assert payload["enrollment_token"] == "secret-token" + assert payload["token_version"] == 2 + assert "fingerprint" in payload + assert len(payload["fingerprint"]) == 64 + assert "ui_url" not in payload + + +@pytest.mark.django_db +def test_build_enrollment_token_payload_omits_fingerprint(tmp_path, settings): + from Common import product_ca + + product_ca._cached_cert_pem = None + product_ca._cached_fingerprint = None + settings.BASE_DIR = tmp_path + settings.DATA_DIR = tmp_path / "data" + settings.DATA_DIR.mkdir(exist_ok=True) + settings.LOGSTASHUI_CONFIG = { + "agent": {"include_ca_fingerprint": False}, + } + + payload = product_ca.build_enrollment_token_payload("secret-token") + assert "fingerprint" not in payload + + +def test_product_ca_endpoint(client, tmp_path, settings): + from Common import product_ca + + product_ca._cached_cert_pem = None + product_ca._cached_fingerprint = None + settings.BASE_DIR = tmp_path + settings.DATA_DIR = tmp_path / "data" + settings.DATA_DIR.mkdir(exist_ok=True) + + resp = client.get("/.well-known/logstashui/ca.crt") + assert resp.status_code == 200 + assert b"BEGIN CERTIFICATE" in resp.content + + +@pytest.mark.django_db +def test_default_ui_server_cert_includes_compose_sans(tmp_path, settings, monkeypatch): + """Product default leaf must cover localhost and logstashui service name.""" + from Common import product_ca + + product_ca._cached_cert_pem = None + product_ca._cached_fingerprint = None + settings.BASE_DIR = tmp_path + settings.DATA_DIR = tmp_path / "data" + settings.DATA_DIR.mkdir(exist_ok=True) + monkeypatch.delenv("LOGSTASHUI_TLS_SANS", raising=False) + monkeypatch.delenv("LOGSTASHUI_HOST_HOSTNAME", raising=False) + monkeypatch.delenv("LOGSTASHUI_HOST_IPS", raising=False) + + cert_path, key_path = product_ca.ensure_default_ui_server_cert() + assert cert_path.is_file() + assert key_path.is_file() + leaf = x509.load_pem_x509_certificate(cert_path.read_bytes()) + ext = leaf.extensions.get_extension_for_class(x509.SubjectAlternativeName) + dns = {n.value for n in ext.value if isinstance(n, x509.DNSName)} + assert "localhost" in dns + assert "logstashui" in dns + assert product_ca.get_ui_server_mode() == "product" + + +@pytest.mark.django_db +def test_ui_cert_includes_host_ips_and_reissues_on_san_change(tmp_path, settings, monkeypatch): + from Common import product_ca + import ipaddress + + product_ca._cached_cert_pem = None + product_ca._cached_fingerprint = None + settings.BASE_DIR = tmp_path + settings.DATA_DIR = tmp_path / "data" + settings.DATA_DIR.mkdir(exist_ok=True) + settings.LOGSTASHUI_CONFIG = {} + + monkeypatch.setenv("LOGSTASHUI_HOST_HOSTNAME", "docker-host.example") + monkeypatch.setenv("LOGSTASHUI_HOST_IPS", "10.20.30.40,10.20.30.41") + monkeypatch.delenv("LOGSTASHUI_TLS_SANS", raising=False) + + product_ca.ensure_default_ui_server_cert() + leaf = x509.load_pem_x509_certificate(product_ca.ui_server_cert_path().read_bytes()) + ext = leaf.extensions.get_extension_for_class(x509.SubjectAlternativeName) + dns = {n.value for n in ext.value if isinstance(n, x509.DNSName)} + ips = {n.value for n in ext.value if isinstance(n, x509.IPAddress)} + assert "docker-host.example" in dns + assert ipaddress.IPv4Address("10.20.30.40") in ips + assert ipaddress.IPv4Address("10.20.30.41") in ips + fp1 = product_ca.fingerprint_sha256_der(leaf) + + # New host IP → must re-issue + monkeypatch.setenv("LOGSTASHUI_HOST_IPS", "10.20.30.40,10.20.30.41,10.20.30.99") + assert product_ca.product_ui_cert_needs_reissue() is True + product_ca.ensure_default_ui_server_cert() + leaf2 = x509.load_pem_x509_certificate(product_ca.ui_server_cert_path().read_bytes()) + ips2 = { + n.value + for n in leaf2.extensions.get_extension_for_class(x509.SubjectAlternativeName).value + if isinstance(n, x509.IPAddress) + } + assert ipaddress.IPv4Address("10.20.30.99") in ips2 + fp2 = product_ca.fingerprint_sha256_der(leaf2) + assert fp1 != fp2 + + +def test_reverse_lookup_fqdns_filters_short_and_ip(monkeypatch): + from Common import product_ca + + def _fake(ip): + if ip == "10.0.0.5": + return ("host.example.com.", ["alias.example.com"], ["10.0.0.5"]) + if ip == "10.0.0.6": + return ("shortname", [], ["10.0.0.6"]) + raise socket.herror("no PTR") + + monkeypatch.setattr(product_ca.socket, "gethostbyaddr", _fake) + assert product_ca._reverse_lookup_fqdns("10.0.0.5") == [ + "host.example.com", + "alias.example.com", + ] + assert product_ca._reverse_lookup_fqdns("10.0.0.6") == [] + assert product_ca._reverse_lookup_fqdns("10.0.0.7") == [] + + +def test_prefer_ptr_fqdns_over_short_hostnames(monkeypatch): + from Common import product_ca + + def _fake(ip): + if ip == "10.9.5.31": + return ("mac.untergeek.dev", [], [ip]) + raise socket.herror("no PTR") + + monkeypatch.setattr(product_ca.socket, "gethostbyaddr", _fake) + + dns = ["localhost", "logstashui", "Palpatine"] + ips = [ + ipaddress.IPv4Address("127.0.0.1"), + ipaddress.IPv4Address("10.9.5.31"), + ] + product_ca._prefer_ptr_fqdns_over_short_hostnames(dns, ips) + assert "mac.untergeek.dev" in dns + assert "localhost" in dns + assert "logstashui" in dns + assert "Palpatine" not in dns + + +def test_collect_desired_ui_sans_uses_ptr_over_bare_hostname(monkeypatch, settings): + from Common import product_ca + + settings.LOGSTASHUI_CONFIG = {} + monkeypatch.setenv("LOGSTASHUI_HOST_HOSTNAME", "Palpatine") + monkeypatch.setenv("LOGSTASHUI_HOST_IPS", "172.19.7.21") + monkeypatch.delenv("LOGSTASHUI_TLS_SANS", raising=False) + + def _fake(ip): + if ip == "172.19.7.21": + return ("palpatine.untergeek.net", [], [ip]) + raise socket.herror("no PTR") + + monkeypatch.setattr(product_ca.socket, "gethostbyaddr", _fake) + # Avoid noise from local interface discovery / getfqdn + monkeypatch.setattr(product_ca.socket, "gethostname", lambda: "Palpatine") + monkeypatch.setattr(product_ca.socket, "getfqdn", lambda: "Palpatine") + monkeypatch.setattr(product_ca.socket, "getaddrinfo", lambda *a, **k: []) + # Skip UDP outbound-IP trick (would discover real LAN IPs and PTR them) + monkeypatch.setattr( + product_ca.socket, + "socket", + lambda *a, **k: (_ for _ in ()).throw(OSError("skip")), + ) + + dns, ips = product_ca.collect_desired_ui_sans() + assert "palpatine.untergeek.net" in dns + assert "Palpatine" not in dns + assert "localhost" in dns + assert "logstashui" in dns + assert ipaddress.IPv4Address("172.19.7.21") in ips + + +@pytest.mark.django_db +def test_sign_agent_csr(tmp_path, settings): + from Common import product_ca + from cryptography.hazmat.primitives.asymmetric import rsa + from cryptography.hazmat.primitives import hashes, serialization + from cryptography.x509.oid import NameOID + from cryptography import x509 + from datetime import datetime, timedelta, timezone + + product_ca._cached_cert_pem = None + product_ca._cached_fingerprint = None + settings.BASE_DIR = tmp_path + settings.DATA_DIR = tmp_path / "data" + settings.DATA_DIR.mkdir(exist_ok=True) + + key = rsa.generate_private_key(public_exponent=65537, key_size=2048) + csr = ( + x509.CertificateSigningRequestBuilder() + .subject_name(x509.Name([x509.NameAttribute(NameOID.COMMON_NAME, "agent1")])) + .add_extension( + x509.SubjectAlternativeName([ + x509.DNSName("agent1"), + x509.DNSName("localhost"), + ]), + critical=False, + ) + .sign(key, hashes.SHA256()) + ) + signed = product_ca.sign_agent_csr(csr.public_bytes(serialization.Encoding.PEM)) + assert "BEGIN CERTIFICATE" in signed["certificate_pem"] + assert "BEGIN CERTIFICATE" in signed["ca_pem"] + leaf = x509.load_pem_x509_certificate(signed["certificate_pem"].encode()) + assert leaf.subject.get_attributes_for_oid(NameOID.COMMON_NAME)[0].value == "agent1" + + +@pytest.mark.django_db +def test_custom_ui_cert_and_revert(tmp_path, settings): + from Common import product_ca + from cryptography.hazmat.primitives.asymmetric import rsa + from cryptography import x509 + from cryptography.hazmat.primitives import hashes, serialization + from cryptography.x509.oid import NameOID + from datetime import datetime, timedelta, timezone + + product_ca._cached_cert_pem = None + product_ca._cached_fingerprint = None + settings.BASE_DIR = tmp_path + settings.DATA_DIR = tmp_path / "data" + settings.DATA_DIR.mkdir(exist_ok=True) + + # Self-signed standalone cert (simulates public/custom CA leaf) + key = rsa.generate_private_key(public_exponent=65537, key_size=2048) + subject = x509.Name([x509.NameAttribute(NameOID.COMMON_NAME, "custom.example")]) + now = datetime.now(timezone.utc) + cert = ( + x509.CertificateBuilder() + .subject_name(subject) + .issuer_name(subject) + .public_key(key.public_key()) + .serial_number(x509.random_serial_number()) + .not_valid_before(now - timedelta(minutes=1)) + .not_valid_after(now + timedelta(days=30)) + .add_extension( + x509.SubjectAlternativeName([x509.DNSName("custom.example")]), + critical=False, + ) + .sign(key, hashes.SHA256()) + ) + cert_pem = cert.public_bytes(serialization.Encoding.PEM) + key_pem = key.private_bytes( + encoding=serialization.Encoding.PEM, + format=serialization.PrivateFormat.TraditionalOpenSSL, + encryption_algorithm=serialization.NoEncryption(), + ) + + info = product_ca.save_custom_ui_certificate(cert_pem, key_pem) + assert info["mode"] == "custom" + assert info["subject_cn"] == "custom.example" + assert product_ca.get_ui_server_mode() == "custom" + + status = product_ca.revert_ui_certificate_to_product_default() + assert status["mode"] == "product" + assert product_ca.ui_server_cert_path().is_file() + + +@pytest.mark.django_db +def test_settings_saves_agent_ui_url(admin_client): + from Management.models import Settings + + # Ensure admin has profile role admin (signal creates admin by default) + resp = admin_client.post( + "/Management/Settings/", + {"experimental_mode": "on", "agent_ui_url": "https://10.0.0.5:8443"}, + ) + assert resp.status_code == 200 + data = resp.json() + assert data["success"] is True + s = Settings.get_settings() + assert s.agent_ui_url == "https://10.0.0.5:8443" + assert s.experimental_mode is True diff --git a/tests/Common/unit/test_validators.py b/tests/Common/unit/test_validators.py new file mode 100644 index 0000000..3c003a1 --- /dev/null +++ b/tests/Common/unit/test_validators.py @@ -0,0 +1,175 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +import pytest +from Common.validators import validate_pipeline_name + + +class TestValidatePipelineName: + """Test validate_pipeline_name function with parametrized tests""" + + @pytest.mark.parametrize("pipeline_name,expected_valid", [ + ("valid_pipeline", True), + ("ValidPipeline", True), + ("_underscore_start", True), + ("pipeline123", True), + ("pipeline_with_dash-123", True), + ("a", True), + ("_", True), + ("Pipeline_Name_123", True), + ("my-pipeline-name", True), + ("my_pipeline_name", True), + ("ABC123_test-pipeline", True), + ]) + def test_valid_pipeline_names(self, pipeline_name, expected_valid): + """Test valid pipeline names""" + is_valid, error_message = validate_pipeline_name(pipeline_name) + assert is_valid == expected_valid + assert error_message is None + + @pytest.mark.parametrize("pipeline_name,expected_error_fragment", [ + ("", "cannot be empty"), + ("123pipeline", "must begin with a letter or underscore"), + ("-pipeline", "must begin with a letter or underscore"), + ("pipeline name", "must begin with a letter or underscore"), + ("pipeline@name", "must begin with a letter or underscore"), + ("pipeline.name", "must begin with a letter or underscore"), + ("pipeline$name", "must begin with a letter or underscore"), + ("pipeline#name", "must begin with a letter or underscore"), + ("pipeline!name", "must begin with a letter or underscore"), + ("pipeline*name", "must begin with a letter or underscore"), + ("pipeline(name)", "must begin with a letter or underscore"), + ("pipeline[name]", "must begin with a letter or underscore"), + ("pipeline{name}", "must begin with a letter or underscore"), + ("pipeline/name", "must begin with a letter or underscore"), + ("pipeline\\name", "must begin with a letter or underscore"), + ("pipeline:name", "must begin with a letter or underscore"), + ("pipeline;name", "must begin with a letter or underscore"), + ("pipeline,name", "must begin with a letter or underscore"), + ("pipeline", "must begin with a letter or underscore"), + ("pipeline?name", "must begin with a letter or underscore"), + ("pipeline|name", "must begin with a letter or underscore"), + ("pipeline~name", "must begin with a letter or underscore"), + ("pipeline`name", "must begin with a letter or underscore"), + ("pipeline'name", "must begin with a letter or underscore"), + ('pipeline"name', "must begin with a letter or underscore"), + ]) + def test_invalid_pipeline_names(self, pipeline_name, expected_error_fragment): + """Test invalid pipeline names""" + is_valid, error_message = validate_pipeline_name(pipeline_name) + assert is_valid is False + assert error_message is not None + assert expected_error_fragment in error_message.lower() + + def test_empty_string(self): + """Test empty string returns specific error""" + is_valid, error_message = validate_pipeline_name("") + assert is_valid is False + assert error_message == "Pipeline name cannot be empty" + + def test_none_value(self): + """Test None value is treated as empty""" + is_valid, error_message = validate_pipeline_name(None) + assert is_valid is False + assert error_message == "Pipeline name cannot be empty" + + def test_starts_with_number(self): + """Test pipeline name starting with number is invalid""" + is_valid, error_message = validate_pipeline_name("1pipeline") + assert is_valid is False + assert "must begin with a letter or underscore" in error_message + assert "[1pipeline]" in error_message + + def test_starts_with_dash(self): + """Test pipeline name starting with dash is invalid""" + is_valid, error_message = validate_pipeline_name("-pipeline") + assert is_valid is False + assert "must begin with a letter or underscore" in error_message + + def test_contains_space(self): + """Test pipeline name with spaces is invalid""" + is_valid, error_message = validate_pipeline_name("my pipeline") + assert is_valid is False + assert "must begin with a letter or underscore" in error_message + + def test_contains_special_characters(self): + """Test pipeline name with special characters is invalid""" + is_valid, error_message = validate_pipeline_name("pipeline@test") + assert is_valid is False + assert "must begin with a letter or underscore" in error_message + + def test_valid_with_all_allowed_characters(self): + """Test pipeline name with all allowed character types""" + is_valid, error_message = validate_pipeline_name("aZ_09-") + assert is_valid is True + assert error_message is None + + def test_single_letter(self): + """Test single letter is valid""" + is_valid, error_message = validate_pipeline_name("a") + assert is_valid is True + assert error_message is None + + def test_single_underscore(self): + """Test single underscore is valid""" + is_valid, error_message = validate_pipeline_name("_") + assert is_valid is True + assert error_message is None + + def test_long_pipeline_name(self): + """Test very long pipeline name is valid if format is correct""" + long_name = "a" * 1000 + is_valid, error_message = validate_pipeline_name(long_name) + assert is_valid is True + assert error_message is None + + def test_error_message_includes_pipeline_name(self): + """Test that error message includes the invalid pipeline name""" + invalid_name = "123invalid" + is_valid, error_message = validate_pipeline_name(invalid_name) + assert is_valid is False + assert invalid_name in error_message + + def test_unicode_characters_invalid(self): + """Test that unicode characters are invalid""" + is_valid, error_message = validate_pipeline_name("pipeline_中文") + assert is_valid is False + assert "must begin with a letter or underscore" in error_message + + def test_emoji_invalid(self): + """Test that emoji characters are invalid""" + is_valid, error_message = validate_pipeline_name("pipeline_🔥") + assert is_valid is False + assert "must begin with a letter or underscore" in error_message + + @pytest.mark.parametrize("valid_start", ["a", "Z", "_"]) + def test_valid_starting_characters(self, valid_start): + """Test all valid starting characters""" + is_valid, error_message = validate_pipeline_name(f"{valid_start}pipeline") + assert is_valid is True + assert error_message is None + + def test_consecutive_dashes_and_underscores(self): + """Test pipeline name with consecutive dashes and underscores""" + is_valid, error_message = validate_pipeline_name("pipeline__--__name") + assert is_valid is True + assert error_message is None + + def test_ends_with_dash(self): + """Test pipeline name ending with dash is valid""" + is_valid, error_message = validate_pipeline_name("pipeline-") + assert is_valid is True + assert error_message is None + + def test_ends_with_underscore(self): + """Test pipeline name ending with underscore is valid""" + is_valid, error_message = validate_pipeline_name("pipeline_") + assert is_valid is True + assert error_message is None + + def test_ends_with_number(self): + """Test pipeline name ending with number is valid""" + is_valid, error_message = validate_pipeline_name("pipeline123") + assert is_valid is True + assert error_message is None diff --git a/tests/Database/integration/__init__.py b/tests/Database/integration/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/Database/integration/conftest.py b/tests/Database/integration/conftest.py new file mode 100644 index 0000000..1ae0172 --- /dev/null +++ b/tests/Database/integration/conftest.py @@ -0,0 +1,244 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +import os +import shutil +import subprocess +import sys +import uuid + +import pytest + + +# --------------------------------------------------------------------------- +# Docker availability — checked once at module import time +# --------------------------------------------------------------------------- + +def _check_docker() -> tuple[bool, str]: + if not shutil.which("docker"): + return False, "docker binary not found in PATH" + try: + r = subprocess.run( + ["docker", "info"], + capture_output=True, + timeout=10, + ) + if r.returncode != 0: + return False, ( + f"docker info returned {r.returncode}: " + f"{r.stderr.decode()[:200]}" + ) + return True, "" + except (subprocess.TimeoutExpired, OSError) as exc: + return False, str(exc) + + +_DOCKER_OK, _DOCKER_REASON = _check_docker() + + +@pytest.fixture(scope="session", autouse=True) +def skip_if_no_docker(): + """Skip every test in the integration suite when Docker is unavailable.""" + if not _DOCKER_OK: + pytest.skip(f"Docker not available: {_DOCKER_REASON}") + + +# --------------------------------------------------------------------------- +# Container fixtures (session scope — start once, reused across all tests) +# --------------------------------------------------------------------------- + +@pytest.fixture(scope="session") +def postgres_container(): + from testcontainers.postgres import PostgresContainer + + with PostgresContainer( + image="postgres:16", + username="logstashui", + password="logstashui", + dbname="logstashui_test", + ) as c: + yield c + + +@pytest.fixture(scope="session") +def mysql_container(): + from testcontainers.mysql import MySqlContainer + + c = MySqlContainer( + image="mysql:8.0", + root_password="logstashui", + dbname="logstashui_test", + ) + c.with_command( + "--character-set-server=utf8mb4 --collation-server=utf8mb4_bin" + ) + with c: + yield c + + +@pytest.fixture(scope="session") +def mariadb_container(): + from testcontainers.mysql import MySqlContainer + + c = MySqlContainer( + image="mariadb:11", + root_password="logstashui", + dbname="logstashui_test", + ) + c.with_command( + "--character-set-server=utf8mb4 --collation-server=utf8mb4_bin" + ) + with c: + yield c + + +# --------------------------------------------------------------------------- +# Env-dict helpers (module-level, not fixtures — importable by test files) +# --------------------------------------------------------------------------- + +def pg_env(container, *, dbname: str = "logstashui_test") -> dict[str, str]: + """Return LOGSTASHUI_DB_* env dict for a PostgreSQL container.""" + return { + "LOGSTASHUI_DB_ENGINE": "postgresql", + "LOGSTASHUI_DB_HOST": container.get_container_host_ip(), + "LOGSTASHUI_DB_PORT": str(container.get_exposed_port(5432)), + "LOGSTASHUI_DB_USER": "logstashui", + "LOGSTASHUI_DB_PASSWORD": "logstashui", + "LOGSTASHUI_DB_NAME": dbname, + } + + +def mysql_env(container, *, dbname: str = "logstashui_test") -> dict[str, str]: + """Return LOGSTASHUI_DB_* env dict for a MySQL/MariaDB container (root user).""" + return { + "LOGSTASHUI_DB_ENGINE": "mysql", + "LOGSTASHUI_DB_HOST": container.get_container_host_ip(), + "LOGSTASHUI_DB_PORT": str(container.get_exposed_port(3306)), + "LOGSTASHUI_DB_USER": "root", + "LOGSTASHUI_DB_PASSWORD": "logstashui", + "LOGSTASHUI_DB_NAME": dbname, + } + + +def new_dbname() -> str: + """Generate a unique database name for test isolation.""" + return f"logstashui_{uuid.uuid4().hex[:8]}" + + +# --------------------------------------------------------------------------- +# Fresh-database helpers (create/drop via native drivers) +# --------------------------------------------------------------------------- + +def create_pg_db(base_env: dict[str, str], dbname: str) -> None: + """Create *dbname* in the PostgreSQL container reachable via *base_env*.""" + import psycopg + + connstr = ( + f"host={base_env['LOGSTASHUI_DB_HOST']} " + f"port={base_env['LOGSTASHUI_DB_PORT']} " + f"user={base_env['LOGSTASHUI_DB_USER']} " + f"password={base_env['LOGSTASHUI_DB_PASSWORD']} " + f"dbname={base_env['LOGSTASHUI_DB_NAME']}" + ) + with psycopg.connect(connstr, autocommit=True) as conn: + conn.execute(f'CREATE DATABASE "{dbname}"') + + +def drop_pg_db(base_env: dict[str, str], dbname: str) -> None: + import psycopg + + connstr = ( + f"host={base_env['LOGSTASHUI_DB_HOST']} " + f"port={base_env['LOGSTASHUI_DB_PORT']} " + f"user={base_env['LOGSTASHUI_DB_USER']} " + f"password={base_env['LOGSTASHUI_DB_PASSWORD']} " + f"dbname={base_env['LOGSTASHUI_DB_NAME']}" + ) + with psycopg.connect(connstr, autocommit=True) as conn: + conn.execute(f'DROP DATABASE IF EXISTS "{dbname}"') + + +def create_mysql_db(base_env: dict[str, str], dbname: str) -> None: + """Create *dbname* with utf8mb4/utf8mb4_bin in the MySQL/MariaDB container.""" + import pymysql + + conn = pymysql.connect( + host=base_env["LOGSTASHUI_DB_HOST"], + port=int(base_env["LOGSTASHUI_DB_PORT"]), + user=base_env["LOGSTASHUI_DB_USER"], + password=base_env["LOGSTASHUI_DB_PASSWORD"], + autocommit=True, + ) + try: + with conn.cursor() as cur: + cur.execute( + f"CREATE DATABASE `{dbname}` " + f"CHARACTER SET utf8mb4 COLLATE utf8mb4_bin" + ) + finally: + conn.close() + + +def drop_mysql_db(base_env: dict[str, str], dbname: str) -> None: + import pymysql + + conn = pymysql.connect( + host=base_env["LOGSTASHUI_DB_HOST"], + port=int(base_env["LOGSTASHUI_DB_PORT"]), + user=base_env["LOGSTASHUI_DB_USER"], + password=base_env["LOGSTASHUI_DB_PASSWORD"], + autocommit=True, + ) + try: + with conn.cursor() as cur: + cur.execute(f"DROP DATABASE IF EXISTS `{dbname}`") + finally: + conn.close() + + +# --------------------------------------------------------------------------- +# Parametrized engine fixture (postgres + mysql) +# --------------------------------------------------------------------------- + +@pytest.fixture(params=["postgres", "mysql"]) +def engine_env(request, postgres_container, mysql_container): + """Yield (engine_name, env_dict) for each supported engine.""" + if request.param == "postgres": + yield "postgresql", pg_env(postgres_container) + else: + yield "mysql", mysql_env(mysql_container) + + +# --------------------------------------------------------------------------- +# Fresh per-test database fixture (for migration / round-trip tests) +# --------------------------------------------------------------------------- + +@pytest.fixture +def fresh_db_env(engine_env, tmp_path): + """ + Yield (engine_name, env_dict) with a unique throwaway database and + LOGSTASHUI_DATA_DIR set. The database is created before the test and + dropped afterwards. + """ + engine, base_env = engine_env + dbname = new_dbname() + + if engine == "postgresql": + create_pg_db(base_env, dbname) + full_env = { + **base_env, + "LOGSTASHUI_DB_NAME": dbname, + "LOGSTASHUI_DATA_DIR": str(tmp_path), + } + yield engine, full_env + drop_pg_db(base_env, dbname) + else: + create_mysql_db(base_env, dbname) + full_env = { + **base_env, + "LOGSTASHUI_DB_NAME": dbname, + "LOGSTASHUI_DATA_DIR": str(tmp_path), + } + yield engine, full_env + drop_mysql_db(base_env, dbname) diff --git a/tests/Database/integration/test_db_config.py b/tests/Database/integration/test_db_config.py new file mode 100644 index 0000000..0aed525 --- /dev/null +++ b/tests/Database/integration/test_db_config.py @@ -0,0 +1,166 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +""" +Integration tests for database configuration and server version checking. +All tests that open real connections use subprocesses so Django settings +are configured in an isolated interpreter with the container env vars. +""" + +import os +import subprocess +import sys + +import pytest + +from LogstashUI.database import build_databases + + +# --------------------------------------------------------------------------- +# Subprocess helper +# --------------------------------------------------------------------------- + +def _run_python(code: str, extra_env: dict[str, str]) -> str: + from LogstashUI import migrate_engine as me + + env = os.environ.copy() + env.update(extra_env) + env.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") + me._with_package_pythonpath(env) + proc = subprocess.run( + [sys.executable, "-c", code], + env=env, + check=False, + capture_output=True, + text=True, + ) + if proc.returncode != 0: + raise AssertionError( + f"python -c exited {proc.returncode}\n" + f"stdout:\n{proc.stdout}\nstderr:\n{proc.stderr}" + ) + return proc.stdout + + +# --------------------------------------------------------------------------- +# Inline scripts +# --------------------------------------------------------------------------- + +_CHECK_SERVER_VERSION = """ +import os +import django +os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") +django.setup() +from django.db import connection +from LogstashUI.database import check_server_version +connection.ensure_connection() +check_server_version(connection) +print("OK") +""" + +_ENSURE_CONNECTION = """ +import os +import django +os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") +django.setup() +from django.db import connection +connection.ensure_connection() +assert connection.connection is not None, "connection is None" +print("OK") +""" + + +# --------------------------------------------------------------------------- +# Tests — build_databases() dict structure (no Docker needed) +# --------------------------------------------------------------------------- + +def test_mysql_options_include_utf8mb4(monkeypatch, tmp_path): + """build_databases() for MySQL must include utf8mb4 charset and utf8mb4_bin collation.""" + for key in ( + "LOGSTASHUI_DB_ENGINE", "LOGSTASHUI_DB_HOST", "LOGSTASHUI_DB_PORT", + "LOGSTASHUI_DB_USER", "LOGSTASHUI_DB_PASSWORD", "LOGSTASHUI_DB_NAME", + "LOGSTASHUI_DB_CONN_MAX_AGE", + ): + monkeypatch.delenv(key, raising=False) + monkeypatch.setenv("LOGSTASHUI_DB_ENGINE", "mysql") + monkeypatch.setenv("LOGSTASHUI_DB_HOST", "127.0.0.1") + monkeypatch.setenv("LOGSTASHUI_DB_USER", "root") + monkeypatch.setattr("LogstashUI.database._import_or_raise", lambda *a, **k: _fake_pymysql()) + db = build_databases(tmp_path)["default"] + assert db["OPTIONS"]["charset"] == "utf8mb4" + assert "utf8mb4_bin" in db["OPTIONS"]["init_command"] + assert db["TEST"]["CHARSET"] == "utf8mb4" + assert db["TEST"]["COLLATION"] == "utf8mb4_bin" + + +def _fake_pymysql(): + from types import SimpleNamespace + fake = SimpleNamespace( + version_info=(1, 1, 1, "final", 0), + install_as_MySQLdb=lambda: None, + ) + return fake + + +def test_conn_max_age_applied(monkeypatch, tmp_path): + """LOGSTASHUI_DB_CONN_MAX_AGE overrides the default 60s for postgres.""" + for key in ( + "LOGSTASHUI_DB_ENGINE", "LOGSTASHUI_DB_HOST", "LOGSTASHUI_DB_PORT", + "LOGSTASHUI_DB_USER", "LOGSTASHUI_DB_PASSWORD", "LOGSTASHUI_DB_NAME", + "LOGSTASHUI_DB_CONN_MAX_AGE", + ): + monkeypatch.delenv(key, raising=False) + monkeypatch.setenv("LOGSTASHUI_DB_ENGINE", "postgresql") + monkeypatch.setenv("LOGSTASHUI_DB_HOST", "db.example") + monkeypatch.setenv("LOGSTASHUI_DB_USER", "lsui") + monkeypatch.setenv("LOGSTASHUI_DB_CONN_MAX_AGE", "120") + monkeypatch.setattr("LogstashUI.database._import_or_raise", lambda *a, **k: None) + db = build_databases(tmp_path)["default"] + assert db["CONN_MAX_AGE"] == 120 + + +def test_build_databases_returns_valid_dict(engine_env, tmp_path, monkeypatch): + """build_databases() produces a valid DATABASES dict for each engine.""" + engine, env = engine_env + for k, v in env.items(): + monkeypatch.setenv(k, v) + monkeypatch.setenv("LOGSTASHUI_DATA_DIR", str(tmp_path)) + # For MySQL, stub _import_or_raise so it runs without the spoof side-effect + if engine == "mysql": + monkeypatch.setattr( + "LogstashUI.database._import_or_raise", + lambda *a, **k: _fake_pymysql(), + ) + db = build_databases(tmp_path)["default"] + assert "ENGINE" in db + assert "HOST" in db + assert "PORT" in db + assert "NAME" in db + assert isinstance(db["PORT"], str) + + +# --------------------------------------------------------------------------- +# Tests — real container connections (subprocess) +# --------------------------------------------------------------------------- + +def test_real_connection_opens(engine_env, tmp_path): + """Django can open a connection to the container database.""" + engine, env = engine_env + full_env = {**env, "LOGSTASHUI_DATA_DIR": str(tmp_path)} + _run_python(_ENSURE_CONNECTION, full_env) + + +def test_check_server_version_passes_on_real_connection(engine_env, tmp_path): + """check_server_version() passes without error on a real container connection.""" + engine, env = engine_env + full_env = {**env, "LOGSTASHUI_DATA_DIR": str(tmp_path)} + _run_python(_CHECK_SERVER_VERSION, full_env) + + +def test_check_server_version_mariadb_branch(mariadb_container, tmp_path): + """check_server_version() MariaDB detection branch passes on a real MariaDB server.""" + from tests.integration.conftest import mysql_env + env = mysql_env(mariadb_container) + full_env = {**env, "LOGSTASHUI_DATA_DIR": str(tmp_path)} + _run_python(_CHECK_SERVER_VERSION, full_env) diff --git a/tests/Database/integration/test_migrate_engine.py b/tests/Database/integration/test_migrate_engine.py new file mode 100644 index 0000000..fab32f8 --- /dev/null +++ b/tests/Database/integration/test_migrate_engine.py @@ -0,0 +1,292 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +""" +Integration tests for cmd_migrate_engine (SQLite → PostgreSQL / MySQL / MariaDB). + +Each test uses a unique throwaway database in the session-scoped container +to prevent cross-test contamination. +""" + +import json +import os +import subprocess +import sys +from argparse import Namespace + +import pytest + +from LogstashUI import migrate_engine as me +from tests.integration.conftest import ( + create_mysql_db, + create_pg_db, + drop_mysql_db, + drop_pg_db, + mysql_env, + new_dbname, + pg_env, +) + + +# --------------------------------------------------------------------------- +# Subprocess helper +# --------------------------------------------------------------------------- + +def _run_python(code: str, extra_env: dict[str, str]) -> str: + env = os.environ.copy() + env.update(extra_env) + env.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") + me._with_package_pythonpath(env) + proc = subprocess.run( + [sys.executable, "-c", code], + env=env, + check=False, + capture_output=True, + text=True, + ) + if proc.returncode != 0: + raise AssertionError( + f"python -c exited {proc.returncode}\n" + f"stdout:\n{proc.stdout}\nstderr:\n{proc.stderr}" + ) + return proc.stdout + + +# --------------------------------------------------------------------------- +# Inline scripts +# --------------------------------------------------------------------------- + +_SEED = """ +import os +import django +os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") +django.setup() +from django.contrib.auth import get_user_model +from PipelineManager.models import Connection, Policy +User = get_user_model() +User.objects.create_user(username="migrate-user", password="migrate-pass") +policy = Policy.objects.create( + name="Migrate Policy", + logstash_yml="http.host: 0.0.0.0", + jvm_options="-Xms1g", + log4j2_properties="status = error", +) +Connection.objects.create( + name="Migrate Conn", + connection_type=Connection.ConnectionType.AGENT, + host="127.0.0.1", + policy=policy, + status_blob={"health": "green", "n": 1}, +) +""" + +_COUNT = """ +import json +import os +import django +os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") +django.setup() +from django.contrib.auth import get_user_model +from PipelineManager.models import Connection, Policy +User = get_user_model() +conn = Connection.objects.filter(name="Migrate Conn").first() +print(json.dumps({ + "users": User.objects.count(), + "policies": Policy.objects.count(), + "migrate_user": User.objects.filter(username="migrate-user").count(), + "migrate_policy": Policy.objects.filter(name="Migrate Policy").count(), + "status_blob": conn.status_blob if conn else None, +})) +""" + +_POST_MIGRATE_INSERT = """ +import json +import os +import django +os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") +django.setup() +from django.contrib.auth import get_user_model +User = get_user_model() +u = User.objects.create_user(username="post-migrate-user", password="x") +assert u.pk is not None +print(json.dumps({"pk": u.pk})) +""" + +_TARGET_KEYS = ( + "LOGSTASHUI_DATA_DIR", + "LOGSTASHUI_DB_ENGINE", + "LOGSTASHUI_DB_HOST", + "LOGSTASHUI_DB_PORT", + "LOGSTASHUI_DB_USER", + "LOGSTASHUI_DB_PASSWORD", + "LOGSTASHUI_DB_NAME", +) + + +# --------------------------------------------------------------------------- +# Core migration helper +# --------------------------------------------------------------------------- + +def _run_to(tmp_path, target_env: dict[str, str]) -> dict: + """ + Seed a fresh SQLite database, run cmd_migrate_engine to the target, + then assert data counts. Returns the parsed count dict. + """ + data_dir = tmp_path + sqlite_path = data_dir / "db.sqlite3" + sqlite_env = { + "LOGSTASHUI_DATA_DIR": str(data_dir), + "LOGSTASHUI_DB_ENGINE": "sqlite", + "LOGSTASHUI_DB_NAME": str(sqlite_path), + } + me.run_manage(["migrate", "--noinput", "--verbosity", "0"], sqlite_env) + _run_python(_SEED, sqlite_env) + + full_target = {**target_env, "LOGSTASHUI_DATA_DIR": str(data_dir)} + previous = {key: os.environ.get(key) for key in _TARGET_KEYS} + try: + os.environ.update(full_target) + ns = Namespace( + to=full_target["LOGSTASHUI_DB_ENGINE"], + i_have_a_backup=True, + pid=None, + write_env=None, + ) + try: + rc = me.cmd_migrate_engine(ns) + except SystemExit as exc: + raise AssertionError( + f"cmd_migrate_engine SystemExit {exc.code}" + ) from exc + assert rc == 0 + raw = _run_python(_COUNT, full_target) + finally: + for key, value in previous.items(): + if value is None: + os.environ.pop(key, None) + else: + os.environ[key] = value + + counts = json.loads(raw.strip().splitlines()[-1]) + assert counts["users"] >= 1 + assert counts["policies"] >= 1 + assert counts["migrate_user"] == 1 + assert counts["migrate_policy"] == 1 + assert counts["status_blob"] == {"health": "green", "n": 1} + return counts + + +# --------------------------------------------------------------------------- +# Tests +# --------------------------------------------------------------------------- + +def test_migrate_engine_to_postgres(postgres_container, tmp_path): + dbname = new_dbname() + base = pg_env(postgres_container) + create_pg_db(base, dbname) + try: + target = pg_env(postgres_container, dbname=dbname) + _run_to(tmp_path, target) + finally: + drop_pg_db(base, dbname) + + +def test_migrate_engine_to_mysql(mysql_container, tmp_path): + dbname = new_dbname() + base = mysql_env(mysql_container) + create_mysql_db(base, dbname) + try: + target = mysql_env(mysql_container, dbname=dbname) + _run_to(tmp_path, target) + finally: + drop_mysql_db(base, dbname) + + +def test_migrate_engine_to_mariadb(mariadb_container, tmp_path): + dbname = new_dbname() + base = mysql_env(mariadb_container) + create_mysql_db(base, dbname) + try: + target = mysql_env(mariadb_container, dbname=dbname) + _run_to(tmp_path, target) + finally: + drop_mysql_db(base, dbname) + + +def test_sequence_reset_postgres(postgres_container, tmp_path): + """After SQLite→Postgres migration, inserting a new User does not fail on sequence.""" + dbname = new_dbname() + base = pg_env(postgres_container) + create_pg_db(base, dbname) + try: + target = pg_env(postgres_container, dbname=dbname) + _run_to(tmp_path, target) + full_env = {**target, "LOGSTASHUI_DATA_DIR": str(tmp_path)} + out = _run_python(_POST_MIGRATE_INSERT, full_env) + result = json.loads(out.strip()) + assert isinstance(result["pk"], int) and result["pk"] > 0 + finally: + drop_pg_db(base, dbname) + + +def test_migrate_engine_write_env(postgres_container, tmp_path): + """--write-env produces a file with engine/host keys but no PASSWORD.""" + dbname = new_dbname() + base = pg_env(postgres_container) + create_pg_db(base, dbname) + env_file = tmp_path / "logstashui.env" + try: + target = pg_env(postgres_container, dbname=dbname) + full_target = {**target, "LOGSTASHUI_DATA_DIR": str(tmp_path)} + sqlite_path = tmp_path / "db.sqlite3" + sqlite_env = { + "LOGSTASHUI_DATA_DIR": str(tmp_path), + "LOGSTASHUI_DB_ENGINE": "sqlite", + "LOGSTASHUI_DB_NAME": str(sqlite_path), + } + me.run_manage(["migrate", "--noinput", "--verbosity", "0"], sqlite_env) + _run_python(_SEED, sqlite_env) + previous = {key: os.environ.get(key) for key in _TARGET_KEYS} + try: + os.environ.update(full_target) + ns = Namespace( + to="postgresql", + i_have_a_backup=True, + pid=None, + write_env=str(env_file), + ) + rc = me.cmd_migrate_engine(ns) + finally: + for key, value in previous.items(): + if value is None: + os.environ.pop(key, None) + else: + os.environ[key] = value + assert rc == 0 + text = env_file.read_text() + assert "LOGSTASHUI_DB_ENGINE=postgresql" in text + assert "LOGSTASHUI_DB_HOST=" in text + assert "PASSWORD" not in text + finally: + drop_pg_db(base, dbname) + + +def test_migrate_engine_idempotent_env(postgres_container, tmp_path): + """Running write_env twice produces no duplicate keys in the output file.""" + dbname = new_dbname() + base = pg_env(postgres_container) + create_pg_db(base, dbname) + env_file = tmp_path / "logstashui.env" + try: + target = pg_env(postgres_container, dbname=dbname) + full_target = {**target, "LOGSTASHUI_DATA_DIR": str(tmp_path)} + # Run migration once + _run_to(tmp_path, target) + # write_env twice, pointing at the already-migrated DB + me.write_env_file(env_file, "postgresql") + me.write_env_file(env_file, "postgresql") + text = env_file.read_text() + assert text.count("LOGSTASHUI_DB_ENGINE=postgresql") == 1 + finally: + drop_pg_db(base, dbname) diff --git a/tests/Database/integration/test_migrations.py b/tests/Database/integration/test_migrations.py new file mode 100644 index 0000000..b20b524 --- /dev/null +++ b/tests/Database/integration/test_migrations.py @@ -0,0 +1,101 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +""" +Integration tests for Django migrations against real database containers. +Each parametrized test gets a fresh isolated database (created and dropped +per-test via the fresh_db_env fixture). +""" + +import os +import subprocess +import sys + +import pytest + +from LogstashUI import migrate_engine as me + + +# --------------------------------------------------------------------------- +# Subprocess helper +# --------------------------------------------------------------------------- + +def _run_python(code: str, extra_env: dict[str, str]) -> str: + env = os.environ.copy() + env.update(extra_env) + env.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") + me._with_package_pythonpath(env) + proc = subprocess.run( + [sys.executable, "-c", code], + env=env, + check=False, + capture_output=True, + text=True, + ) + if proc.returncode != 0: + raise AssertionError( + f"python -c exited {proc.returncode}\n" + f"stdout:\n{proc.stdout}\nstderr:\n{proc.stderr}" + ) + return proc.stdout + + +# --------------------------------------------------------------------------- +# Inline scripts +# --------------------------------------------------------------------------- + +_NO_UNAPPLIED = """ +import os +import django +os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") +django.setup() +from django.db import connection +from django.db.migrations.executor import MigrationExecutor +from django.db.migrations.loader import MigrationLoader +loader = MigrationLoader(connection) +executor = MigrationExecutor(connection) +plan = executor.migration_plan(loader.graph.leaf_nodes()) +assert plan == [], f"Unapplied migrations: {[str(m) for m, _ in plan]}" +print("OK") +""" + + +# --------------------------------------------------------------------------- +# Tests — container migrations +# --------------------------------------------------------------------------- + +def test_migrate_runs_clean(fresh_db_env): + """migrate --noinput completes without error on a fresh container database.""" + engine, env = fresh_db_env + me.run_manage(["migrate", "--noinput", "--verbosity", "0"], env) + + +def test_migrate_is_idempotent(fresh_db_env): + """Running migrate twice is safe (no errors, no unexpected state).""" + engine, env = fresh_db_env + me.run_manage(["migrate", "--noinput", "--verbosity", "0"], env) + me.run_manage(["migrate", "--noinput", "--verbosity", "0"], env) + + +def test_no_unapplied_migrations(fresh_db_env): + """After migrate, MigrationExecutor reports an empty plan.""" + engine, env = fresh_db_env + me.run_manage(["migrate", "--noinput", "--verbosity", "0"], env) + _run_python(_NO_UNAPPLIED, env) + + +# --------------------------------------------------------------------------- +# SQLite baseline (no Docker needed — fast sanity check) +# --------------------------------------------------------------------------- + +def test_sqlite_migrate_baseline(tmp_path): + """migrate --noinput works against SQLite; ensures the test runner itself is healthy.""" + sqlite_path = tmp_path / "db.sqlite3" + env = { + "LOGSTASHUI_DATA_DIR": str(tmp_path), + "LOGSTASHUI_DB_ENGINE": "sqlite", + "LOGSTASHUI_DB_NAME": str(sqlite_path), + } + me.run_manage(["migrate", "--noinput", "--verbosity", "0"], env) + assert sqlite_path.exists() diff --git a/tests/Database/integration/test_orm.py b/tests/Database/integration/test_orm.py new file mode 100644 index 0000000..5aef6ab --- /dev/null +++ b/tests/Database/integration/test_orm.py @@ -0,0 +1,389 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +""" +ORM integration tests against real database containers. + +All Django interaction happens in subprocesses so each test gets a clean +interpreter with the container env applied to settings. Each test migrates +(idempotently) then runs its CRUD script which self-cleans at the end. +""" + +import json +import os +import subprocess +import sys + +import pytest + +from LogstashUI import migrate_engine as me + + +# --------------------------------------------------------------------------- +# Subprocess helper +# --------------------------------------------------------------------------- + +def _run_python(code: str, extra_env: dict[str, str]) -> str: + env = os.environ.copy() + env.update(extra_env) + env.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") + me._with_package_pythonpath(env) + proc = subprocess.run( + [sys.executable, "-c", code], + env=env, + check=False, + capture_output=True, + text=True, + ) + if proc.returncode != 0: + raise AssertionError( + f"python -c exited {proc.returncode}\n" + f"stdout:\n{proc.stdout}\nstderr:\n{proc.stderr}" + ) + return proc.stdout + + +def _migrate(env: dict[str, str]) -> None: + me.run_manage(["migrate", "--noinput", "--verbosity", "0"], env) + + +# --------------------------------------------------------------------------- +# Inline CRUD scripts (each cleans up its own data) +# --------------------------------------------------------------------------- + +_POLICY_CRUD = """ +import json, os, django +os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") +django.setup() +from PipelineManager.models import Policy +TAG = "crud-policy-test" +Policy.objects.filter(name=TAG).delete() +p = Policy.objects.create( + name=TAG, + logstash_yml="http.host: 0.0.0.0", + jvm_options="-Xms512m", + log4j2_properties="status = error", +) +pk = p.pk +assert Policy.objects.get(pk=pk).name == TAG +Policy.objects.filter(pk=pk).update(logstash_yml="http.host: 127.0.0.1") +assert Policy.objects.get(pk=pk).logstash_yml == "http.host: 127.0.0.1" +Policy.objects.filter(pk=pk).delete() +assert Policy.objects.filter(pk=pk).count() == 0 +print(json.dumps({"ok": True})) +""" + +_CONNECTION_CRUD = """ +import json, os, django +os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") +django.setup() +from PipelineManager.models import Connection, Policy +Policy.objects.filter(name="crud-conn-policy").delete() +policy = Policy.objects.create( + name="crud-conn-policy", + logstash_yml="http.host: 0.0.0.0", + jvm_options="-Xms512m", + log4j2_properties="status = error", +) +Connection.objects.filter(name="crud-conn-test").delete() +c = Connection.objects.create( + name="crud-conn-test", + connection_type=Connection.ConnectionType.AGENT, + host="127.0.0.1", + policy=policy, + status_blob={"health": "green"}, +) +pk = c.pk +assert Connection.objects.get(pk=pk).name == "crud-conn-test" +Connection.objects.filter(pk=pk).update(status_blob={"health": "yellow"}) +assert Connection.objects.get(pk=pk).status_blob["health"] == "yellow" +Connection.objects.filter(pk=pk).delete() +policy.delete() +assert Connection.objects.filter(pk=pk).count() == 0 +print(json.dumps({"ok": True})) +""" + +_PIPELINE_CRUD = """ +import json, os, django +os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") +django.setup() +from PipelineManager.models import Pipeline, Policy +Policy.objects.filter(name="crud-pipe-policy").delete() +policy = Policy.objects.create( + name="crud-pipe-policy", + logstash_yml="http.host: 0.0.0.0", + jvm_options="-Xms512m", + log4j2_properties="status = error", +) +Pipeline.objects.filter(policy=policy, name="test-pipeline").delete() +p = Pipeline.objects.create( + policy=policy, + name="test-pipeline", + lscl="input { stdin{} } output { stdout{} }", +) +pk = p.pk +assert Pipeline.objects.get(pk=pk).name == "test-pipeline" +Pipeline.objects.filter(pk=pk).delete() +policy.delete() +assert Pipeline.objects.filter(pk=pk).count() == 0 +print(json.dumps({"ok": True})) +""" + +_REVISION_JSON = """ +import json, os, django +os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") +django.setup() +from PipelineManager.models import Policy, Revision +Policy.objects.filter(name="rev-json-policy").delete() +policy = Policy.objects.create( + name="rev-json-policy", + logstash_yml="http.host: 0.0.0.0", + jvm_options="-Xms512m", + log4j2_properties="status = error", +) +snapshot = { + "pipelines": [{"name": "main", "lscl": "input{} output{}"}], + "meta": {"tags": ["a", "b"], "nested": {"k": 1}}, +} +r = Revision.objects.create( + policy=policy, revision_number=1, snapshot_json=snapshot, created_by="testrunner" +) +pk = r.pk +fetched = Revision.objects.get(pk=pk) +assert fetched.snapshot_json == snapshot, f"mismatch: {fetched.snapshot_json!r}" +Revision.objects.filter(pk=pk).delete() +policy.delete() +print(json.dumps({"ok": True})) +""" + +_STATUS_BLOB_JSON = """ +import json, os, django +os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") +django.setup() +from PipelineManager.models import Connection, Policy +Policy.objects.filter(name="blob-policy").delete() +policy = Policy.objects.create( + name="blob-policy", + logstash_yml="http.host: 0.0.0.0", + jvm_options="-Xms512m", + log4j2_properties="status = error", +) +blob = {"health": "green", "n": 42, "nested": {"k": [1, 2, 3]}} +Connection.objects.filter(name="blob-conn").delete() +c = Connection.objects.create( + name="blob-conn", + connection_type=Connection.ConnectionType.AGENT, + host="127.0.0.1", + policy=policy, + status_blob=blob, +) +pk = c.pk +fetched = Connection.objects.get(pk=pk) +assert fetched.status_blob == blob, f"mismatch: {fetched.status_blob!r}" +Connection.objects.filter(pk=pk).delete() +policy.delete() +print(json.dumps({"ok": True})) +""" + +_STATUS_BLOB_NULL = """ +import json, os, django +os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") +django.setup() +from PipelineManager.models import Connection, Policy +Policy.objects.filter(name="null-blob-policy").delete() +policy = Policy.objects.create( + name="null-blob-policy", + logstash_yml="http.host: 0.0.0.0", + jvm_options="-Xms512m", + log4j2_properties="status = error", +) +Connection.objects.filter(name="null-blob-conn").delete() +c = Connection.objects.create( + name="null-blob-conn", + connection_type=Connection.ConnectionType.AGENT, + host="127.0.0.1", + policy=policy, + status_blob=None, +) +pk = c.pk +assert Connection.objects.get(pk=pk).status_blob is None +Connection.objects.filter(pk=pk).delete() +policy.delete() +print(json.dumps({"ok": True})) +""" + +_SNMP_NETWORK_CRUD = """ +import json, os, django +os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") +django.setup() +from SNMP.models import Network +Network.objects.filter(name="crud-network-test").delete() +n = Network.objects.create(name="crud-network-test", network_range="10.0.0.0/24") +pk = n.pk +assert Network.objects.get(pk=pk).name == "crud-network-test" +Network.objects.filter(pk=pk).delete() +assert Network.objects.filter(pk=pk).count() == 0 +print(json.dumps({"ok": True})) +""" + +_UNIQUE_POLICY_NAME = """ +import json, os, django +os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") +django.setup() +from django.db import IntegrityError +from PipelineManager.models import Policy +Policy.objects.filter(name="dupe-policy").delete() +p1 = Policy.objects.create( + name="dupe-policy", + logstash_yml="http.host: 0.0.0.0", + jvm_options="-Xms512m", + log4j2_properties="status = error", +) +try: + Policy.objects.create( + name="dupe-policy", + logstash_yml="different", + jvm_options="-Xmx512m", + log4j2_properties="status = warn", + ) + raise AssertionError("Expected IntegrityError for duplicate Policy.name") +except IntegrityError: + pass +finally: + Policy.objects.filter(name="dupe-policy").delete() +print(json.dumps({"ok": True})) +""" + +_UNIQUE_PIPELINE_PER_POLICY = """ +import json, os, django +os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") +django.setup() +from django.db import IntegrityError +from PipelineManager.models import Pipeline, Policy +Policy.objects.filter(name__in=["uq-pol-a", "uq-pol-b"]).delete() +pol_a = Policy.objects.create( + name="uq-pol-a", + logstash_yml="http.host: 0.0.0.0", + jvm_options="-Xms512m", + log4j2_properties="status = error", +) +pol_b = Policy.objects.create( + name="uq-pol-b", + logstash_yml="http.host: 0.0.0.0", + jvm_options="-Xms512m", + log4j2_properties="status = error", +) +# Same name under different policies is allowed +Pipeline.objects.create(policy=pol_a, name="shared-pipe", lscl="input{} output{}") +Pipeline.objects.create(policy=pol_b, name="shared-pipe", lscl="input{} output{}") +# Same name under same policy must raise +try: + Pipeline.objects.create(policy=pol_a, name="shared-pipe", lscl="input{} output{}") + raise AssertionError("Expected IntegrityError for duplicate pipeline name per policy") +except IntegrityError: + pass +pol_a.delete() +pol_b.delete() +print(json.dumps({"ok": True})) +""" + +# Validates utf8mb4_bin on MySQL and native case-sensitivity on PostgreSQL. +# "CaseNet" and "casenet" must be distinct; second "CaseNet" must fail. +_CASE_SENSITIVE_UNIQUE = """ +import json, os, django +os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") +django.setup() +from django.db import IntegrityError +from SNMP.models import Network +Network.objects.filter(name__in=["CaseNet", "casenet"]).delete() +Network.objects.create(name="CaseNet", network_range="10.1.0.0/24") +# Different case must succeed +Network.objects.create(name="casenet", network_range="10.2.0.0/24") +# Exact duplicate must fail +try: + Network.objects.create(name="CaseNet", network_range="10.3.0.0/24") + raise AssertionError("Expected IntegrityError for duplicate Network.name") +except IntegrityError: + pass +finally: + Network.objects.filter(name__in=["CaseNet", "casenet"]).delete() +print(json.dumps({"ok": True})) +""" + + +# --------------------------------------------------------------------------- +# Tests (all parametrized over postgres + mysql via engine_env) +# --------------------------------------------------------------------------- + +def test_policy_crud(engine_env, tmp_path): + engine, env = engine_env + full_env = {**env, "LOGSTASHUI_DATA_DIR": str(tmp_path)} + _migrate(full_env) + assert json.loads(_run_python(_POLICY_CRUD, full_env).strip())["ok"] is True + + +def test_connection_crud(engine_env, tmp_path): + engine, env = engine_env + full_env = {**env, "LOGSTASHUI_DATA_DIR": str(tmp_path)} + _migrate(full_env) + assert json.loads(_run_python(_CONNECTION_CRUD, full_env).strip())["ok"] is True + + +def test_pipeline_crud(engine_env, tmp_path): + engine, env = engine_env + full_env = {**env, "LOGSTASHUI_DATA_DIR": str(tmp_path)} + _migrate(full_env) + assert json.loads(_run_python(_PIPELINE_CRUD, full_env).strip())["ok"] is True + + +def test_revision_json_roundtrip(engine_env, tmp_path): + engine, env = engine_env + full_env = {**env, "LOGSTASHUI_DATA_DIR": str(tmp_path)} + _migrate(full_env) + assert json.loads(_run_python(_REVISION_JSON, full_env).strip())["ok"] is True + + +def test_status_blob_roundtrip(engine_env, tmp_path): + engine, env = engine_env + full_env = {**env, "LOGSTASHUI_DATA_DIR": str(tmp_path)} + _migrate(full_env) + assert json.loads(_run_python(_STATUS_BLOB_JSON, full_env).strip())["ok"] is True + + +def test_status_blob_null_roundtrip(engine_env, tmp_path): + engine, env = engine_env + full_env = {**env, "LOGSTASHUI_DATA_DIR": str(tmp_path)} + _migrate(full_env) + assert json.loads(_run_python(_STATUS_BLOB_NULL, full_env).strip())["ok"] is True + + +def test_snmp_network_crud(engine_env, tmp_path): + engine, env = engine_env + full_env = {**env, "LOGSTASHUI_DATA_DIR": str(tmp_path)} + _migrate(full_env) + assert json.loads(_run_python(_SNMP_NETWORK_CRUD, full_env).strip())["ok"] is True + + +def test_unique_policy_name(engine_env, tmp_path): + engine, env = engine_env + full_env = {**env, "LOGSTASHUI_DATA_DIR": str(tmp_path)} + _migrate(full_env) + assert json.loads(_run_python(_UNIQUE_POLICY_NAME, full_env).strip())["ok"] is True + + +def test_unique_pipeline_per_policy(engine_env, tmp_path): + engine, env = engine_env + full_env = {**env, "LOGSTASHUI_DATA_DIR": str(tmp_path)} + _migrate(full_env) + assert json.loads(_run_python(_UNIQUE_PIPELINE_PER_POLICY, full_env).strip())["ok"] is True + + +def test_case_sensitive_unique(engine_env, tmp_path): + """Both engines treat unique names as case-sensitive. + On MySQL this validates utf8mb4_bin is active; on PostgreSQL it's the default. + """ + engine, env = engine_env + full_env = {**env, "LOGSTASHUI_DATA_DIR": str(tmp_path)} + _migrate(full_env) + assert json.loads(_run_python(_CASE_SENSITIVE_UNIQUE, full_env).strip())["ok"] is True diff --git a/tests/Database/unit/__init__.py b/tests/Database/unit/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/Database/unit/test_database.py b/tests/Database/unit/test_database.py new file mode 100644 index 0000000..5afb0de --- /dev/null +++ b/tests/Database/unit/test_database.py @@ -0,0 +1,245 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +from pathlib import Path +from types import SimpleNamespace + +import pytest + +from LogstashUI.database import ( + build_databases, + canonical_engine, + check_server_version, +) + + +def _clear_db_env(monkeypatch): + for name in ( + "LOGSTASHUI_DB_ENGINE", + "LOGSTASHUI_DB_NAME", + "LOGSTASHUI_DB_HOST", + "LOGSTASHUI_DB_PORT", + "LOGSTASHUI_DB_USER", + "LOGSTASHUI_DB_PASSWORD", + "LOGSTASHUI_DB_SSLMODE", + "LOGSTASHUI_DB_SSL_CA", + "LOGSTASHUI_DB_CONN_MAX_AGE", + "LOGSTASHUI_DB_CONN_HEALTH_CHECKS", + ): + monkeypatch.delenv(name, raising=False) + + +def test_build_databases_sqlite_default(tmp_path, monkeypatch): + _clear_db_env(monkeypatch) + db = build_databases(tmp_path) + assert db["default"]["ENGINE"] == "django.db.backends.sqlite3" + assert db["default"]["NAME"] == tmp_path / "db.sqlite3" + assert db["default"]["OPTIONS"]["timeout"] == 20 + assert "PRAGMA journal_mode=WAL" in db["default"]["OPTIONS"]["init_command"] + + +@pytest.mark.parametrize( + "raw,expected", + [ + ("", "sqlite"), + ("sqlite", "sqlite"), + ("sqlite3", "sqlite"), + ("postgres", "postgresql"), + ("postgresql", "postgresql"), + ("mysql", "mysql"), + ("mariadb", "mysql"), + ("my", "mysql"), + ("POSTGRESQL", "postgresql"), + ], +) +def test_canonical_engine_aliases(raw, expected): + assert canonical_engine(raw) == expected + + +def test_unknown_engine_fails(tmp_path, monkeypatch): + _clear_db_env(monkeypatch) + monkeypatch.setenv("LOGSTASHUI_DB_ENGINE", "oracle") + with pytest.raises(RuntimeError, match="Unknown LOGSTASHUI_DB_ENGINE"): + build_databases(tmp_path) + + +def test_postgresql_requires_host_user(tmp_path, monkeypatch): + _clear_db_env(monkeypatch) + monkeypatch.setenv("LOGSTASHUI_DB_ENGINE", "postgresql") + monkeypatch.setattr("LogstashUI.database._import_or_raise", lambda *a, **k: None) + with pytest.raises(RuntimeError, match="LOGSTASHUI_DB_HOST"): + build_databases(tmp_path) + monkeypatch.setenv("LOGSTASHUI_DB_HOST", "db.example") + with pytest.raises(RuntimeError, match="LOGSTASHUI_DB_USER"): + build_databases(tmp_path) + + +def test_build_databases_postgresql(tmp_path, monkeypatch): + _clear_db_env(monkeypatch) + monkeypatch.setenv("LOGSTASHUI_DB_ENGINE", "postgres") + monkeypatch.setenv("LOGSTASHUI_DB_HOST", "db.example") + monkeypatch.setenv("LOGSTASHUI_DB_USER", "lsui") + monkeypatch.setenv("LOGSTASHUI_DB_PASSWORD", "s3cret") + monkeypatch.setenv("LOGSTASHUI_DB_SSLMODE", "require") + monkeypatch.setenv("LOGSTASHUI_DB_SSL_CA", "/etc/ssl/db-ca.pem") + monkeypatch.setattr("LogstashUI.database._import_or_raise", lambda *a, **k: None) + db = build_databases(tmp_path)["default"] + assert db["ENGINE"] == "django.db.backends.postgresql" + assert db["NAME"] == "logstashui" + assert db["HOST"] == "db.example" + assert db["PORT"] == "5432" + assert db["USER"] == "lsui" + assert db["PASSWORD"] == "s3cret" + assert db["CONN_MAX_AGE"] == 60 + assert db["CONN_HEALTH_CHECKS"] is True + assert db["OPTIONS"]["sslmode"] == "require" + assert db["OPTIONS"]["sslrootcert"] == "/etc/ssl/db-ca.pem" + + +def test_build_databases_mysql_mariadb_alias(tmp_path, monkeypatch): + _clear_db_env(monkeypatch) + monkeypatch.setenv("LOGSTASHUI_DB_ENGINE", "mariadb") + monkeypatch.setenv("LOGSTASHUI_DB_HOST", "127.0.0.1") + monkeypatch.setenv("LOGSTASHUI_DB_USER", "lsui") + monkeypatch.setenv("LOGSTASHUI_DB_PORT", "3307") + monkeypatch.setenv("LOGSTASHUI_DB_CONN_MAX_AGE", "0") + monkeypatch.setenv("LOGSTASHUI_DB_CONN_HEALTH_CHECKS", "false") + monkeypatch.setattr("LogstashUI.database._import_or_raise", lambda *a, **k: None) + db = build_databases(tmp_path)["default"] + assert db["ENGINE"] == "django.db.backends.mysql" + assert db["PORT"] == "3307" + assert db["CONN_MAX_AGE"] == 0 + assert db["CONN_HEALTH_CHECKS"] is False + assert db["OPTIONS"]["charset"] == "utf8mb4" + assert "utf8mb4_bin" in db["OPTIONS"]["init_command"] + assert db["TEST"]["CHARSET"] == "utf8mb4" + assert db["TEST"]["COLLATION"] == "utf8mb4_bin" + + +def test_mysql_spoofs_pymysql_version_info(tmp_path, monkeypatch): + _clear_db_env(monkeypatch) + monkeypatch.setenv("LOGSTASHUI_DB_ENGINE", "mysql") + monkeypatch.setenv("LOGSTASHUI_DB_HOST", "127.0.0.1") + monkeypatch.setenv("LOGSTASHUI_DB_USER", "lsui") + installed = [] + fake = SimpleNamespace( + version_info=(1, 1, 1, "final", 0), + install_as_MySQLdb=lambda: installed.append(True), + ) + monkeypatch.setattr("LogstashUI.database._import_or_raise", lambda *a, **k: fake) + build_databases(tmp_path) + assert fake.version_info == (2, 2, 1, "final", 0) + assert installed + + +def test_postgresql_missing_driver(tmp_path, monkeypatch): + _clear_db_env(monkeypatch) + monkeypatch.setenv("LOGSTASHUI_DB_ENGINE", "postgresql") + monkeypatch.setenv("LOGSTASHUI_DB_HOST", "localhost") + monkeypatch.setenv("LOGSTASHUI_DB_USER", "lsui") + + def boom(module, extra): + raise RuntimeError( + f"{module} is not installed. Install with: uv pip install 'LogstashUI[{extra}]'" + ) + + monkeypatch.setattr("LogstashUI.database._import_or_raise", boom) + with pytest.raises(RuntimeError, match=r"LogstashUI\[postgres\]"): + build_databases(tmp_path) + + +def test_check_server_version_sqlite_noop(): + class Conn: + vendor = "sqlite" + + check_server_version(Conn()) + + +def test_check_server_version_postgres_too_old(): + class Conn: + vendor = "postgresql" + pg_version = 130000 + + with pytest.raises(RuntimeError, match="PostgreSQL 14"): + check_server_version(Conn()) + + +def test_check_server_version_postgres_zero_is_too_old(): + class Conn: + vendor = "postgresql" + pg_version = 0 + + with pytest.raises(RuntimeError, match="PostgreSQL 14"): + check_server_version(Conn()) + + +def test_conn_max_age_invalid_raises_runtimeerror(tmp_path, monkeypatch): + _clear_db_env(monkeypatch) + monkeypatch.setenv("LOGSTASHUI_DB_CONN_MAX_AGE", "nope") + with pytest.raises(RuntimeError, match="LOGSTASHUI_DB_CONN_MAX_AGE"): + build_databases(tmp_path) + + +def test_password_is_stripped(tmp_path, monkeypatch): + _clear_db_env(monkeypatch) + monkeypatch.setenv("LOGSTASHUI_DB_ENGINE", "postgresql") + monkeypatch.setenv("LOGSTASHUI_DB_HOST", "db.example") + monkeypatch.setenv("LOGSTASHUI_DB_USER", "lsui") + monkeypatch.setenv("LOGSTASHUI_DB_PASSWORD", " secret\n") + monkeypatch.setattr("LogstashUI.database._import_or_raise", lambda *a, **k: None) + db = build_databases(tmp_path)["default"] + assert db["PASSWORD"] == "secret" + + +def test_ensure_psycopg_gevent_assigns_wait_select(): + from types import SimpleNamespace + + from LogstashUI.database import ensure_psycopg_gevent + + def wait_select(*args, **kwargs): + return "select" + + waiting = SimpleNamespace(wait_select=wait_select, wait=None) + ensure_psycopg_gevent(waiting) + assert waiting.wait is wait_select + + +def test_ensure_psycopg_gevent_does_not_raise(): + from LogstashUI.database import ensure_psycopg_gevent + + ensure_psycopg_gevent() + + +def test_check_server_version_mysql_and_mariadb(): + class Mysql: + vendor = "mysql" + mysql_is_mariadb = False + mysql_server_info = "8.0.36" + + def get_database_version(self): + return (8, 0, 36) + + check_server_version(Mysql()) + + class OldMysql: + vendor = "mysql" + mysql_is_mariadb = False + mysql_server_info = "5.7.44" + + def get_database_version(self): + return (5, 7, 44) + + with pytest.raises(RuntimeError, match="MySQL 8.0"): + check_server_version(OldMysql()) + + class Maria: + vendor = "mysql" + mysql_is_mariadb = True + mysql_server_info = "10.5.22-MariaDB" + + def get_database_version(self): + return (10, 5, 22) + + with pytest.raises(RuntimeError, match="MariaDB 10.6"): + check_server_version(Maria()) diff --git a/tests/Database/unit/test_migrate_engine.py b/tests/Database/unit/test_migrate_engine.py new file mode 100644 index 0000000..c0008f7 --- /dev/null +++ b/tests/Database/unit/test_migrate_engine.py @@ -0,0 +1,111 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +from argparse import Namespace +from pathlib import Path + +import pytest + +from LogstashUI import migrate_engine as me + + +def test_refuses_without_backup_flag(capsys): + ns = Namespace(to="postgresql", i_have_a_backup=False, pid=None, write_env=None) + with pytest.raises(SystemExit) as exc: + me.cmd_migrate_engine(ns) + assert exc.value.code == 2 + assert "back up" in capsys.readouterr().err.lower() + + +def test_refuses_sqlite_target(tmp_path, monkeypatch, capsys): + monkeypatch.setenv("LOGSTASHUI_DATA_DIR", str(tmp_path)) + ns = Namespace(to="sqlite", i_have_a_backup=True, pid=None, write_env=None) + with pytest.raises(SystemExit): + me.cmd_migrate_engine(ns) + + +def test_refuses_missing_sqlite_file(tmp_path, monkeypatch, capsys): + monkeypatch.setenv("LOGSTASHUI_DATA_DIR", str(tmp_path)) + ns = Namespace(to="postgresql", i_have_a_backup=True, pid=None, write_env=None) + with pytest.raises(SystemExit) as exc: + me.cmd_migrate_engine(ns) + assert exc.value.code == 1 + assert "db.sqlite3" in capsys.readouterr().err + + +def test_stop_pid_sends_sigterm(tmp_path, monkeypatch): + pidfile = tmp_path / "gunicorn.pid" + pidfile.write_text("4242\n") + sent = {} + calls = {"n": 0} + + def kill_then_gone(pid, sig): + calls["n"] += 1 + if calls["n"] == 1: + sent["pid"] = pid + sent["sig"] = sig + return + raise ProcessLookupError() + + monkeypatch.setattr(me.os, "kill", kill_then_gone) + monkeypatch.setattr(me.time, "sleep", lambda s: None) + me.stop_gunicorn(pidfile) + assert sent["pid"] == 4242 + assert sent["sig"] == me.signal.SIGTERM + assert not pidfile.exists() + + +def test_write_env_appends(tmp_path, monkeypatch): + envf = tmp_path / "logstashui.default" + envf.write_text("LOGSTASHUI_DATA_DIR=/var/lib/logstashui\n") + monkeypatch.setenv("LOGSTASHUI_DB_HOST", "db.example") + monkeypatch.setenv("LOGSTASHUI_DB_USER", "lsui") + monkeypatch.setenv("LOGSTASHUI_DB_NAME", "logstashui") + me.write_env_file(envf, "postgresql") + me.write_env_file(envf, "postgresql") + text = envf.read_text() + assert text.count("LOGSTASHUI_DB_ENGINE=postgresql") == 1 + assert "LOGSTASHUI_DB_HOST=db.example" in text + assert "PASSWORD" not in text + + +def test_run_manage_sets_package_pythonpath(monkeypatch): + captured = {} + + def fake_run(cmd, env=None, check=False): + captured["env"] = env + class Result: + returncode = 0 + return Result() + + monkeypatch.setattr(me.subprocess, "run", fake_run) + me.run_manage(["migrate", "--noinput"], {"LOGSTASHUI_DB_ENGINE": "sqlite"}) + pythonpath = captured["env"]["PYTHONPATH"] + pkg_root = str(Path(me.__file__).resolve().parent.parent) + assert pythonpath.split(me.os.pathsep)[0] == pkg_root + + +def test_reset_postgres_sequences_does_not_require_psql(monkeypatch): + captured = {} + + def fake_run(cmd, env=None, check=False, capture_output=False, text=False): + captured["cmd"] = cmd + captured["env"] = env + class Result: + returncode = 0 + stdout = "" + stderr = "" + return Result() + + monkeypatch.setattr(me.subprocess, "run", fake_run) + me._reset_postgres_sequences({"LOGSTASHUI_DB_ENGINE": "postgresql"}) + assert captured["cmd"][0] == me.sys.executable + assert captured["cmd"][1] == "-c" + code = captured["cmd"][2] + assert "dbshell" not in code + assert "psql" not in code + assert "sequence_reset_sql" in code + assert "cursor.execute" in code + pkg_root = str(Path(me.__file__).resolve().parent.parent) + assert captured["env"]["PYTHONPATH"].split(me.os.pathsep)[0] == pkg_root diff --git a/tests/LogstashUI/__init__.py b/tests/LogstashUI/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/LogstashUI/unit/__init__.py b/tests/LogstashUI/unit/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/LogstashUI/unit/test_cli.py b/tests/LogstashUI/unit/test_cli.py new file mode 100644 index 0000000..ba63103 --- /dev/null +++ b/tests/LogstashUI/unit/test_cli.py @@ -0,0 +1,230 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +from argparse import Namespace + +from django.core.management.base import CommandError + +from LogstashUI.cli import _exec_gunicorn, build_parser, cmd_serve, install_systemd + + +def test_parser_defaults_to_serve(): + parser = build_parser() + ns = parser.parse_args([]) + assert ns.command == "serve" + + +def test_parser_manage_passthrough(): + parser = build_parser() + ns = parser.parse_args(["manage", "migrate", "--noinput"]) + assert ns.command == "manage" + assert ns.manage_args == ["migrate", "--noinput"] + + +def test_systemd_dry_run_writes_unit_and_default(tmp_path): + result = install_systemd( + output_dir=tmp_path, + exec_start="/usr/bin/logstashui serve", + user="logstashui", + group="logstashui", + data_dir="/var/lib/logstashui", + bind="0.0.0.0:8443", + workers=2, + allowed_hosts="ui.example", + csrf_trusted_origins="https://ui.example:8443", + tls="true", + host_hostname="ui.example", + host_ips="10.0.0.5", + tls_sans="ui.example,10.0.0.5", + agent_ui_url="https://ui.example:8443", + no_auth="false", + dry_run=True, + ) + unit = tmp_path / "logstashui.service" + envf = tmp_path / "logstashui.default" + assert unit.is_file() + assert envf.is_file() + unit_text = unit.read_text() + env_text = envf.read_text() + assert "EnvironmentFile=-/etc/default/logstashui" in unit_text + assert "ExecStart=/usr/bin/logstashui serve" in unit_text + assert "User=logstashui" in unit_text + assert "LOGSTASHUI_DATA_DIR=/var/lib/logstashui" in env_text + assert "LOGSTASHUI_BIND=0.0.0.0:8443" in env_text + assert "LOGSTASHUI_NO_AUTH=false" in env_text + assert result["unit"] == unit + assert result["default"] == envf + assert "LOGSTASHUI_DB_ENGINE=postgresql" not in env_text + assert "# LOGSTASHUI_DB_ENGINE=sqlite" in env_text + + +def test_systemd_env_includes_postgres_when_passed(tmp_path): + result = install_systemd( + output_dir=tmp_path, + exec_start="/usr/bin/logstashui serve", + user="logstashui", + group="logstashui", + data_dir="/var/lib/logstashui", + bind="0.0.0.0:8443", + workers=2, + allowed_hosts="*", + csrf_trusted_origins="", + tls="true", + host_hostname="", + host_ips="", + tls_sans="", + agent_ui_url="", + no_auth="false", + dry_run=True, + db_engine="postgresql", + db_host="db.example", + db_port="5432", + db_name="logstashui", + db_user="lsui", + ) + text = (tmp_path / "logstashui.default").read_text() + assert "LOGSTASHUI_DB_ENGINE=postgresql" in text + assert "LOGSTASHUI_DB_HOST=db.example" in text + assert "LOGSTASHUI_DB_PORT=5432" in text + assert "LOGSTASHUI_DB_NAME=logstashui" in text + assert "LOGSTASHUI_DB_USER=lsui" in text + assert result["default"] == tmp_path / "logstashui.default" + + +def test_serve_snmp_commanderror_does_not_abort(monkeypatch): + """execute_from_command_line sys.exits on CommandError; call_command must not.""" + from LogstashUI import cli + + monkeypatch.setattr(cli, "_manage", lambda argv: None) + monkeypatch.setattr(cli, "_check_db_floor", lambda: None) + monkeypatch.setenv("LOGSTASHUI_TLS", "false") + + def fake_call(name, *args, **kwargs): + if name == "sync_snmp_official_data": + raise CommandError("sync failed") + return None + + monkeypatch.setattr("django.core.management.call_command", fake_call) + + exec_called = {} + + def fake_execvp(file, args): + exec_called["file"] = file + raise SystemExit(0) + + monkeypatch.setattr(cli.os, "execvp", fake_execvp) + + ns = Namespace(skip_migrate=False, no_tls=True, bind="127.0.0.1:8443", workers=1) + try: + cmd_serve(ns) + except SystemExit as exc: + assert exc.code == 0 + assert exec_called.get("file") == "gunicorn" + + +def test_parser_migrate_engine_requires_backup_flag(): + parser = build_parser() + ns = parser.parse_args(["migrate-engine", "--to", "postgresql"]) + assert ns.command == "migrate-engine" + assert ns.to == "postgresql" + assert ns.i_have_a_backup is False + + +def test_parser_migrate_engine_accepts_mariadb_alias(): + parser = build_parser() + ns = parser.parse_args(["migrate-engine", "--to", "mariadb", "--i-have-a-backup"]) + assert ns.to == "mariadb" + assert ns.i_have_a_backup is True + + +def test_serve_checks_version_before_migrate(monkeypatch): + from LogstashUI import cli + + order = [] + monkeypatch.setattr(cli, "_check_db_floor", lambda: order.append("check")) + monkeypatch.setattr(cli, "_manage", lambda argv: order.append(argv[0])) + monkeypatch.setattr(cli, "_best_effort_call", lambda *a, **k: None) + monkeypatch.setenv("LOGSTASHUI_TLS", "false") + + def fake_execvp(file, args): + raise SystemExit(0) + + monkeypatch.setattr(cli.os, "execvp", fake_execvp) + ns = Namespace(skip_migrate=False, no_tls=True, bind="127.0.0.1:8443", workers=1) + try: + cmd_serve(ns) + except SystemExit: + pass + assert order[0] == "check" + assert "migrate" in order + + +def test_serve_checks_version_when_skip_migrate(monkeypatch, tmp_path): + from LogstashUI import cli + + called = [] + monkeypatch.setattr(cli, "_check_db_floor", lambda: called.append(True)) + monkeypatch.setattr(cli, "_manage", lambda argv: None) + monkeypatch.setenv("LOGSTASHUI_TLS", "false") + monkeypatch.setenv("LOGSTASHUI_DATA_DIR", str(tmp_path)) + + def fake_execvp(file, args): + raise SystemExit(0) + + monkeypatch.setattr(cli.os, "execvp", fake_execvp) + ns = Namespace(skip_migrate=True, no_tls=True, bind="127.0.0.1:8443", workers=1) + try: + cmd_serve(ns) + except SystemExit: + pass + assert called == [True] + + +def test_serve_adds_pidfile_and_warns_sqlite(monkeypatch, tmp_path, capsys): + from LogstashUI import cli + + monkeypatch.setattr(cli, "_manage", lambda argv: None) + monkeypatch.setattr(cli, "_check_db_floor", lambda: None) + monkeypatch.setenv("LOGSTASHUI_TLS", "false") + monkeypatch.setenv("LOGSTASHUI_DATA_DIR", str(tmp_path)) + monkeypatch.delenv("LOGSTASHUI_DB_ENGINE", raising=False) + + captured = {} + + def fake_execvp(file, args): + captured["file"] = file + captured["args"] = list(args) + raise SystemExit(0) + + monkeypatch.setattr(cli.os, "execvp", fake_execvp) + ns = Namespace(skip_migrate=True, no_tls=True, bind="127.0.0.1:8443", workers=2) + try: + cmd_serve(ns) + except SystemExit: + pass + assert "--pid" in captured["args"] + pid_idx = captured["args"].index("--pid") + assert captured["args"][pid_idx + 1].endswith("gunicorn.pid") + err = capsys.readouterr().err + assert "SQLite is the small-install default" in err + + +def test_exec_gunicorn_frozen_runs_in_process(monkeypatch): + import sys + + monkeypatch.setattr(sys, "frozen", True, raising=False) + seen = {} + + def fake_run(): + seen["argv"] = list(sys.argv) + return 0 + + monkeypatch.setattr("gunicorn.app.wsgiapp.run", fake_run) + rc = _exec_gunicorn( + ["gunicorn", "LogstashUI.wsgi:application", "--bind", "0.0.0.0:8443"] + ) + assert rc == 0 + assert seen["argv"][0] == "gunicorn" + assert "LogstashUI.wsgi:application" in seen["argv"] + diff --git a/tests/LogstashUI/unit/test_config.py b/tests/LogstashUI/unit/test_config.py new file mode 100644 index 0000000..3c8e862 --- /dev/null +++ b/tests/LogstashUI/unit/test_config.py @@ -0,0 +1,61 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +from LogstashUI.config import load_config, merge_allowed_hosts + + +def test_merge_allowed_hosts_wildcard_unchanged(): + assert merge_allowed_hosts(allowed="*", host_ips="10.11.3.107") == ["*"] + + +def test_merge_allowed_hosts_appends_pod_ip(): + hosts = merge_allowed_hosts( + allowed="logstashui.example.com,logstashui", + host_ips="10.11.3.107", + pod_ip="", + ) + assert hosts == ["logstashui.example.com", "logstashui", "10.11.3.107"] + + +def test_merge_allowed_hosts_pod_ip_env_and_no_dupes(monkeypatch): + monkeypatch.setenv("ALLOWED_HOSTS", "logstashui") + monkeypatch.setenv("LOGSTASHUI_HOST_IPS", "10.11.3.107") + monkeypatch.setenv("POD_IP", "10.11.3.107") + assert merge_allowed_hosts() == ["logstashui", "10.11.3.107"] + + +def test_load_config_defaults(monkeypatch): + monkeypatch.delenv("LOGSTASHUI_NO_AUTH", raising=False) + monkeypatch.delenv("LOGSTASHUI_AGENT_UI_URL", raising=False) + monkeypatch.delenv("LOGSTASHUI_INCLUDE_CA_FINGERPRINT", raising=False) + cfg = load_config() + assert cfg["no_auth"]["enabled"] is False + assert cfg["agent"]["ui_url"] == "" + assert cfg["agent"]["include_ca_fingerprint"] is True + + +def test_load_config_no_auth_env(monkeypatch): + monkeypatch.setenv("LOGSTASHUI_NO_AUTH", "true") + cfg = load_config() + assert cfg["no_auth"]["enabled"] is True + monkeypatch.setenv("LOGSTASHUI_NO_AUTH", "0") + cfg = load_config() + assert cfg["no_auth"]["enabled"] is False + + +def test_load_config_agent_env(monkeypatch): + monkeypatch.setenv("LOGSTASHUI_AGENT_UI_URL", "https://ui.example:8443/") + monkeypatch.setenv("LOGSTASHUI_INCLUDE_CA_FINGERPRINT", "false") + cfg = load_config() + assert cfg["agent"]["ui_url"] == "https://ui.example:8443" + assert cfg["agent"]["include_ca_fingerprint"] is False + + +def test_load_config_ignores_yaml_env(monkeypatch, tmp_path): + yml = tmp_path / "logstashui.yml" + yml.write_text("no_auth:\n enabled: true\n") + monkeypatch.setenv("LOGSTASHUI_CONFIG", str(yml)) + monkeypatch.delenv("LOGSTASHUI_NO_AUTH", raising=False) + cfg = load_config() + assert cfg["no_auth"]["enabled"] is False diff --git a/tests/LogstashUI/unit/test_logging_config.py b/tests/LogstashUI/unit/test_logging_config.py new file mode 100644 index 0000000..b798d5a --- /dev/null +++ b/tests/LogstashUI/unit/test_logging_config.py @@ -0,0 +1,50 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +import pytest + +from LogstashUI.logging_config import resolve_django_log_levels, resolve_log_level + + +def test_log_level_defaults_follow_debug_flag(monkeypatch): + monkeypatch.delenv("LOGSTASHUI_LOG_LEVEL", raising=False) + assert resolve_log_level("LOGSTASHUI_LOG_LEVEL", default="INFO") == "INFO" + assert resolve_log_level("LOGSTASHUI_LOG_LEVEL", default="DEBUG") == "DEBUG" + + +def test_log_level_env_override(monkeypatch): + monkeypatch.setenv("LOGSTASHUI_LOG_LEVEL", "warning") + assert resolve_log_level("LOGSTASHUI_LOG_LEVEL", default="INFO") == "WARNING" + monkeypatch.setenv("LOGSTASHUI_LOG_LEVEL", "WARN") + assert resolve_log_level("LOGSTASHUI_LOG_LEVEL", default="INFO") == "WARNING" + + +def test_log_level_invalid(monkeypatch): + monkeypatch.setenv("LOGSTASHUI_LOG_LEVEL", "verbose") + with pytest.raises(RuntimeError, match="LOGSTASHUI_LOG_LEVEL"): + resolve_log_level("LOGSTASHUI_LOG_LEVEL", default="INFO") + + +def test_django_levels_default(monkeypatch): + monkeypatch.delenv("LOGSTASHUI_DJANGO_LOG_LEVEL", raising=False) + monkeypatch.delenv("DJANGO_LOG_LEVEL", raising=False) + django_level, request_level = resolve_django_log_levels() + assert django_level == "INFO" + assert request_level == "ERROR" + + +def test_django_levels_prefixed_env(monkeypatch): + monkeypatch.setenv("LOGSTASHUI_DJANGO_LOG_LEVEL", "debug") + monkeypatch.setenv("DJANGO_LOG_LEVEL", "error") + django_level, request_level = resolve_django_log_levels() + assert django_level == "DEBUG" + assert request_level == "DEBUG" + + +def test_django_levels_alias(monkeypatch): + monkeypatch.delenv("LOGSTASHUI_DJANGO_LOG_LEVEL", raising=False) + monkeypatch.setenv("DJANGO_LOG_LEVEL", "warning") + django_level, request_level = resolve_django_log_levels() + assert django_level == "WARNING" + assert request_level == "WARNING" diff --git a/tests/LogstashUI/unit/test_paths.py b/tests/LogstashUI/unit/test_paths.py new file mode 100644 index 0000000..86af809 --- /dev/null +++ b/tests/LogstashUI/unit/test_paths.py @@ -0,0 +1,91 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +from pathlib import Path + +from LogstashUI.paths import ( + PROJECT_ROOT, + maybe_migrate_legacy_data, + resolve_data_dir, + resolve_logs_dir, +) + + +def test_env_data_dir_wins(tmp_path, monkeypatch): + dest = tmp_path / "from-env" + dest.mkdir() + monkeypatch.setenv("LOGSTASHUI_DATA_DIR", str(dest)) + monkeypatch.delenv("LOGSTASHUI_LOGS_DIR", raising=False) + assert resolve_data_dir(migrate_legacy=False) == dest + assert resolve_logs_dir(dest) == dest / "logs" + + +def test_env_logs_dir_wins(tmp_path, monkeypatch): + data = tmp_path / "data" + logs = tmp_path / "logs" + data.mkdir() + logs.mkdir() + monkeypatch.setenv("LOGSTASHUI_DATA_DIR", str(data)) + monkeypatch.setenv("LOGSTASHUI_LOGS_DIR", str(logs)) + assert resolve_logs_dir() == logs + + +def test_relative_env_path_is_absolute(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + monkeypatch.setenv("LOGSTASHUI_DATA_DIR", "relative-data") + resolved = resolve_data_dir(migrate_legacy=False) + assert resolved.is_absolute() + assert resolved == (tmp_path / "relative-data").resolve() + + +def test_migrate_legacy_copies_sqlite(tmp_path): + legacy = tmp_path / "legacy" + dest = tmp_path / "dest" + legacy.mkdir() + (legacy / "db.sqlite3").write_bytes(b"sqlite") + (legacy / "tls").mkdir() + (legacy / "tls" / "product-ca.crt").write_text("ca") + # Point LEGACY_DATA_DIR by copying into maybe_migrate with patched constant + from LogstashUI import paths as paths_mod + + original = paths_mod.LEGACY_DATA_DIR + try: + paths_mod.LEGACY_DATA_DIR = legacy + maybe_migrate_legacy_data(dest) + assert (dest / "db.sqlite3").read_bytes() == b"sqlite" + assert (dest / "tls" / "product-ca.crt").read_text() == "ca" + maybe_migrate_legacy_data(dest) # idempotent + assert (dest / "db.sqlite3").read_bytes() == b"sqlite" + finally: + paths_mod.LEGACY_DATA_DIR = original + + +def test_pytest_default_is_legacy_not_checkout_bind(): + # Under pytest, default must not be /logstashui_data + resolved = resolve_data_dir(migrate_legacy=False) + assert resolved != PROJECT_ROOT / "logstashui_data" + assert resolved.name == "data" + + +def test_native_default_is_cwd_logstashui_data(tmp_path, monkeypatch): + """Installed / CLI default is $(pwd)/logstashui_data, not site-packages.""" + from LogstashUI import paths as paths_mod + + monkeypatch.chdir(tmp_path) + monkeypatch.delenv("LOGSTASHUI_DATA_DIR", raising=False) + monkeypatch.setattr(paths_mod, "_is_pytest", lambda: False) + resolved = paths_mod.resolve_data_dir(migrate_legacy=False) + assert resolved == (tmp_path / "logstashui_data").resolve() + + +def test_packaged_docs_dir_expects_content_images(tmp_path, monkeypatch): + """Wheel install: docs root is Documentation/content (images live under that).""" + from LogstashUI import paths as paths_mod + from LogstashUI.paths import resolve_docs_dir + + monkeypatch.delenv("LOGSTASHUI_DOCS_DIR", raising=False) + monkeypatch.setattr(paths_mod, "PROJECT_ROOT", tmp_path / "not-a-checkout") + resolved = resolve_docs_dir() + assert resolved == paths_mod.BASE_DIR / "Documentation" / "content" + assert resolved / "images" == paths_mod.BASE_DIR / "Documentation" / "content" / "images" diff --git a/tests/Management/__init__.py b/tests/Management/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/Management/unit/__init__.py b/tests/Management/unit/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/Management/unit/test_views.py b/tests/Management/unit/test_views.py new file mode 100644 index 0000000..02b83cc --- /dev/null +++ b/tests/Management/unit/test_views.py @@ -0,0 +1,1066 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +import pytest +from django.contrib.auth.models import User + + +# ============================================================================ +# FIXTURES +# ============================================================================ + +@pytest.fixture +def readonly_user(db): + """Create a readonly-role test user""" + user = User.objects.create_user( + username='readonlyuser', + password='testpass123', + email='readonly@example.com' + ) + user.is_superuser = False + user.is_staff = False + user.save() + # The signal creates a profile with role='admin' by default; override it. + user.profile.role = 'readonly' + user.profile.save() + return user + + +@pytest.fixture +def readonly_client(client, readonly_user): + """Client authenticated as a readonly user""" + client.login(username='readonlyuser', password='testpass123') + return client + + +# ============================================================================ +# SECTION 1: BootstrapLoginView — First-Run & Login Tests +# ============================================================================ + +@pytest.mark.django_db +class TestFirstRunLogin: + """ + Tests for BootstrapLoginView: the first-user registration flow and + the normal login flow (finding 4e). + """ + + def test_first_run_shows_registration_form(self, client): + """ + With no users in the database, the login page should render the + UserCreationForm (first-run registration mode). + """ + assert not User.objects.exists() + response = client.get('/Management/Login/') + assert response.status_code == 200 + # Context flag must be True to drive the template + assert response.context['is_first_run'] is True + # Registration fields should be present + assert b'password1' in response.content or b'Create Your Account' in response.content + + def test_first_run_creates_admin_user(self, client): + """ + POSTing valid credentials on first run should create a user with + is_superuser=True and role='admin'. + """ + assert not User.objects.exists() + + response = client.post('/Management/Login/', { + 'username': 'firstadmin', + 'password1': 'StrongPass123!', + 'password2': 'StrongPass123!', + }) + + # Should redirect after successful creation + assert response.status_code in (200, 302) + assert User.objects.filter(username='firstadmin').exists() + + user = User.objects.get(username='firstadmin') + assert user.is_superuser, "First user must be is_superuser" + assert user.is_staff, "First user must be is_staff" + assert user.profile.role == 'admin', "First user must have role='admin'" + + def test_first_run_weak_password_rejected(self, client): + """ + A weak password on the first-run form should be rejected and no user + should be created. + """ + assert not User.objects.exists() + + response = client.post('/Management/Login/', { + 'username': 'firstadmin', + 'password1': '123', + 'password2': '123', + }) + + # Form should re-render with errors, not redirect + assert response.status_code == 200 + assert not User.objects.exists(), "No user should be created with a weak password" + + def test_normal_login_shows_auth_form(self, db): + """ + When at least one user exists, the login page should render the + AuthenticationForm (normal login mode). + """ + from django.test import Client + User.objects.create_user(username='existinguser', password='pass123') + client = Client() + response = client.get('/Management/Login/') + assert response.status_code == 200 + assert response.context['is_first_run'] is False + assert b'Sign In' in response.content or b'password' in response.content + + def test_normal_login_success(self, db): + """ + Correct credentials on the standard login form should log the user in + and redirect to the home page. + """ + from django.test import Client + User.objects.create_user(username='loginuser', password='ValidPass123!') + client = Client() + response = client.post('/Management/Login/', { + 'username': 'loginuser', + 'password': 'ValidPass123!', + }) + # Successful login redirects + assert response.status_code == 302 + + def test_normal_login_wrong_password(self, db): + """ + Wrong credentials should re-render the form with errors and not log in. + """ + from django.test import Client + User.objects.create_user(username='loginuser', password='ValidPass123!') + client = Client() + response = client.post('/Management/Login/', { + 'username': 'loginuser', + 'password': 'WrongPassword!', + }) + # Should re-render the page (200), not redirect + assert response.status_code == 200 + assert response.context['form'].errors + + def test_first_run_form_not_shown_after_user_exists(self, db): + """ + Once a user exists, a second visitor should NOT see the registration + form — even if they know to hit /Management/Login/ on a fresh browser. + This tests the first-run guard doesn't leak to after setup. + """ + from django.test import Client + User.objects.create_user(username='alreadysetup', password='pass123') + client = Client() + response = client.get('/Management/Login/') + assert response.context['is_first_run'] is False + # Registration-only fields should not appear + assert b'password1' not in response.content + + +# ============================================================================ +# SECTION 2: User Management CRUD Tests +# ============================================================================ + +@pytest.mark.django_db +class TestUserManagementCRUD: + """Test User Create, Read, Update, Delete operations""" + + def test_create_user_success(self, authenticated_client): + """Test successful user creation""" + response = authenticated_client.post('/Management/Users/', { + 'action': 'add', + 'username': 'newuser', + 'password': 'SecurePass123!', + 'password2': 'SecurePass123!', + 'email': 'newuser@example.com', + 'role': 'admin' + }) + + assert response.status_code == 200 + assert b'window.location.reload()' in response.content + + # Verify user was created + assert User.objects.filter(username='newuser').exists() + new_user = User.objects.get(username='newuser') + assert new_user.is_superuser + assert new_user.is_staff + + def test_create_user_duplicate_username(self, authenticated_client, test_user): + """Test creating user with duplicate username""" + response = authenticated_client.post('/Management/Users/', { + 'action': 'add', + 'username': 'testuser', # Already exists + 'password': 'SecurePass123!', + 'password2': 'SecurePass123!', + 'email': 'duplicate@example.com' + }) + + assert response.status_code == 200 + assert b'Username already exists' in response.content + + def test_create_user_password_mismatch(self, authenticated_client): + """Test creating user with mismatched passwords""" + response = authenticated_client.post('/Management/Users/', { + 'action': 'add', + 'username': 'newuser', + 'password': 'SecurePass123!', + 'password2': 'DifferentPass123!', + 'email': 'newuser@example.com' + }) + + assert response.status_code == 200 + assert b"didn't match" in response.content + + def test_create_user_weak_password(self, authenticated_client): + """Test creating user with weak password""" + response = authenticated_client.post('/Management/Users/', { + 'action': 'add', + 'username': 'newuser', + 'password': '123', # Too short + 'password2': '123', + 'email': 'newuser@example.com' + }) + + assert response.status_code == 200 + # Should contain password validation error + assert b'red-500' in response.content + + def test_update_user_password_success(self, authenticated_client, db): + """Test successful user password update""" + # Create a second user to update + other_user = User.objects.create_user( + username='otheruser', + password='oldpass123', + email='other@example.com' + ) + + response = authenticated_client.post('/Management/Users/', { + 'action': 'update_password', + 'user_id': other_user.id, + 'new_password': 'NewSecurePass123!', + 'new_password2': 'NewSecurePass123!' + }) + + assert response.status_code == 200 + assert b'window.location.reload()' in response.content + + # Verify password was updated + other_user.refresh_from_db() + assert other_user.check_password('NewSecurePass123!') + + def test_update_user_password_mismatch(self, authenticated_client, db): + """Test updating user password with mismatch""" + other_user = User.objects.create_user( + username='otheruser', + password='oldpass123', + email='other@example.com' + ) + + response = authenticated_client.post('/Management/Users/', { + 'action': 'update_password', + 'user_id': other_user.id, + 'new_password': 'NewPass123!', + 'new_password2': 'DifferentPass123!' + }) + + assert response.status_code == 200 + assert b"didn't match" in response.content + + def test_delete_user_success(self, authenticated_client, db): + """Test successful user deletion""" + # Create a second user to delete + other_user = User.objects.create_user( + username='deleteuser', + password='pass123', + email='delete@example.com' + ) + user_id = other_user.id + + response = authenticated_client.post('/Management/Users/', { + 'action': 'delete', + 'user_id': user_id + }) + + assert response.status_code == 200 + + # Verify user was deleted + assert not User.objects.filter(id=user_id).exists() + + def test_delete_last_user_prevented(self, authenticated_client, test_user): + """Test that deleting the last user is prevented""" + response = authenticated_client.post('/Management/Users/', { + 'action': 'delete', + 'user_id': test_user.id + }) + + assert response.status_code == 200 + assert b'Cannot delete the last user' in response.content + + # Verify user still exists + assert User.objects.filter(id=test_user.id).exists() + + def test_delete_own_account_prevented(self, authenticated_client, test_user, db): + """Test that users cannot delete their own account""" + # Create a second user so we're not the last user + User.objects.create_user( + username='otheruser', + password='pass123', + email='other@example.com' + ) + + response = authenticated_client.post('/Management/Users/', { + 'action': 'delete', + 'user_id': test_user.id + }) + + assert response.status_code == 200 + assert b'cannot delete your own account' in response.content + + # Verify user still exists + assert User.objects.filter(id=test_user.id).exists() + + +# ============================================================================ +# SECTION 3: Authorization / Role Enforcement Tests +# ============================================================================ + +@pytest.mark.django_db +class TestReadonlyUserBlocked: + """ + Verify that a user with role='readonly' cannot perform any write + operations on the Users management endpoint (finding 4a / issue 2c). + """ + + def test_readonly_cannot_add_user(self, readonly_client): + """Readonly user should receive 403 when attempting to add a user""" + response = readonly_client.post('/Management/Users/', { + 'action': 'add', + 'username': 'shouldnotexist', + 'password': 'SecurePass123!', + 'password2': 'SecurePass123!', + 'email': 'nope@example.com' + }) + + assert response.status_code == 403 + assert b'Access denied' in response.content + assert not User.objects.filter(username='shouldnotexist').exists() + + def test_readonly_cannot_delete_user(self, readonly_client, test_user): + """Readonly user should receive 403 when attempting to delete a user""" + response = readonly_client.post('/Management/Users/', { + 'action': 'delete', + 'user_id': test_user.id + }) + + assert response.status_code == 403 + assert b'Access denied' in response.content + # Confirm the user was NOT deleted + assert User.objects.filter(id=test_user.id).exists() + + def test_readonly_cannot_update_password(self, readonly_client, test_user): + """Readonly user should receive 403 when attempting to update a password""" + response = readonly_client.post('/Management/Users/', { + 'action': 'update_password', + 'user_id': test_user.id, + 'new_password': 'HackedPass123!', + 'new_password2': 'HackedPass123!' + }) + + assert response.status_code == 403 + assert b'Access denied' in response.content + # Confirm original password still works + test_user.refresh_from_db() + assert test_user.check_password('testpass123') + + def test_readonly_cannot_update_role(self, readonly_client, test_user): + """Readonly user should receive 403 when attempting to change a user role""" + response = readonly_client.post('/Management/Users/', { + 'action': 'update_role', + 'user_id': test_user.id, + 'role': 'readonly' + }) + + assert response.status_code == 403 + assert b'Access denied' in response.content + + def test_readonly_can_view_users_page(self, readonly_client): + """Readonly user should still be able to GET the users page""" + response = readonly_client.get('/Management/Users/') + assert response.status_code == 200 + + +@pytest.mark.django_db +class TestRoleValidation: + """ + Confirm that only valid role values ('admin', 'readonly') are accepted + server-side in add and update_role actions (finding 2c). + """ + + def test_add_user_with_invalid_role_rejected(self, authenticated_client): + """Submitting an invalid role string should return an error and not create the user""" + response = authenticated_client.post('/Management/Users/', { + 'action': 'add', + 'username': 'rolebreaker', + 'password': 'SecurePass123!', + 'password2': 'SecurePass123!', + 'email': 'rolebreaker@example.com', + 'role': 'superadmin' # Not a valid choice + }) + # Should return error response + assert response.status_code == 200 + assert b'Invalid role' in response.content + # User should not be created + assert not User.objects.filter(username='rolebreaker').exists() + + def test_update_role_with_invalid_role_rejected(self, authenticated_client, test_user): + """Submitting an invalid role to update_role should not persist it""" + response = authenticated_client.post('/Management/Users/', { + 'action': 'update_role', + 'user_id': test_user.id, + 'role': 'god' # Not a valid choice + }) + test_user.refresh_from_db() + assert test_user.profile.role in ('admin', 'readonly'), ( + f"Invalid role '{test_user.profile.role}' was saved to the database" + ) + + +# ============================================================================ +# SECTION 4: Update Role Tests +# ============================================================================ + +@pytest.mark.django_db +class TestUpdateRole: + """Test the update_role action on the Users management endpoint (finding 4b)""" + + def test_update_role_admin_to_readonly(self, authenticated_client, test_user, db): + """Successfully change a user's role from admin to readonly""" + # Create a second user with admin role (signal default) + other_user = User.objects.create_user( + username='otheradmin', + password='pass123', + email='other@example.com' + ) + assert other_user.profile.role == 'admin' + + response = authenticated_client.post('/Management/Users/', { + 'action': 'update_role', + 'user_id': other_user.id, + 'role': 'readonly' + }) + + assert response.status_code == 200 + other_user.refresh_from_db() + assert other_user.profile.role == 'readonly' + # Verify Django permissions were synced + assert not other_user.is_superuser + assert not other_user.is_staff + + def test_update_role_readonly_to_admin(self, authenticated_client, db): + """Successfully change a user's role from readonly to admin""" + user = User.objects.create_user( + username='readonlyuser', + password='pass123', + email='ro@example.com' + ) + user.profile.role = 'readonly' + user.profile.save() + + response = authenticated_client.post('/Management/Users/', { + 'action': 'update_role', + 'user_id': user.id, + 'role': 'admin' + }) + + assert response.status_code == 200 + user.refresh_from_db() + assert user.profile.role == 'admin' + # Verify Django permissions were synced + assert user.is_superuser + assert user.is_staff + + def test_update_role_no_change_returns_message(self, authenticated_client, db): + """Submitting the same role that already exists should return a message, not reload""" + user = User.objects.create_user( + username='sameroleuser', + password='pass123', + email='same@example.com' + ) + # Default role is 'admin' + assert user.profile.role == 'admin' + + response = authenticated_client.post('/Management/Users/', { + 'action': 'update_role', + 'user_id': user.id, + 'role': 'admin' + }) + + assert response.status_code == 200 + # Should say no changes made, NOT trigger a reload + assert b'No changes made' in response.content + assert b'window.location.reload()' not in response.content + + def test_update_role_user_not_found(self, authenticated_client): + """Passing a non-existent user_id should return a friendly error""" + response = authenticated_client.post('/Management/Users/', { + 'action': 'update_role', + 'user_id': 999999, + 'role': 'readonly' + }) + + assert response.status_code == 200 + assert b'User not found' in response.content + + +# ============================================================================ +# SECTION 5: Logs Endpoint Tests +# ============================================================================ + +@pytest.mark.django_db +class TestLogsEndpoints: + """Tests for the Logs view, LogsFilter, and LogsDownload (finding 4d)""" + + def test_logs_page_loads(self, authenticated_client): + """The Logs page should render successfully""" + response = authenticated_client.get('/Management/Logs/') + assert response.status_code == 200 + assert b'Log Entries' in response.content + + def test_logs_page_no_file_shows_empty(self, authenticated_client, settings, tmp_path): + """ + When the log file does not exist, the page should still render and + show zero entries rather than raising an error. + """ + # Point LOGS_DIR to a temp directory that has NO log file + settings.LOGS_DIR = tmp_path + response = authenticated_client.get('/Management/Logs/') + # Should still render (not 500) + assert response.status_code == 200 + + def test_logs_filter_returns_fragment(self, authenticated_client): + """LogsFilter should return an HTML fragment, not a full page""" + response = authenticated_client.get('/Management/Logs/filter') + assert response.status_code == 200 + # Should be an HTML fragment, not a full Django page with base template + assert b'' not in response.content + assert b'alert("xss") logged in\n', + encoding='utf-8' + ) + settings.LOGS_DIR = tmp_path + + response = authenticated_client.get('/Management/Logs/filter') + + assert response.status_code == 200 + content = response.content.decode() + # The raw " + mock_test_connectivity.return_value = (False, xss_payload) + + response = authenticated_client.get(f'/ConnectionManager/TestConnectivity?test={test_connection.id}') + + assert response.status_code == 200 + # Verify the script tag is escaped, not executed + content = response.content.decode('utf-8') + assert '<script>' in content + assert '", + "run_id": "test-run-xss", + "malicious_field": "'; DROP TABLE users; --" + } + + response = client.post( + '/ConnectionManager/StreamSimulate/', + data=json.dumps(event_data), + content_type='application/json' + ) + + assert response.status_code == 200 + + # Verify data is stored as-is (will be escaped when rendered) + with simulation_lock: + assert len(simulation_results) == 1 + stored_event = simulation_results[0] + # Data should be stored but will be escaped during rendering + assert stored_event['message'] == "" + + def test_stream_simulate_method_not_allowed(self, client): + """Test StreamSimulate with GET request""" + response = client.get('/ConnectionManager/StreamSimulate/') + + assert response.status_code == 405 + data = json.loads(response.content) + assert 'error' in data + + +# ============================================================================ +# GetSimulationResults Tests +# ============================================================================ + +@pytest.mark.django_db +class TestGetSimulationResults: + """Test GetSimulationResults view""" + + def test_get_simulation_results_success(self, authenticated_client): + """Test successful retrieval of simulation results""" + # Clear and populate queue + with simulation_lock: + simulation_results.clear() + simulation_results.append({ + "message": "event 1", + "run_id": "test-run-1" + }) + simulation_results.append({ + "message": "event 2", + "run_id": "test-run-1" + }) + simulation_results.append({ + "message": "event 3", + "run_id": "test-run-2" + }) + + response = authenticated_client.get('/ConnectionManager/GetSimulationResults/?run_id=test-run-1') + + assert response.status_code == 200 + data = json.loads(response.content) + assert 'results' in data + assert len(data['results']) == 2 + assert all(r['run_id'] == 'test-run-1' for r in data['results']) + + # Verify only run-1 events were removed from queue + with simulation_lock: + assert len(simulation_results) == 1 + assert simulation_results[0]['run_id'] == 'test-run-2' + + def test_get_simulation_results_no_run_id(self, authenticated_client): + """Test GetSimulationResults without run_id parameter""" + response = authenticated_client.get('/ConnectionManager/GetSimulationResults/') + + assert response.status_code == 400 + data = json.loads(response.content) + assert 'error' in data + assert 'run_id' in data['error'] + + def test_get_simulation_results_race_condition_safety(self, authenticated_client): + """Test GetSimulationResults handles concurrent access safely""" + # Populate queue + with simulation_lock: + simulation_results.clear() + for i in range(100): + simulation_results.append({ + "message": f"event {i}", + "run_id": "test-run-race" + }) + + # Make multiple concurrent-like requests + response1 = authenticated_client.get('/ConnectionManager/GetSimulationResults/?run_id=test-run-race') + response2 = authenticated_client.get('/ConnectionManager/GetSimulationResults/?run_id=test-run-race') + + assert response1.status_code == 200 + assert response2.status_code == 200 + + data1 = json.loads(response1.content) + data2 = json.loads(response2.content) + + # First request should get all events, second should get none + assert len(data1['results']) == 100 + assert len(data2['results']) == 0 + + def test_get_simulation_results_empty_queue(self, authenticated_client): + """Test GetSimulationResults when no results exist""" + with simulation_lock: + simulation_results.clear() + + response = authenticated_client.get('/ConnectionManager/GetSimulationResults/?run_id=nonexistent') + + assert response.status_code == 200 + data = json.loads(response.content) + assert data['results'] == [] + + +# ============================================================================ +# CheckIfPipelineLoaded Tests +# ============================================================================ + +@pytest.mark.django_db +class TestCheckIfPipelineLoaded: + """Test CheckIfPipelineLoaded view""" + + @patch('PipelineManager.simulation.requests.get') + def test_check_pipeline_loaded_running(self, mock_get, authenticated_client): + """Test checking a running pipeline""" + mock_response = Mock() + mock_response.status_code = 200 + mock_response.json.return_value = { + 'running_pipelines': ['slot1-filter1', 'slot2-filter1', 'main'] + } + mock_get.return_value = mock_response + + response = authenticated_client.get('/ConnectionManager/CheckIfPipelineLoaded/?pipeline_name=slot1-filter1') + + assert response.status_code == 200 + data = json.loads(response.content) + assert data['is_running'] is True + assert data['pipeline_name'] == 'slot1-filter1' + + @patch('PipelineManager.simulation.requests.get') + def test_check_pipeline_loaded_not_running(self, mock_get, authenticated_client): + """Test checking a non-running pipeline""" + mock_response = Mock() + mock_response.status_code = 200 + mock_response.json.return_value = { + 'running_pipelines': ['slot1-filter1', 'main'] + } + mock_get.return_value = mock_response + + response = authenticated_client.get('/ConnectionManager/CheckIfPipelineLoaded/?pipeline_name=slot2-filter1') + + assert response.status_code == 200 + data = json.loads(response.content) + assert data['is_running'] is False + assert data['pipeline_name'] == 'slot2-filter1' + + def test_check_pipeline_loaded_no_pipeline_name(self, authenticated_client): + """Test CheckIfPipelineLoaded without pipeline_name parameter""" + response = authenticated_client.get('/ConnectionManager/CheckIfPipelineLoaded/') + + assert response.status_code == 400 + data = json.loads(response.content) + assert 'error' in data + assert 'pipeline_name' in data['error'] + + @patch('PipelineManager.simulation.requests.get') + def test_check_pipeline_loaded_service_unavailable(self, mock_get, authenticated_client): + """Test CheckIfPipelineLoaded when logstashagent is unavailable""" + mock_get.side_effect = Exception("Connection refused") + + response = authenticated_client.get('/ConnectionManager/CheckIfPipelineLoaded/?pipeline_name=slot1-filter1') + + assert response.status_code == 500 + data = json.loads(response.content) + assert 'error' in data + assert data['is_running'] is False + + +# ============================================================================ +# GetRelatedLogs Tests +# ============================================================================ + +@pytest.mark.django_db +class TestGetRelatedLogs: + """Test GetRelatedLogs view""" + + @patch('PipelineManager.simulation.requests.get') + def test_get_related_logs_success(self, mock_get, authenticated_client): + """Test successful log retrieval""" + # Mock slots endpoint + mock_slots_response = Mock() + mock_slots_response.status_code = 200 + mock_slots_response.json.return_value = { + '1': { + 'created_at_millis': 1609459200000, + 'pipeline_name': 'slot1-filter1' + } + } + + # Mock logs endpoint + mock_logs_response = Mock() + mock_logs_response.status_code = 200 + mock_logs_response.json.return_value = { + 'pipeline_id': 'slot1-filter1', + 'log_count': 2, + 'logs': [ + {'level': 'INFO', 'message': 'Pipeline started', 'timeMillis': 1609459201000}, + {'level': 'DEBUG', 'message': 'Processing event', 'timeMillis': 1609459202000} + ] + } + + mock_get.side_effect = [mock_slots_response, mock_logs_response] + + response = authenticated_client.get('/ConnectionManager/GetRelatedLogs/?slot_id=1') + + assert response.status_code == 200 + data = json.loads(response.content) + assert data['log_count'] == 2 + assert len(data['logs']) == 2 + + def test_get_related_logs_no_slot_id(self, authenticated_client): + """Test GetRelatedLogs without slot_id parameter""" + response = authenticated_client.get('/ConnectionManager/GetRelatedLogs/') + + assert response.status_code == 400 + data = json.loads(response.content) + assert 'error' in data + assert 'slot_id' in data['error'] + + @patch('PipelineManager.simulation.requests.get') + def test_get_related_logs_with_filters(self, mock_get, authenticated_client): + """Test GetRelatedLogs with max_entries and min_level filters""" + mock_slots_response = Mock() + mock_slots_response.status_code = 200 + mock_slots_response.json.return_value = { + '1': {'created_at_millis': 1609459200000} + } + + mock_logs_response = Mock() + mock_logs_response.status_code = 200 + mock_logs_response.json.return_value = { + 'pipeline_id': 'slot1-filter1', + 'log_count': 1, + 'logs': [ + {'level': 'ERROR', 'message': 'Error occurred', 'timeMillis': 1609459201000} + ] + } + + mock_get.side_effect = [mock_slots_response, mock_logs_response] + + response = authenticated_client.get( + '/ConnectionManager/GetRelatedLogs/?slot_id=1&max_entries=50&min_level=ERROR' + ) + + assert response.status_code == 200 + data = json.loads(response.content) + assert data['log_count'] == 1 + + @patch('PipelineManager.simulation.requests.get') + def test_get_related_logs_service_unavailable(self, mock_get, authenticated_client): + """Test GetRelatedLogs when logstashagent is unavailable""" + mock_get.side_effect = Exception("Connection refused") + + response = authenticated_client.get('/ConnectionManager/GetRelatedLogs/?slot_id=1') + + assert response.status_code == 500 + data = json.loads(response.content) + assert 'error' in data + assert data['log_count'] == 0 + + +# ============================================================================ +# UploadFile Tests +# ============================================================================ + +@pytest.mark.django_db +class TestUploadFile: + """Test UploadFile view""" + + @patch('PipelineManager.simulation.requests.post') + def test_upload_file_success(self, mock_post, authenticated_client): + """Test successful file upload""" + mock_response = Mock() + mock_response.status_code = 200 + mock_response.json.return_value = {'status': 'ok'} + mock_post.return_value = mock_response + + file_content = b'test file content' + uploaded_file = SimpleUploadedFile("test.txt", file_content, content_type="text/plain") + + response = authenticated_client.post('/ConnectionManager/UploadFile/', { + 'file': uploaded_file, + 'filename': 'test.txt' + }) + + assert response.status_code == 200 + data = json.loads(response.content) + assert data['status'] == 'ok' + assert data['filename'] == 'test.txt' + + # Verify base64 encoding was used + mock_post.assert_called_once() + call_args = mock_post.call_args + posted_data = call_args[1]['json'] + assert 'content' in posted_data + assert 'filename' in posted_data + # Verify content is base64 encoded + decoded = base64.b64decode(posted_data['content']) + assert decoded == file_content + + def test_upload_file_no_file(self, authenticated_client): + """Test UploadFile with no file provided""" + response = authenticated_client.post('/ConnectionManager/UploadFile/', { + 'filename': 'test.txt' + }) + + assert response.status_code == 400 + data = json.loads(response.content) + assert 'error' in data + assert 'No file provided' in data['error'] + + def test_upload_file_no_filename(self, authenticated_client): + """Test UploadFile with no filename provided""" + file_content = b'test content' + uploaded_file = SimpleUploadedFile("test.txt", file_content) + + response = authenticated_client.post('/ConnectionManager/UploadFile/', { + 'file': uploaded_file + }) + + assert response.status_code == 400 + data = json.loads(response.content) + assert 'error' in data + assert 'No filename provided' in data['error'] + + @patch('PipelineManager.simulation.requests.post') + def test_upload_file_oversized(self, mock_post, authenticated_client): + """Test UploadFile with large file""" + # Create a 10MB file + large_content = b'x' * (10 * 1024 * 1024) + uploaded_file = SimpleUploadedFile("large.txt", large_content) + + mock_response = Mock() + mock_response.status_code = 200 + mock_post.return_value = mock_response + + response = authenticated_client.post('/ConnectionManager/UploadFile/', { + 'file': uploaded_file, + 'filename': 'large.txt' + }) + + # Should handle large files (or return appropriate error if size limit exists) + assert response.status_code in [200, 400, 413, 500] + + @patch('PipelineManager.simulation.requests.post') + def test_upload_file_agent_failure(self, mock_post, authenticated_client): + """Test UploadFile when logstashagent fails""" + mock_post.side_effect = Exception("Connection refused") + + file_content = b'test content' + uploaded_file = SimpleUploadedFile("test.txt", file_content) + + response = authenticated_client.post('/ConnectionManager/UploadFile/', { + 'file': uploaded_file, + 'filename': 'test.txt' + }) + + assert response.status_code == 500 + data = json.loads(response.content) + assert 'error' in data + # The actual error message is just the exception message + assert 'Connection refused' in data['error'] + + def test_upload_file_requires_admin(self, client): + """Test that UploadFile requires admin role""" + from django.contrib.auth.models import User + from Management.models import UserProfile + + readonly_user = User.objects.create_user( + username='readonly_upload', + password='testpass123', + is_staff=False + ) + readonly_user.profile.role = 'readonly' + readonly_user.profile.save() + client.login(username='readonly_upload', password='testpass123') + + file_content = b'test content' + uploaded_file = SimpleUploadedFile("test.txt", file_content) + + response = client.post('/ConnectionManager/UploadFile/', { + 'file': uploaded_file, + 'filename': 'test.txt' + }) + + assert response.status_code == 403 + + @patch('PipelineManager.simulation.requests.post') + def test_upload_file_binary_content(self, mock_post, authenticated_client): + """Test UploadFile with binary file content""" + mock_response = Mock() + mock_response.status_code = 200 + mock_post.return_value = mock_response + + # Binary content (e.g., image file) + binary_content = bytes([0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A]) + uploaded_file = SimpleUploadedFile("image.png", binary_content, content_type="image/png") + + response = authenticated_client.post('/ConnectionManager/UploadFile/', { + 'file': uploaded_file, + 'filename': 'image.png' + }) + + assert response.status_code == 200 + + # Verify binary content was properly encoded + call_args = mock_post.call_args + posted_data = call_args[1]['json'] + decoded = base64.b64decode(posted_data['content']) + assert decoded == binary_content diff --git a/tests/SNMP/__init__.py b/tests/SNMP/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/SNMP/data b/tests/SNMP/data new file mode 120000 index 0000000..51f1258 --- /dev/null +++ b/tests/SNMP/data @@ -0,0 +1 @@ +../../src/logstashui/SNMP/data \ No newline at end of file diff --git a/tests/SNMP/unit/__init__.py b/tests/SNMP/unit/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/SNMP/unit/test_commands.py b/tests/SNMP/unit/test_commands.py new file mode 100644 index 0000000..ec797c2 --- /dev/null +++ b/tests/SNMP/unit/test_commands.py @@ -0,0 +1,776 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +""" +Tests for the sync_snmp_official_data management command and the +sync_official_profiles / sync_official_device_templates snmp_crud helpers +it delegates to. + +Test categories: + - sync_official_profiles() individual function behaviour + - sync_official_device_templates() individual function behaviour + - call_command('sync_snmp_official_data') end-to-end command behaviour + - --cleanup path (delete / orphan stale official records) +""" + +import json +import os + +import pytest +from django.core.management import call_command +from io import StringIO + +from SNMP.snmp_crud import sync_official_profiles, sync_official_device_templates +from SNMP.models import Profile, DeviceTemplate + + +# --------------------------------------------------------------------------- +# Shared data +# --------------------------------------------------------------------------- + +MINIMAL_PROFILE = { + "official_key": "test_profile_key", + "description": "A test profile", + "vendor": "Generic", + "product": "", +} + +MINIMAL_TEMPLATE = { + "official_key": "test_template_key", + "name": "Test Template", + "description": "A test template", + "vendor": "Generic", + "model": "", + "product": "", + "matching_rules": [], + "profiles": [], +} + + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + +def _write_json(directory, filename, data): + with open(os.path.join(directory, filename), "w", encoding="utf-8") as f: + json.dump(data, f) + + +# --------------------------------------------------------------------------- +# Fixtures +# --------------------------------------------------------------------------- + +@pytest.fixture +def profile_dir(tmp_path, settings): + """ + Create the official_profiles directory and point settings.BASE_DIR to + the temp root. Tests that only exercise sync_official_profiles() use this. + """ + dirpath = tmp_path / "SNMP" / "data" / "official_profiles" + dirpath.mkdir(parents=True) + settings.BASE_DIR = str(tmp_path) + return str(dirpath) + + +@pytest.fixture +def template_dir(tmp_path, settings): + """ + Create BOTH data directories and point settings.BASE_DIR to the temp root. + Tests that exercise sync_official_device_templates() use this. + """ + (tmp_path / "SNMP" / "data" / "official_profiles").mkdir(parents=True) + dirpath = tmp_path / "SNMP" / "data" / "official_device_templates" + dirpath.mkdir(parents=True) + settings.BASE_DIR = str(tmp_path) + return str(dirpath) + + +@pytest.fixture +def both_dirs(tmp_path, settings): + """ + Creates both directories and returns (profile_dir_str, template_dir_str). + Used by command-level tests that exercise the full pipeline. + """ + p = tmp_path / "SNMP" / "data" / "official_profiles" + t = tmp_path / "SNMP" / "data" / "official_device_templates" + p.mkdir(parents=True) + t.mkdir(parents=True) + settings.BASE_DIR = str(tmp_path) + return str(p), str(t) + + +# --------------------------------------------------------------------------- +# sync_official_profiles — new record creation +# --------------------------------------------------------------------------- + +@pytest.mark.django_db +def test_sync_profiles_creates_new_profile(profile_dir): + _write_json(profile_dir, "test_profile.json", MINIMAL_PROFILE) + sync_official_profiles() + assert Profile.objects.filter(official_key="test_profile_key").exists() + + +@pytest.mark.django_db +def test_sync_profiles_stores_name_with_json_extension(profile_dir): + _write_json(profile_dir, "test_profile.json", MINIMAL_PROFILE) + sync_official_profiles() + profile = Profile.objects.get(official_key="test_profile_key") + assert profile.name == "test_profile.json" + + +@pytest.mark.django_db +def test_sync_profiles_sets_placeholder_flag(profile_dir): + _write_json(profile_dir, "test_profile.json", MINIMAL_PROFILE) + sync_official_profiles() + profile = Profile.objects.get(official_key="test_profile_key") + assert profile.profile_data == {"is_official_placeholder": True} + + +@pytest.mark.django_db +def test_sync_profiles_stores_vendor_and_product(profile_dir): + data = {**MINIMAL_PROFILE, "vendor": "Cisco", "product": "Catalyst"} + _write_json(profile_dir, "test_profile.json", data) + sync_official_profiles() + profile = Profile.objects.get(official_key="test_profile_key") + assert profile.vendor == "Cisco" + assert profile.product == "Catalyst" + + +# --------------------------------------------------------------------------- +# sync_official_profiles — update existing record +# --------------------------------------------------------------------------- + +@pytest.mark.django_db +def test_sync_profiles_updates_existing_record(profile_dir): + _write_json(profile_dir, "test_profile.json", MINIMAL_PROFILE) + sync_official_profiles() + + updated = {**MINIMAL_PROFILE, "description": "Updated description", "vendor": "Cisco"} + _write_json(profile_dir, "test_profile.json", updated) + sync_official_profiles() + + profile = Profile.objects.get(official_key="test_profile_key") + assert profile.description == "Updated description" + assert profile.vendor == "Cisco" + assert Profile.objects.filter(official_key="test_profile_key").count() == 1 + + +@pytest.mark.django_db +def test_sync_profiles_idempotent(profile_dir): + _write_json(profile_dir, "test_profile.json", MINIMAL_PROFILE) + sync_official_profiles() + sync_official_profiles() + assert Profile.objects.filter(official_key="test_profile_key").count() == 1 + + +# --------------------------------------------------------------------------- +# sync_official_profiles — fix #1: is_orphaned cleared on restoration +# --------------------------------------------------------------------------- + +@pytest.mark.django_db +def test_sync_profiles_clears_is_orphaned_on_restoration(profile_dir): + """ + Regression test for fix #1. + A profile previously marked is_orphaned=True must have the flag removed + the next time its backing JSON is present during sync. + """ + _write_json(profile_dir, "test_profile.json", MINIMAL_PROFILE) + sync_official_profiles() + + # Simulate cleanup having orphaned the profile + profile = Profile.objects.get(official_key="test_profile_key") + profile.profile_data = {"is_official_placeholder": True, "is_orphaned": True} + profile.save() + assert profile.profile_data.get("is_orphaned") is True + + # Re-sync with the JSON still present + sync_official_profiles() + profile.refresh_from_db() + + assert "is_orphaned" not in profile.profile_data + assert profile.profile_data == {"is_official_placeholder": True} + + +@pytest.mark.django_db +def test_sync_profiles_clears_arbitrary_stale_flags(profile_dir): + """profile_data is always reset to a clean placeholder on sync.""" + _write_json(profile_dir, "test_profile.json", MINIMAL_PROFILE) + sync_official_profiles() + + profile = Profile.objects.get(official_key="test_profile_key") + profile.profile_data = {"is_official_placeholder": True, "custom_flag": "leftover"} + profile.save() + + sync_official_profiles() + profile.refresh_from_db() + assert profile.profile_data == {"is_official_placeholder": True} + + +# --------------------------------------------------------------------------- +# sync_official_profiles — fix #6: missing vendor defaults to 'Any' +# --------------------------------------------------------------------------- + +@pytest.mark.django_db +def test_sync_profiles_missing_vendor_defaults_to_any(profile_dir): + """ + Regression test for fix #6. + A JSON file that omits the vendor key must still produce a DB record + (vendor='Any') rather than being silently skipped by a full_clean() failure. + """ + no_vendor = {k: v for k, v in MINIMAL_PROFILE.items() if k != "vendor"} + _write_json(profile_dir, "no_vendor.json", no_vendor) + sync_official_profiles() + + profile = Profile.objects.get(official_key="test_profile_key") + assert profile.vendor == "Any" + + +# --------------------------------------------------------------------------- +# sync_official_profiles — skip / guard cases +# --------------------------------------------------------------------------- + +@pytest.mark.django_db +def test_sync_profiles_skips_file_without_official_key(profile_dir): + no_key = {k: v for k, v in MINIMAL_PROFILE.items() if k != "official_key"} + _write_json(profile_dir, "no_key.json", no_key) + sync_official_profiles() + assert Profile.objects.count() == 0 + + +@pytest.mark.django_db +def test_sync_profiles_ignores_non_json_files(profile_dir): + with open(os.path.join(profile_dir, "readme.txt"), "w") as f: + f.write("not a profile") + sync_official_profiles() + assert Profile.objects.count() == 0 + + +@pytest.mark.django_db +def test_sync_profiles_handles_empty_directory(profile_dir): + sync_official_profiles() + assert Profile.objects.count() == 0 + + +@pytest.mark.django_db +def test_sync_profiles_handles_malformed_json_gracefully(profile_dir): + with open(os.path.join(profile_dir, "bad.json"), "w") as f: + f.write("{ not valid json }") + sync_official_profiles() + assert Profile.objects.count() == 0 + + +@pytest.mark.django_db +def test_sync_profiles_handles_multiple_files(profile_dir): + for i in range(5): + _write_json(profile_dir, f"profile_{i}.json", { + **MINIMAL_PROFILE, + "official_key": f"key_{i}", + }) + sync_official_profiles() + assert Profile.objects.filter(official_key__startswith="key_").count() == 5 + + +# --------------------------------------------------------------------------- +# sync_official_profiles — legacy backfill path +# --------------------------------------------------------------------------- + +@pytest.mark.django_db +def test_sync_profiles_backfills_official_key_for_legacy_record(profile_dir): + """ + A pre-existing DB record that lacks official_key should have it backfilled + when a JSON file with the matching name is found. + """ + legacy = Profile.objects.create( + name="legacy_profile.json", + official_key=None, + vendor="Generic", + profile_data={"is_official_placeholder": True}, + ) + _write_json(profile_dir, "legacy_profile.json", { + **MINIMAL_PROFILE, + "official_key": "legacy_key", + }) + sync_official_profiles() + + legacy.refresh_from_db() + assert legacy.official_key == "legacy_key" + assert Profile.objects.filter(official_key="legacy_key").count() == 1 + + +# --------------------------------------------------------------------------- +# sync_official_device_templates — new record creation +# --------------------------------------------------------------------------- + +@pytest.mark.django_db +def test_sync_templates_creates_new_template(template_dir): + _write_json(template_dir, "test_template.json", MINIMAL_TEMPLATE) + sync_official_device_templates() + assert DeviceTemplate.objects.filter(official_key="test_template_key").exists() + + +@pytest.mark.django_db +def test_sync_templates_is_marked_official(template_dir): + _write_json(template_dir, "test_template.json", MINIMAL_TEMPLATE) + sync_official_device_templates() + template = DeviceTemplate.objects.get(official_key="test_template_key") + assert template.official is True + + +@pytest.mark.django_db +def test_sync_templates_uses_name_field_not_filename(template_dir): + """The 'name' key in JSON is used as the DB name, not the filename stem.""" + data = {**MINIMAL_TEMPLATE, "name": "My Custom Name"} + _write_json(template_dir, "file_name_irrelevant.json", data) + sync_official_device_templates() + assert DeviceTemplate.objects.filter(name="My Custom Name").exists() + + +@pytest.mark.django_db +def test_sync_templates_stores_vendor_model_product(template_dir): + data = {**MINIMAL_TEMPLATE, "vendor": "Dell", "model": "PowerEdge", "product": "iDRAC"} + _write_json(template_dir, "test_template.json", data) + sync_official_device_templates() + template = DeviceTemplate.objects.get(official_key="test_template_key") + assert template.vendor == "Dell" + assert template.model == "PowerEdge" + assert template.product == "iDRAC" + + +# --------------------------------------------------------------------------- +# sync_official_device_templates — update existing record +# --------------------------------------------------------------------------- + +@pytest.mark.django_db +def test_sync_templates_updates_existing_record(template_dir): + _write_json(template_dir, "test_template.json", MINIMAL_TEMPLATE) + sync_official_device_templates() + + updated = {**MINIMAL_TEMPLATE, "description": "Updated", "vendor": "Cisco"} + _write_json(template_dir, "test_template.json", updated) + sync_official_device_templates() + + template = DeviceTemplate.objects.get(official_key="test_template_key") + assert template.description == "Updated" + assert template.vendor == "Cisco" + assert DeviceTemplate.objects.filter(official_key="test_template_key").count() == 1 + + +@pytest.mark.django_db +def test_sync_templates_idempotent(template_dir): + _write_json(template_dir, "test_template.json", MINIMAL_TEMPLATE) + sync_official_device_templates() + sync_official_device_templates() + assert DeviceTemplate.objects.filter(official_key="test_template_key").count() == 1 + + +# --------------------------------------------------------------------------- +# sync_official_device_templates — profile linking (three lookup paths) +# --------------------------------------------------------------------------- + +@pytest.mark.django_db +def test_sync_templates_links_profiles_via_official_key(tmp_path, settings): + """Primary path: profile resolved by its official_key value.""" + (tmp_path / "SNMP" / "data" / "official_profiles").mkdir(parents=True) + tdir = tmp_path / "SNMP" / "data" / "official_device_templates" + tdir.mkdir(parents=True) + settings.BASE_DIR = str(tmp_path) + + profile = Profile.objects.create( + official_key="linked_profile_key", + name="linked_profile.json", + vendor="Generic", + profile_data={"is_official_placeholder": True}, + ) + data = {**MINIMAL_TEMPLATE, "profiles": ["linked_profile_key"]} + _write_json(str(tdir), "test_template.json", data) + sync_official_device_templates() + + template = DeviceTemplate.objects.get(official_key="test_template_key") + assert profile in template.profiles.all() + + +@pytest.mark.django_db +def test_sync_templates_links_profiles_via_json_name_fallback(tmp_path, settings): + """ + Fallback path: profile referenced without .json extension is found by + appending .json to the lookup name (un-migrated official profile). + """ + (tmp_path / "SNMP" / "data" / "official_profiles").mkdir(parents=True) + tdir = tmp_path / "SNMP" / "data" / "official_device_templates" + tdir.mkdir(parents=True) + settings.BASE_DIR = str(tmp_path) + + profile = Profile.objects.create( + official_key=None, + name="linked_profile.json", + vendor="Generic", + profile_data={"is_official_placeholder": True}, + ) + data = {**MINIMAL_TEMPLATE, "profiles": ["linked_profile"]} + _write_json(str(tdir), "test_template.json", data) + sync_official_device_templates() + + template = DeviceTemplate.objects.get(official_key="test_template_key") + assert profile in template.profiles.all() + + +@pytest.mark.django_db +def test_sync_templates_links_profiles_via_bare_name_fallback(tmp_path, settings): + """ + Fallback path: user-created custom profile matched by exact bare name. + """ + (tmp_path / "SNMP" / "data" / "official_profiles").mkdir(parents=True) + tdir = tmp_path / "SNMP" / "data" / "official_device_templates" + tdir.mkdir(parents=True) + settings.BASE_DIR = str(tmp_path) + + profile = Profile.objects.create( + official_key=None, + name="custom_profile", + vendor="Generic", + profile_data={"get": {}, "walk": {}, "table": {}}, + ) + data = {**MINIMAL_TEMPLATE, "profiles": ["custom_profile"]} + _write_json(str(tdir), "test_template.json", data) + sync_official_device_templates() + + template = DeviceTemplate.objects.get(official_key="test_template_key") + assert profile in template.profiles.all() + + +@pytest.mark.django_db +def test_sync_templates_links_multiple_profiles(tmp_path, settings): + (tmp_path / "SNMP" / "data" / "official_profiles").mkdir(parents=True) + tdir = tmp_path / "SNMP" / "data" / "official_device_templates" + tdir.mkdir(parents=True) + settings.BASE_DIR = str(tmp_path) + + profiles = [] + for i in range(3): + p = Profile.objects.create( + official_key=f"profile_key_{i}", + name=f"profile_{i}.json", + vendor="Generic", + profile_data={"is_official_placeholder": True}, + ) + profiles.append(p) + + data = {**MINIMAL_TEMPLATE, "profiles": [f"profile_key_{i}" for i in range(3)]} + _write_json(str(tdir), "test_template.json", data) + sync_official_device_templates() + + template = DeviceTemplate.objects.get(official_key="test_template_key") + assert template.profiles.count() == 3 + + +# --------------------------------------------------------------------------- +# sync_official_device_templates — profile linking edge cases +# --------------------------------------------------------------------------- + +@pytest.mark.django_db +def test_sync_templates_missing_profile_skipped_gracefully(template_dir): + """ + A profile name listed in the JSON that doesn't exist in the DB should be + silently skipped. The template itself must still be created. + """ + data = {**MINIMAL_TEMPLATE, "profiles": ["nonexistent_profile"]} + _write_json(template_dir, "test_template.json", data) + sync_official_device_templates() + + template = DeviceTemplate.objects.get(official_key="test_template_key") + assert template.profiles.count() == 0 + + +@pytest.mark.django_db +def test_sync_templates_empty_profiles_list_does_not_clear_existing(tmp_path, settings): + """ + profiles: [] in JSON is treated as a no-op — existing M2M rows must not + be cleared. This is the current documented behaviour of the if-guard. + """ + (tmp_path / "SNMP" / "data" / "official_profiles").mkdir(parents=True) + tdir = tmp_path / "SNMP" / "data" / "official_device_templates" + tdir.mkdir(parents=True) + settings.BASE_DIR = str(tmp_path) + + profile = Profile.objects.create( + official_key="kept_profile", + name="kept_profile.json", + vendor="Generic", + profile_data={"is_official_placeholder": True}, + ) + template = DeviceTemplate.objects.create( + official_key="test_template_key", + name="Test Template", + vendor="Generic", + official=True, + ) + template.profiles.add(profile) + assert template.profiles.count() == 1 + + _write_json(str(tdir), "test_template.json", {**MINIMAL_TEMPLATE, "profiles": []}) + sync_official_device_templates() + + template.refresh_from_db() + assert template.profiles.count() == 1 + + +# --------------------------------------------------------------------------- +# sync_official_device_templates — fix #6 and skip/guard cases +# --------------------------------------------------------------------------- + +@pytest.mark.django_db +def test_sync_templates_missing_vendor_defaults_to_any(template_dir): + """Regression test for fix #6 on the template sync path.""" + no_vendor = {k: v for k, v in MINIMAL_TEMPLATE.items() if k != "vendor"} + _write_json(template_dir, "no_vendor.json", no_vendor) + sync_official_device_templates() + + template = DeviceTemplate.objects.get(official_key="test_template_key") + assert template.vendor == "Any" + + +@pytest.mark.django_db +def test_sync_templates_skips_file_without_official_key(template_dir): + no_key = {k: v for k, v in MINIMAL_TEMPLATE.items() if k != "official_key"} + _write_json(template_dir, "no_key.json", no_key) + sync_official_device_templates() + assert DeviceTemplate.objects.count() == 0 + + +@pytest.mark.django_db +def test_sync_templates_handles_malformed_json_gracefully(template_dir): + with open(os.path.join(template_dir, "bad.json"), "w") as f: + f.write("{ not valid json }") + sync_official_device_templates() + assert DeviceTemplate.objects.count() == 0 + + +@pytest.mark.django_db +def test_sync_templates_handles_empty_directory(template_dir): + sync_official_device_templates() + assert DeviceTemplate.objects.count() == 0 + + +# --------------------------------------------------------------------------- +# Management command — call_command end-to-end +# --------------------------------------------------------------------------- + +@pytest.mark.django_db +def test_command_syncs_profiles_and_templates(both_dirs): + profile_dir, template_dir = both_dirs + _write_json(profile_dir, "p.json", {**MINIMAL_PROFILE, "official_key": "cmd_profile_key"}) + _write_json(template_dir, "t.json", {**MINIMAL_TEMPLATE, "official_key": "cmd_template_key"}) + + call_command("sync_snmp_official_data", stdout=StringIO()) + + assert Profile.objects.filter(official_key="cmd_profile_key").exists() + assert DeviceTemplate.objects.filter(official_key="cmd_template_key").exists() + + +@pytest.mark.django_db +def test_command_output_mentions_profiles_and_templates(both_dirs): + profile_dir, _ = both_dirs + _write_json(profile_dir, "p.json", {**MINIMAL_PROFILE, "official_key": "out_key"}) + + out = StringIO() + call_command("sync_snmp_official_data", stdout=out) + output = out.getvalue().lower() + + assert "profile" in output + assert "template" in output + + +@pytest.mark.django_db +def test_command_does_not_raise_on_malformed_json(both_dirs): + """A corrupt JSON file must not abort startup.""" + profile_dir, _ = both_dirs + with open(os.path.join(profile_dir, "bad.json"), "w") as f: + f.write("{ not valid json }") + + out = StringIO() + call_command("sync_snmp_official_data", stdout=out) + + +# --------------------------------------------------------------------------- +# Management command — --cleanup: stale-by-official_key records +# --------------------------------------------------------------------------- + +@pytest.mark.django_db +def test_command_without_cleanup_leaves_stale_records(both_dirs): + """Without --cleanup, stale official records must persist in the DB.""" + _, _ = both_dirs + Profile.objects.create( + official_key="stale_key", + name="stale.json", + vendor="Any", + profile_data={"is_official_placeholder": True}, + ) + + call_command("sync_snmp_official_data", stdout=StringIO()) + assert Profile.objects.filter(official_key="stale_key").exists() + + +@pytest.mark.django_db +def test_command_cleanup_deletes_unused_stale_profile(both_dirs): + """Stale official profile with no DeviceTemplate referencing it is deleted.""" + _, _ = both_dirs + Profile.objects.create( + official_key="stale_key", + name="stale.json", + vendor="Any", + profile_data={"is_official_placeholder": True}, + ) + + call_command("sync_snmp_official_data", cleanup=True, stdout=StringIO()) + assert not Profile.objects.filter(official_key="stale_key").exists() + + +@pytest.mark.django_db +def test_command_cleanup_orphans_in_use_stale_profile(both_dirs): + """ + Stale official profile referenced by a DeviceTemplate must be marked + is_orphaned=True instead of deleted. + """ + _, _ = both_dirs + stale_profile = Profile.objects.create( + official_key="stale_in_use_key", + name="stale_in_use.json", + vendor="Any", + profile_data={"is_official_placeholder": True}, + ) + using_template = DeviceTemplate.objects.create( + name="Uses Stale Profile", + vendor="Generic", + official=False, + ) + using_template.profiles.add(stale_profile) + + call_command("sync_snmp_official_data", cleanup=True, stdout=StringIO()) + + stale_profile.refresh_from_db() + assert Profile.objects.filter(official_key="stale_in_use_key").exists() + assert stale_profile.profile_data.get("is_orphaned") is True + + +@pytest.mark.django_db +def test_command_cleanup_deletes_unused_stale_template(both_dirs): + """Stale official template with no devices assigned is deleted.""" + _, _ = both_dirs + DeviceTemplate.objects.create( + official_key="stale_template_key", + name="Stale Template", + vendor="Any", + official=True, + ) + + call_command("sync_snmp_official_data", cleanup=True, stdout=StringIO()) + assert not DeviceTemplate.objects.filter(official_key="stale_template_key").exists() + + +@pytest.mark.django_db +def test_command_cleanup_does_not_delete_in_use_stale_template(both_dirs): + """ + Stale official template with devices still assigned must be kept. + The command should log a warning but not delete it. + """ + from SNMP.models import Credential, Network, Device + from PipelineManager.models import Connection + + _, _ = both_dirs + stale_template = DeviceTemplate.objects.create( + official_key="stale_in_use_template", + name="Stale In Use", + vendor="Any", + official=True, + ) + + conn = Connection.objects.create( + name="Test Conn", + connection_type="CENTRALIZED", + host="https://localhost:9200", + username="elastic", + password="changeme", + ) + cred = Credential.objects.create(name="cred", version="2c", community="public") + net = Network.objects.create( + name="net", + network_range="10.0.0.0/24", + connection=conn, + discovery_credential=cred, + interval=30, + ) + Device.objects.create( + name="test_device", + ip_address="10.0.0.1", + port=161, + retries=2, + timeout=1000, + credential=cred, + network=net, + device_template=stale_template, + ) + + call_command("sync_snmp_official_data", cleanup=True, stdout=StringIO()) + assert DeviceTemplate.objects.filter(official_key="stale_in_use_template").exists() + + +# --------------------------------------------------------------------------- +# Management command — --cleanup: legacy (stale-by-flag) records +# --------------------------------------------------------------------------- + +@pytest.mark.django_db +def test_command_cleanup_deletes_legacy_stale_profile(both_dirs): + """ + Old-style official profiles (no official_key, has is_official_placeholder) + that were not backfilled during sync are treated as stale and deleted. + """ + _, _ = both_dirs + legacy = Profile.objects.create( + official_key=None, + name="legacy_no_key.json", + vendor="Any", + profile_data={"is_official_placeholder": True}, + ) + + call_command("sync_snmp_official_data", cleanup=True, stdout=StringIO()) + assert not Profile.objects.filter(pk=legacy.pk).exists() + + +@pytest.mark.django_db +def test_command_cleanup_deletes_legacy_stale_template(both_dirs): + """ + Official templates with no official_key after sync ran are stale and + must be deleted when not in use. + """ + _, _ = both_dirs + legacy = DeviceTemplate.objects.create( + official_key=None, + name="Legacy Template", + vendor="Any", + official=True, + ) + + call_command("sync_snmp_official_data", cleanup=True, stdout=StringIO()) + assert not DeviceTemplate.objects.filter(pk=legacy.pk).exists() + + +# --------------------------------------------------------------------------- +# Management command — --cleanup: output counts +# --------------------------------------------------------------------------- + +@pytest.mark.django_db +def test_command_cleanup_output_reports_deleted_counts(both_dirs): + _, _ = both_dirs + Profile.objects.create( + official_key="deleted_profile", + name="deleted.json", + vendor="Any", + profile_data={"is_official_placeholder": True}, + ) + + out = StringIO() + call_command("sync_snmp_official_data", cleanup=True, stdout=out) + output = out.getvalue().lower() + + assert "deleted" in output or "profile" in output diff --git a/tests/SNMP/unit/test_inline_grounding.py b/tests/SNMP/unit/test_inline_grounding.py new file mode 100644 index 0000000..6b121d8 --- /dev/null +++ b/tests/SNMP/unit/test_inline_grounding.py @@ -0,0 +1,66 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. +""" +Unit tests for SNMP/inline_grounding.py. + +No network / no live LLM: a temp data dir is used so the selection logic is +deterministic. +""" +import json +import os +import tempfile +from unittest import mock + +from django.test import SimpleTestCase + +from SNMP import inline_grounding as ig + + +def _seed(tmp): + for sub in ("official_profiles", "schema_reference", "mib_reference"): + os.makedirs(os.path.join(tmp, sub)) + with open(os.path.join(tmp, "authoring_instructions.md"), "w") as f: + f.write("AUTHORING RULES") + with open(os.path.join(tmp, "schema_reference", "s.md"), "w") as f: + f.write("NAMING DICT") + with open(os.path.join(tmp, "mib_reference", "std_x.json"), "w") as f: + json.dump({"name": "std_x"}, f) + for name, vendor in [("generic_interfaces", "Any"), ("cisco_x", "Cisco"), ("arista_x", "Arista")]: + with open(os.path.join(tmp, "official_profiles", f"{name}.json"), "w") as f: + json.dump({"name": name, "vendor": vendor, "get": {"o": "1"}}, f) + + +class InlineGroundingTests(SimpleTestCase): + def test_relevance_rules(self): + self.assertTrue(ig._relevant({"vendor": "Any"}, "Arista")) + self.assertTrue(ig._relevant({"vendor": ""}, "whatever")) + self.assertTrue(ig._relevant({"vendor": "Arista"}, "Arista Networks EOS")) + self.assertFalse(ig._relevant({"vendor": "Cisco"}, "Arista")) + self.assertFalse(ig._relevant({"vendor": "Cisco"}, "")) + + def test_grounding_includes_generic_and_vendor_match_only(self): + with tempfile.TemporaryDirectory() as tmp: + _seed(tmp) + with mock.patch.object(ig, "_DATA", tmp): + g = ig.build_grounding("Arista Networks EOS 4.36") + self.assertIn("NAMING DICT", g) + self.assertIn("std_x", g) + self.assertIn("generic_interfaces", g) + self.assertIn("arista_x", g) + self.assertNotIn("cisco_x", g) + + def test_grounding_has_all_sections(self): + with tempfile.TemporaryDirectory() as tmp: + _seed(tmp) + with mock.patch.object(ig, "_DATA", tmp): + g = ig.build_grounding("Any") + self.assertIn("FIELD NAMING SCHEMA", g) + self.assertIn("STANDARD-MIB REFERENCES", g) + self.assertIn("REFERENCE PROFILES", g) + + def test_load_instructions_reads_local_file(self): + with tempfile.TemporaryDirectory() as tmp: + _seed(tmp) + with mock.patch.object(ig, "_DATA", tmp): + self.assertEqual(ig.load_instructions(), "AUTHORING RULES") diff --git a/tests/SNMP/unit/test_models.py b/tests/SNMP/unit/test_models.py new file mode 100644 index 0000000..7896c35 --- /dev/null +++ b/tests/SNMP/unit/test_models.py @@ -0,0 +1,498 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +import pytest +from django.core.exceptions import ValidationError +from django.utils import timezone + +from SNMP.models import ( + Credential, Device, DeviceTemplate, Network, Profile, SNMPDeploymentState +) +from PipelineManager.models import Connection + + +# --------------------------------------------------------------------------- +# Shared fixtures +# --------------------------------------------------------------------------- + +@pytest.fixture +def test_connection(db): + return Connection.objects.create( + name='Test Connection', + connection_type='CENTRALIZED', + host='https://localhost:9200', + username='elastic', + password='changeme', + ) + + +@pytest.fixture +def test_credential_v2c(db): + return Credential.objects.create( + name='Cred v2c', + version='2c', + community='public', + ) + + +@pytest.fixture +def test_network(db, test_connection, test_credential_v2c): + return Network.objects.create( + name='Net', + network_range='10.0.0.0/8', + connection=test_connection, + ) + + +@pytest.fixture +def default_template(db): + return DeviceTemplate.objects.create( + name='default', + vendor='Any', + official=True, + ) + + +# =========================================================================== +# SNMPDeploymentState +# =========================================================================== + +@pytest.mark.django_db +class TestSNMPDeploymentState: + + def test_mark_config_changed_creates_state(self): + """mark_config_changed creates the singleton row when it doesn't exist.""" + SNMPDeploymentState.objects.all().delete() + SNMPDeploymentState.mark_config_changed() + state = SNMPDeploymentState.objects.get(id=1) + assert state.last_config_change is not None + + def test_mark_config_changed_updates_timestamp(self): + """Successive calls to mark_config_changed advance the timestamp.""" + SNMPDeploymentState.objects.all().delete() + SNMPDeploymentState.mark_config_changed() + first = SNMPDeploymentState.objects.get(id=1).last_config_change + SNMPDeploymentState.mark_config_changed() + second = SNMPDeploymentState.objects.get(id=1).last_config_change + assert second >= first + + def test_has_undeployed_changes_no_state(self): + """has_undeployed_changes returns True when no row exists (never deployed).""" + SNMPDeploymentState.objects.all().delete() + assert SNMPDeploymentState.has_undeployed_changes() is True + + def test_has_undeployed_changes_no_deployment(self): + """has_undeployed_changes is True when config changed but never deployed.""" + SNMPDeploymentState.objects.all().delete() + SNMPDeploymentState.mark_config_changed() + assert SNMPDeploymentState.has_undeployed_changes() is True + + def test_has_undeployed_changes_after_sync(self): + """has_undeployed_changes is False when last_deployment >= last_config_change.""" + SNMPDeploymentState.objects.all().delete() + now = timezone.now() + SNMPDeploymentState.objects.create( + id=1, + last_config_change=now, + last_deployment=now, + ) + assert SNMPDeploymentState.has_undeployed_changes() is False + + def test_has_undeployed_changes_after_new_change(self): + """has_undeployed_changes is True again when config changes after deployment.""" + from datetime import timedelta + SNMPDeploymentState.objects.all().delete() + # Seed the state with timestamps 60 seconds in the past so mark_config_changed + # will produce a strictly later timestamp. + past = timezone.now() - timedelta(seconds=60) + SNMPDeploymentState.objects.create( + id=1, + last_config_change=past, + last_deployment=past, + ) + SNMPDeploymentState.mark_config_changed() + assert SNMPDeploymentState.has_undeployed_changes() is True + + def test_str_never_deployed(self): + """__str__ returns 'Never deployed' when last_deployment is None.""" + SNMPDeploymentState.objects.all().delete() + state = SNMPDeploymentState.objects.create(id=1) + assert str(state) == 'Never deployed' + + def test_str_with_deployment(self): + """__str__ includes the timestamp when last_deployment is set.""" + SNMPDeploymentState.objects.all().delete() + now = timezone.now() + state = SNMPDeploymentState.objects.create(id=1, last_deployment=now) + assert 'Last deployed' in str(state) + + +# =========================================================================== +# DeviceTemplate.matches_device +# =========================================================================== + +@pytest.mark.django_db +class TestDeviceTemplateMatchesDevice: + + def test_matches_with_all_rules(self): + """matches_device returns True when all rules appear in device_info.""" + tmpl = DeviceTemplate.objects.create( + name='cisco_switch', + vendor='Cisco', + matching_rules=['cisco', 'catalyst'], + ) + assert tmpl.matches_device('Cisco Catalyst 9300 switch') is True + + def test_no_match_when_rule_absent(self): + """matches_device returns False when a rule is not found.""" + tmpl = DeviceTemplate.objects.create( + name='cisco_switch2', + vendor='Cisco', + matching_rules=['cisco', 'catalyst'], + ) + assert tmpl.matches_device('Juniper EX2300') is False + + def test_case_insensitive(self): + """matches_device is case-insensitive.""" + tmpl = DeviceTemplate.objects.create( + name='cisco_switch3', + vendor='Cisco', + matching_rules=['CISCO'], + ) + assert tmpl.matches_device('cisco ios') is True + + def test_empty_matching_rules(self): + """matches_device returns False when matching_rules is empty.""" + tmpl = DeviceTemplate.objects.create( + name='generic_tmpl', + vendor='Generic', + matching_rules=[], + ) + assert tmpl.matches_device('anything') is False + + def test_empty_device_info(self): + """matches_device returns False when device_info is empty/None.""" + tmpl = DeviceTemplate.objects.create( + name='cisco_switch4', + vendor='Cisco', + matching_rules=['cisco'], + ) + assert tmpl.matches_device('') is False + assert tmpl.matches_device(None) is False + + def test_partial_substring_match(self): + """A single matching rule appearing in a longer string is sufficient.""" + tmpl = DeviceTemplate.objects.create( + name='dell_idrac', + vendor='Dell', + matching_rules=['idrac'], + ) + assert tmpl.matches_device('Dell iDRAC 9 server') is True + + +# =========================================================================== +# DeviceTemplate.clean – matching_rules validation +# =========================================================================== + +@pytest.mark.django_db +class TestDeviceTemplateClean: + + def test_matching_rules_must_be_list(self): + """matching_rules must be a list; a dict raises ValidationError.""" + with pytest.raises(ValidationError): + DeviceTemplate.objects.create( + name='bad_rules_dict', + vendor='Any', + matching_rules={'key': 'value'}, + ) + + def test_matching_rules_items_must_be_strings(self): + """Each item in matching_rules must be a string.""" + with pytest.raises(ValidationError): + DeviceTemplate.objects.create( + name='bad_rules_int', + vendor='Any', + matching_rules=[1, 2, 3], + ) + + def test_empty_matching_rules_valid(self): + """An empty list is a valid matching_rules value.""" + tmpl = DeviceTemplate.objects.create( + name='empty_rules_ok', + vendor='Any', + matching_rules=[], + ) + assert tmpl.id is not None + + +# =========================================================================== +# Credential – decryption helpers +# =========================================================================== + +@pytest.mark.django_db +class TestCredentialDecryption: + + def test_get_community_returns_plaintext(self): + """get_community() decrypts and returns the community string.""" + cred = Credential.objects.create( + name='comm_test', + version='2c', + community='secret_community', + ) + cred.refresh_from_db() + assert cred.get_community() == 'secret_community' + + def test_get_community_none_when_blank(self): + """get_community() returns None when community is blank.""" + cred = Credential.objects.create( + name='comm_blank', + version='2c', + community='placeholder', # must pass model clean + ) + # Manually blank out after creation to avoid validation + Credential.objects.filter(pk=cred.pk).update(community='') + cred.refresh_from_db() + assert cred.get_community() is None + + def test_get_auth_pass_returns_plaintext(self): + """get_auth_pass() decrypts and returns the auth passphrase.""" + cred = Credential.objects.create( + name='auth_test', + version='3', + security_name='user1', + security_level='authPriv', + auth_protocol='sha', + auth_pass='authsecret', + priv_protocol='aes', + priv_pass='privsecret', + ) + cred.refresh_from_db() + assert cred.get_auth_pass() == 'authsecret' + + def test_get_priv_pass_returns_plaintext(self): + """get_priv_pass() decrypts and returns the priv passphrase.""" + cred = Credential.objects.create( + name='priv_test', + version='3', + security_name='user2', + security_level='authPriv', + auth_protocol='sha', + auth_pass='authsecret2', + priv_protocol='aes', + priv_pass='privsecret2', + ) + cred.refresh_from_db() + assert cred.get_priv_pass() == 'privsecret2' + + def test_get_auth_pass_none_when_blank(self): + """get_auth_pass() returns None when auth_pass is blank.""" + cred = Credential.objects.create( + name='no_auth_pass', + version='3', + security_name='user3', + security_level='noAuthNoPriv', + ) + cred.refresh_from_db() + assert cred.get_auth_pass() is None + + def test_double_save_does_not_double_encrypt(self): + """Saving a credential twice does not encrypt an already-encrypted value.""" + cred = Credential.objects.create( + name='double_save_test', + version='2c', + community='test_community', + ) + cred.refresh_from_db() + first_community = cred.community # encrypted token + cred.description = 'Updated' + cred.save() + cred.refresh_from_db() + assert cred.community == first_community + assert cred.get_community() == 'test_community' + + +# =========================================================================== +# Credential.clean – SNMP version validation +# =========================================================================== + +@pytest.mark.django_db +class TestCredentialClean: + + def test_v2c_requires_community(self): + """v2c credential requires a community string.""" + with pytest.raises(ValidationError): + cred = Credential(name='no_comm', version='2c', community='') + cred.full_clean() + + def test_v3_noauthnopriv_rejects_auth_fields(self): + """noAuthNoPriv should not have auth/priv fields set.""" + with pytest.raises(ValidationError): + Credential.objects.create( + name='bad_noauth', + version='3', + security_name='user', + security_level='noAuthNoPriv', + auth_protocol='sha', + auth_pass='pass', + ) + + def test_v3_authnopriv_requires_auth_protocol(self): + """authNoPriv requires auth_protocol.""" + with pytest.raises(ValidationError): + Credential.objects.create( + name='bad_authnopriv', + version='3', + security_name='user', + security_level='authNoPriv', + auth_protocol='', + auth_pass='pass', + ) + + def test_v3_authnopriv_rejects_priv_fields(self): + """authNoPriv must not have priv fields set.""" + with pytest.raises(ValidationError): + Credential.objects.create( + name='bad_priv', + version='3', + security_name='user', + security_level='authNoPriv', + auth_protocol='sha', + auth_pass='pass', + priv_protocol='aes', + ) + + def test_v3_authpriv_requires_all_fields(self): + """authPriv requires both auth and priv protocol/pass.""" + with pytest.raises(ValidationError): + Credential.objects.create( + name='bad_authpriv', + version='3', + security_name='user', + security_level='authPriv', + auth_protocol='sha', + auth_pass='pass', + priv_protocol='aes', + priv_pass='', # missing + ) + + +# =========================================================================== +# Network.clean – CIDR validation +# =========================================================================== + +@pytest.mark.django_db +class TestNetworkClean: + + def test_valid_cidr_saves(self, test_connection): + """A valid CIDR network range saves without error.""" + net = Network.objects.create( + name='valid_net', + network_range='192.168.0.0/16', + connection=test_connection, + ) + assert net.id is not None + + def test_invalid_cidr_raises_validation_error(self, test_connection): + """An invalid CIDR raises ValidationError on save.""" + with pytest.raises(ValidationError): + Network.objects.create( + name='invalid_net', + network_range='not-a-cidr', + connection=test_connection, + ) + + def test_host_cidr_accepted_non_strict(self, test_connection): + """Non-strict CIDR (host bits set) is accepted by the model.""" + net = Network.objects.create( + name='host_cidr', + network_range='192.168.1.1/24', + connection=test_connection, + ) + assert net.id is not None + + +# =========================================================================== +# Device.clean – validation +# =========================================================================== + +@pytest.mark.django_db +class TestDeviceClean: + + def test_device_requires_ip_or_hostname(self, test_credential_v2c, test_network): + """Device.clean raises ValidationError if neither ip_address nor hostname is set.""" + with pytest.raises(ValidationError): + Device.objects.create( + name='no_addr', + ip_address=None, + hostname=None, + credential=test_credential_v2c, + network=test_network, + ) + + def test_device_invalid_ip_raises(self, test_credential_v2c, test_network): + """Device.clean raises ValidationError for an invalid IP address.""" + with pytest.raises(ValidationError): + Device.objects.create( + name='bad_ip', + ip_address='999.999.999.999', + credential=test_credential_v2c, + network=test_network, + ) + + def test_device_valid_hostname_only(self, test_credential_v2c, test_network): + """A device with only a hostname (no IP) is valid.""" + device = Device.objects.create( + name='hostname_only_dev', + hostname='mydevice.example.com', + ip_address=None, + credential=test_credential_v2c, + network=test_network, + ) + assert device.id is not None + + def test_device_str_uses_ip(self, test_credential_v2c, test_network): + """Device.__str__ uses the IP address when present.""" + device = Device.objects.create( + name='str_test', + ip_address='10.0.0.1', + credential=test_credential_v2c, + network=test_network, + ) + assert '10.0.0.1' in str(device) + + def test_device_str_fallback_no_address(self, test_credential_v2c, test_network): + """Device.__str__ falls back to 'no address' when both ip/hostname are None after object construction.""" + # Bypass model validation by using update() to set both to None + device = Device.objects.create( + name='str_no_addr', + ip_address='1.2.3.4', + credential=test_credential_v2c, + network=test_network, + ) + Device.objects.filter(pk=device.pk).update(ip_address=None, hostname=None) + device.refresh_from_db() + assert 'no address' in str(device) + + +# =========================================================================== +# Profile.clean – validation +# =========================================================================== + +@pytest.mark.django_db +class TestProfileClean: + + def test_profile_data_must_be_dict(self): + """Profile.clean raises ValidationError when profile_data is not a dict.""" + with pytest.raises(ValidationError): + p = Profile(name='bad_profile', vendor='Generic', profile_data='not a dict') + p.full_clean() + + def test_valid_profile_saves(self): + """A profile with valid dict profile_data saves successfully.""" + p = Profile.objects.create( + name='ok_profile', + vendor='Generic', + profile_data={'get': {'sysDescr': '1.3.6.1.2.1.1.1.0'}}, + ) + assert p.id is not None diff --git a/tests/SNMP/unit/test_network_map.py b/tests/SNMP/unit/test_network_map.py new file mode 100644 index 0000000..cba6d1c --- /dev/null +++ b/tests/SNMP/unit/test_network_map.py @@ -0,0 +1,618 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +""" +Tests for SNMP.network_map — adjacency-to-graph conversion and the +get_networks_list / get_network_map_data view helpers. +All Elasticsearch I/O is mocked. +""" + +import json +import pytest +from unittest.mock import patch, MagicMock +from django.test import RequestFactory + +from SNMP.network_map import ( + convert_adjacency_to_graph, + get_networks_list, + get_network_map_data, + get_cdp_adjacencies, + get_edge_interface_detail, +) +from SNMP.models import Network, Device, Credential +from PipelineManager.models import Connection + + +# =========================================================================== +# Fixtures +# =========================================================================== + +@pytest.fixture +def test_connection(db): + return Connection.objects.create( + name='NM Test Connection', + connection_type='CENTRALIZED', + host='https://localhost:9200', + username='elastic', + password='changeme' + ) + + +@pytest.fixture +def test_credential(db): + return Credential.objects.create( + name='nm_test_cred', + version='2c', + community='public' + ) + + +@pytest.fixture +def test_network(db, test_connection, test_credential): + return Network.objects.create( + name='NM Test Network', + network_range='10.0.0.0/24', + connection=test_connection, + discovery_credential=test_credential, + interval=30 + ) + + +@pytest.fixture +def test_device(db, test_network, test_credential): + return Device.objects.create( + name='switch-a', + ip_address='10.0.0.1', + port=161, + retries=1, + timeout=500, + credential=test_credential, + network=test_network, + ) + + +@pytest.fixture +def rf(): + return RequestFactory() + + +# =========================================================================== +# convert_adjacency_to_graph +# =========================================================================== + +class TestConvertAdjacencyToGraph: + """ + Tests for the pure graph-conversion function. + The only DB touch is the device-ID lookup at the end, which is + guarded by try/except; we let it fail silently in the test DB. + """ + + def test_empty_adjacency_table_returns_empty_graph(self, db): + result = convert_adjacency_to_graph({}) + assert result == {'nodes': [], 'edges': []} + + def test_single_device_no_neighbors_creates_node(self, db): + adjacency = { + 'Production (10.0.0.0/24)': { + 'switch-a': {} + } + } + result = convert_adjacency_to_graph(adjacency) + assert len(result['nodes']) == 1 + assert result['nodes'][0]['id'] == 'switch-a' + assert result['edges'] == [] + + def test_device_with_one_neighbor_creates_edge(self, db): + adjacency = { + 'Production': { + 'switch-a': { + 'GigabitEthernet0/1': { + 'device_id': 'switch-b', + 'port': 'GigabitEthernet0/2', + 'platform': 'Cisco IOS', + 'capabilities': 'Switch', + 'address': '10.0.0.2', + 'version': '15.2' + } + } + } + } + result = convert_adjacency_to_graph(adjacency) + assert len(result['nodes']) == 2 + assert len(result['edges']) == 1 + edge = result['edges'][0] + assert edge['source'] == 'switch-a' + assert edge['target'] == 'switch-b' + assert edge['source_interface'] == 'GigabitEthernet0/1' + assert edge['target_interface'] == 'GigabitEthernet0/2' + + def test_bidirectional_connection_creates_single_edge(self, db): + adjacency = { + 'Production': { + 'switch-a': { + 'Gi0/1': { + 'device_id': 'switch-b', 'port': 'Gi0/2', + 'platform': '', 'capabilities': '', 'address': '', 'version': '' + } + }, + 'switch-b': { + 'Gi0/2': { + 'device_id': 'switch-a', 'port': 'Gi0/1', + 'platform': '', 'capabilities': '', 'address': '', 'version': '' + } + } + } + } + result = convert_adjacency_to_graph(adjacency) + assert len(result['edges']) == 1 + + def test_managed_device_has_managed_true(self, db): + adjacency = { + 'Production': { + 'switch-a': { + 'Gi0/1': { + 'device_id': 'external-router', 'port': 'Eth0', + 'platform': '', 'capabilities': '', 'address': '', 'version': '' + } + } + } + } + result = convert_adjacency_to_graph(adjacency) + managed_node = next(n for n in result['nodes'] if n['id'] == 'switch-a') + assert managed_node['managed'] is True + + def test_discovered_only_device_has_managed_false(self, db): + adjacency = { + 'Production': { + 'switch-a': { + 'Gi0/1': { + 'device_id': 'external-router', 'port': 'Eth0', + 'platform': '', 'capabilities': '', 'address': '', 'version': '' + } + } + } + } + result = convert_adjacency_to_graph(adjacency) + discovered_node = next(n for n in result['nodes'] if n['id'] == 'external-router') + assert discovered_node['managed'] is False + + def test_device_that_appears_in_both_sides_is_managed(self, db): + adjacency = { + 'Production': { + 'switch-a': { + 'Gi0/1': { + 'device_id': 'switch-b', 'port': 'Gi0/2', + 'platform': '', 'capabilities': '', 'address': '', 'version': '' + } + }, + 'switch-b': {} + } + } + result = convert_adjacency_to_graph(adjacency) + b_node = next(n for n in result['nodes'] if n['id'] == 'switch-b') + assert b_node['managed'] is True + + def test_interface_count_increments_per_neighbor(self, db): + adjacency = { + 'Production': { + 'switch-a': { + 'Gi0/1': { + 'device_id': 'switch-b', 'port': 'Gi0/2', + 'platform': '', 'capabilities': '', 'address': '', 'version': '' + }, + 'Gi0/2': { + 'device_id': 'switch-c', 'port': 'Gi0/1', + 'platform': '', 'capabilities': '', 'address': '', 'version': '' + } + } + } + } + result = convert_adjacency_to_graph(adjacency) + a_node = next(n for n in result['nodes'] if n['id'] == 'switch-a') + assert a_node['interface_count'] == 2 + + def test_neighbor_without_device_id_skipped(self, db): + adjacency = { + 'Production': { + 'switch-a': { + 'Gi0/1': { + 'device_id': '', # empty — no neighbor name + 'port': 'Gi0/2', + 'platform': '', 'capabilities': '', 'address': '', 'version': '' + } + } + } + } + result = convert_adjacency_to_graph(adjacency) + # No edge should be created for an empty device_id + assert result['edges'] == [] + + def test_multiple_networks_all_included(self, db): + adjacency = { + 'Network A': {'device-a': {}}, + 'Network B': {'device-b': {}}, + } + result = convert_adjacency_to_graph(adjacency) + node_ids = {n['id'] for n in result['nodes']} + assert 'device-a' in node_ids + assert 'device-b' in node_ids + + def test_db_device_id_enrichment(self, test_device, db): + """Managed nodes get a device_id from the DB if their id matches.""" + adjacency = { + 'NM Test Network (10.0.0.0/24)': { + 'switch-a': {} + } + } + result = convert_adjacency_to_graph(adjacency) + # Node 'switch-a' matches the device name in the DB + a_node = next(n for n in result['nodes'] if n['id'] == 'switch-a') + assert a_node.get('device_id') == test_device.id + + def test_edge_contains_platform_and_capabilities(self, db): + adjacency = { + 'Production': { + 'switch-a': { + 'Gi0/1': { + 'device_id': 'switch-b', + 'port': 'Gi0/2', + 'platform': 'Cisco 3750', + 'capabilities': 'Switch Router', + 'address': '10.0.0.2', + 'version': '15.2' + } + } + } + } + result = convert_adjacency_to_graph(adjacency) + edge = result['edges'][0] + assert edge['platform'] == 'Cisco 3750' + assert edge['capabilities'] == 'Switch Router' + + +# =========================================================================== +# get_networks_list +# =========================================================================== + +@pytest.mark.django_db +class TestGetNetworksList: + + def test_no_networks_returns_empty_list(self, rf): + request = rf.get('/SNMP/GetNetworksList/') + response = get_networks_list(request) + data = json.loads(response.content) + assert data['success'] is True + assert data['networks'] == [] + + def test_returns_network_with_correct_fields(self, rf, test_network): + request = rf.get('/SNMP/GetNetworksList/') + response = get_networks_list(request) + data = json.loads(response.content) + assert data['success'] is True + assert len(data['networks']) == 1 + network = data['networks'][0] + assert network['id'] == test_network.id + assert network['name'] == test_network.name + assert network['network_range'] == test_network.network_range + assert 'device_count' in network + + def test_device_count_is_correct(self, rf, test_network, test_device): + request = rf.get('/SNMP/GetNetworksList/') + response = get_networks_list(request) + data = json.loads(response.content) + assert data['networks'][0]['device_count'] == 1 + + def test_networks_returned_alphabetically(self, rf, db, test_credential, test_connection): + Network.objects.create( + name='Zebra Network', network_range='10.2.0.0/24', + connection=test_connection, discovery_credential=test_credential, interval=30 + ) + Network.objects.create( + name='Alpha Network', network_range='10.3.0.0/24', + connection=test_connection, discovery_credential=test_credential, interval=30 + ) + request = rf.get('/SNMP/GetNetworksList/') + response = get_networks_list(request) + data = json.loads(response.content) + names = [n['name'] for n in data['networks']] + assert names == sorted(names) + + +# =========================================================================== +# get_network_map_data +# =========================================================================== + +@pytest.mark.django_db +class TestGetNetworkMapData: + + @patch('SNMP.network_map.get_cdp_adjacencies') + def test_no_networks_returns_empty_graph(self, mock_cdp, rf): + mock_cdp.return_value = { + 'success': False, + 'error': 'No connections', + 'adjacency_table': {} + } + request = rf.get('/SNMP/GetNetworkMap/') + response = get_network_map_data(request) + data = json.loads(response.content) + assert data['graph']['nodes'] == [] + assert data['graph']['edges'] == [] + + @patch('SNMP.network_map.get_cdp_adjacencies') + def test_with_adjacency_data_returns_graph(self, mock_cdp, rf): + mock_cdp.return_value = { + 'success': True, + 'adjacency_table': { + 'Production': { + 'switch-a': { + 'Gi0/1': { + 'device_id': 'switch-b', 'port': 'Gi0/2', + 'platform': '', 'capabilities': '', 'address': '', 'version': '' + } + } + } + }, + 'errors': None + } + request = rf.get('/SNMP/GetNetworkMap/') + response = get_network_map_data(request) + data = json.loads(response.content) + assert data['success'] is True + assert len(data['graph']['nodes']) == 2 + assert len(data['graph']['edges']) == 1 + + @patch('SNMP.network_map.get_cdp_adjacencies') + def test_network_filter_passed_to_cdp(self, mock_cdp, rf): + mock_cdp.return_value = { + 'success': False, + 'adjacency_table': {}, + 'error': 'none' + } + request = rf.get('/SNMP/GetNetworkMap/?networks=1&networks=2') + get_network_map_data(request) + mock_cdp.assert_called_once_with(network_ids=[1, 2]) + + @patch('SNMP.network_map.get_cdp_adjacencies') + def test_no_network_filter_passes_none(self, mock_cdp, rf): + mock_cdp.return_value = { + 'success': False, + 'adjacency_table': {}, + 'error': 'none' + } + request = rf.get('/SNMP/GetNetworkMap/') + get_network_map_data(request) + mock_cdp.assert_called_once_with(network_ids=None) + + @patch('SNMP.network_map.get_cdp_adjacencies') + def test_exception_returns_500_response(self, mock_cdp, rf): + mock_cdp.side_effect = Exception('Unexpected failure') + request = rf.get('/SNMP/GetNetworkMap/') + response = get_network_map_data(request) + assert response.status_code == 500 + data = json.loads(response.content) + assert data['success'] is False + + +# =========================================================================== +# get_cdp_adjacencies +# =========================================================================== + +@pytest.mark.django_db +class TestGetCdpAdjacencies: + """ + get_cdp_adjacencies queries ES for CDP/LLDP neighbor data. + All ES I/O is mocked; only the Django ORM layer is real. + """ + + def _empty_cdp_response(self): + """ES response with no CDP buckets.""" + return {'aggregations': {'cdp_adjacencies': {'buckets': []}}} + + def _cdp_response(self, host_sysname, table_index, neighbor_device_id, neighbor_port, + polled_address='10.0.0.1', network_name=''): + """Minimal ES response with one CDP bucket.""" + return { + 'aggregations': { + 'cdp_adjacencies': { + 'buckets': [ + { + 'key': {'host_name': host_sysname, 'cdp_row_index': table_index}, + 'latest': { + 'hits': { + 'hits': [ + { + '_source': { + 'host': { + 'sysname': host_sysname, + 'polled_address': polled_address, + 'hostname': '', + }, + 'network': { + 'name': network_name, + 'neighbor': { + 'index': table_index, + 'device_id': neighbor_device_id, + 'port': neighbor_port, + 'platform': 'Cisco IOS', + 'capabilities': 'Switch', + 'address': '10.0.0.2', + 'version': '15.2', + } + }, + 'event': {'category': 'network.neighbor'}, + } + } + ] + } + } + } + ] + } + } + } + + def test_no_networks_returns_failure(self): + result = get_cdp_adjacencies() + assert result['success'] is False + assert 'adjacency_table' in result + + def test_network_without_connection_not_queried(self, test_credential, db): + Network.objects.create( + name='No Conn', + network_range='10.99.0.0/24', + discovery_credential=test_credential, + interval=30, + connection=None, + ) + result = get_cdp_adjacencies() + assert result['success'] is False + + @patch('SNMP.network_map.get_elastic_connection') + def test_empty_cdp_response_returns_empty_adjacency(self, mock_get_es, test_network): + mock_es = MagicMock() + mock_es.search.return_value = self._empty_cdp_response() + mock_get_es.return_value = mock_es + + result = get_cdp_adjacencies() + assert result['success'] is True + assert result['adjacency_table'] == {} + assert result['errors'] is None + + @patch('SNMP.network_map.get_elastic_connection') + def test_cdp_data_populates_adjacency_table(self, mock_get_es, test_network, test_device): + mock_es = MagicMock() + mock_es.search.side_effect = [ + self._cdp_response( + host_sysname='switch-a', + table_index='1.1', + neighbor_device_id='switch-b', + neighbor_port='Gi0/2', + polled_address='10.0.0.1', + network_name='NM Test Network (10.0.0.0/24)', + ), + {'hits': {'hits': []}}, # interface name lookup returns nothing + ] + mock_get_es.return_value = mock_es + + result = get_cdp_adjacencies() + assert result['success'] is True + # adjacency table is non-empty + assert result['adjacency_table'] + + @patch('SNMP.network_map.get_elastic_connection') + def test_es_error_recorded_in_errors_list(self, mock_get_es, test_network): + mock_get_es.side_effect = Exception('ES down') + result = get_cdp_adjacencies() + assert result['success'] is True + assert result['errors'] is not None + assert len(result['errors']) == 1 + + @patch('SNMP.network_map.get_elastic_connection') + def test_network_id_filter_restricts_scope(self, mock_get_es, test_network): + mock_es = MagicMock() + mock_es.search.return_value = self._empty_cdp_response() + mock_get_es.return_value = mock_es + + result = get_cdp_adjacencies(network_ids=[test_network.id]) + assert result['success'] is True + + @patch('SNMP.network_map.get_elastic_connection') + def test_outer_exception_returns_failure(self, mock_get_es, test_network): + # Trigger the outer try/except by making Network.objects.filter raise + with patch('SNMP.network_map.Network.objects') as mock_objs: + mock_objs.filter.side_effect = Exception('DB error') + result = get_cdp_adjacencies() + assert result['success'] is False + assert 'error' in result + + +# =========================================================================== +# get_edge_interface_detail +# =========================================================================== + +@pytest.mark.django_db +class TestGetEdgeInterfaceDetail: + + @pytest.fixture + def rf(self): + from django.test import RequestFactory + return RequestFactory() + + def test_missing_source_returns_400(self, rf): + request = rf.get('/SNMP/GetEdgeInterfaceDetail/') + response = get_edge_interface_detail(request) + assert response.status_code == 400 + data = json.loads(response.content) + assert data['success'] is False + + def test_missing_source_iface_returns_400(self, rf): + request = rf.get('/SNMP/GetEdgeInterfaceDetail/?source=switch-a') + response = get_edge_interface_detail(request) + assert response.status_code == 400 + + def test_no_es_connections_returns_400(self, rf): + request = rf.get( + '/SNMP/GetEdgeInterfaceDetail/?source=switch-a&source_iface=Gi0/1' + ) + response = get_edge_interface_detail(request) + assert response.status_code == 400 + data = json.loads(response.content) + assert data['success'] is False + + @patch('SNMP.network_map.get_elastic_connection') + def test_returns_interface_data_for_source(self, mock_get_es, rf, test_network): + mock_es = MagicMock() + mock_es.search.return_value = { + 'hits': { + 'hits': [ + {'_source': {'interface': {'name': 'Gi0/1', 'speed': 1000}}} + ] + } + } + mock_get_es.return_value = mock_es + + request = rf.get( + '/SNMP/GetEdgeInterfaceDetail/' + '?source=switch-a&source_iface=Gi0/1' + '&target=switch-b&target_iface=Gi0/2' + ) + response = get_edge_interface_detail(request) + assert response.status_code == 200 + data = json.loads(response.content) + assert data['success'] is True + assert data['source']['sysname'] == 'switch-a' + assert data['source']['iface_name'] == 'Gi0/1' + assert data['target']['sysname'] == 'switch-b' + + @patch('SNMP.network_map.get_elastic_connection') + def test_no_hits_returns_none_interface(self, mock_get_es, rf, test_network): + mock_es = MagicMock() + mock_es.search.return_value = {'hits': {'hits': []}} + mock_get_es.return_value = mock_es + + request = rf.get( + '/SNMP/GetEdgeInterfaceDetail/' + '?source=unknown-device&source_iface=Gi0/1' + ) + response = get_edge_interface_detail(request) + assert response.status_code == 200 + data = json.loads(response.content) + assert data['success'] is True + assert data['source']['interface'] is None + + @patch('SNMP.network_map.get_elastic_connection') + def test_es_exception_on_lookup_still_returns_200(self, mock_get_es, rf, test_network): + mock_es = MagicMock() + mock_es.search.side_effect = Exception('ES lookup failed') + mock_get_es.return_value = mock_es + + request = rf.get( + '/SNMP/GetEdgeInterfaceDetail/' + '?source=switch-a&source_iface=Gi0/1' + ) + response = get_edge_interface_detail(request) + assert response.status_code == 200 + data = json.loads(response.content) + assert data['success'] is True + assert data['source']['interface'] is None diff --git a/tests/SNMP/unit/test_overview.py b/tests/SNMP/unit/test_overview.py new file mode 100644 index 0000000..b4384e5 --- /dev/null +++ b/tests/SNMP/unit/test_overview.py @@ -0,0 +1,482 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +""" +Tests for SNMP.overview — Elasticsearch query functions for the Overview page. +All Elasticsearch I/O is mocked; only the Django DB layer is real. +""" + +import pytest +from unittest.mock import patch, MagicMock + +from SNMP.overview import ( + get_discovered_devices_count, + get_high_resource_usage, + get_template_data_categories, +) +from SNMP.models import Network, Device, Credential +from PipelineManager.models import Connection + + +# =========================================================================== +# Fixtures +# =========================================================================== + +@pytest.fixture +def test_connection(db): + return Connection.objects.create( + name='Overview Test Connection', + connection_type='CENTRALIZED', + host='https://localhost:9200', + username='elastic', + password='changeme' + ) + + +@pytest.fixture +def test_credential(db): + return Credential.objects.create( + name='overview_test_cred', + version='2c', + community='public' + ) + + +@pytest.fixture +def test_network(db, test_connection, test_credential): + return Network.objects.create( + name='Overview Test Network', + network_range='10.0.0.0/24', + connection=test_connection, + discovery_credential=test_credential, + interval=30 + ) + + +@pytest.fixture +def test_device(db, test_network, test_credential): + return Device.objects.create( + name='overview_test_device', + ip_address='10.0.0.1', + port=161, + retries=1, + timeout=500, + credential=test_credential, + network=test_network, + ) + + +def _make_es_client(cardinality_value=5): + """Return a mock ES client with a canned discovered-devices response.""" + mock_es = MagicMock() + mock_es.search.return_value = { + 'aggregations': { + 'unique_hosts': { + 'value': cardinality_value + } + } + } + return mock_es + + +# =========================================================================== +# get_discovered_devices_count +# =========================================================================== + +@pytest.mark.django_db +class TestGetDiscoveredDevicesCount: + + def test_no_networks_returns_success_false(self): + result = get_discovered_devices_count() + assert result['success'] is False + assert result['count'] == 0 + assert 'No Elasticsearch connections' in result['error'] + + def test_network_without_connection_not_queried(self, db, test_credential): + Network.objects.create( + name='Unconnected Network', + network_range='192.168.0.0/24', + discovery_credential=test_credential, + interval=30, + connection=None + ) + result = get_discovered_devices_count() + assert result['success'] is False + + @patch('SNMP.overview.get_elastic_connection') + def test_returns_count_from_es_aggregation(self, mock_get_es, test_network): + mock_get_es.return_value = _make_es_client(cardinality_value=7) + + result = get_discovered_devices_count() + assert result['success'] is True + assert result['count'] == 7 + + @patch('SNMP.overview.get_elastic_connection') + def test_merges_counts_across_multiple_connections(self, mock_get_es, db, test_credential): + conn1 = Connection.objects.create( + name='OV Conn 1', connection_type='CENTRALIZED', + host='https://es1:9200', username='e', password='p' + ) + conn2 = Connection.objects.create( + name='OV Conn 2', connection_type='CENTRALIZED', + host='https://es2:9200', username='e', password='p' + ) + Network.objects.create( + name='Net 1', network_range='10.1.0.0/24', + connection=conn1, discovery_credential=test_credential, interval=30 + ) + Network.objects.create( + name='Net 2', network_range='10.2.0.0/24', + connection=conn2, discovery_credential=test_credential, interval=30 + ) + + mock_es1 = _make_es_client(cardinality_value=3) + mock_es2 = _make_es_client(cardinality_value=4) + mock_get_es.side_effect = [mock_es1, mock_es2] + + result = get_discovered_devices_count() + assert result['success'] is True + assert result['count'] == 7 + + @patch('SNMP.overview.get_elastic_connection') + def test_es_error_tracked_in_errors_list(self, mock_get_es, test_network): + mock_get_es.side_effect = Exception('Connection refused') + + result = get_discovered_devices_count() + assert result['success'] is True # overall success even with per-connection error + assert result['count'] == 0 + assert result['errors'] is not None + assert len(result['errors']) == 1 + + @patch('SNMP.overview.get_elastic_connection') + def test_no_errors_returns_none_for_errors_key(self, mock_get_es, test_network): + mock_get_es.return_value = _make_es_client(cardinality_value=2) + + result = get_discovered_devices_count() + assert result['errors'] is None + + @patch('SNMP.overview.get_elastic_connection') + def test_response_without_aggregations_treated_as_zero(self, mock_get_es, test_network): + mock_es = MagicMock() + mock_es.search.return_value = {} # no 'aggregations' key + mock_get_es.return_value = mock_es + + result = get_discovered_devices_count() + assert result['success'] is True + assert result['count'] == 0 + + +# =========================================================================== +# get_high_resource_usage +# =========================================================================== + +@pytest.mark.django_db +class TestGetHighResourceUsage: + + def test_no_devices_returns_empty_lists(self): + result = get_high_resource_usage() + assert result['success'] is True + assert result['high_cpu'] == [] + assert result['high_memory'] == [] + + def test_device_without_network_connection_skipped(self, db, test_credential, test_network): + # Create a network with no ES connection + net_no_conn = Network.objects.create( + name='No Conn Net', + network_range='172.16.0.0/24', + discovery_credential=test_credential, + interval=30, + connection=None + ) + Device.objects.create( + name='disconnected_device', + ip_address='172.16.0.1', + port=161, + retries=1, + timeout=500, + credential=test_credential, + network=net_no_conn, + ) + + result = get_high_resource_usage() + assert result['success'] is True + assert result['high_cpu'] == [] + assert result['high_memory'] == [] + + @patch('SNMP.overview.get_elastic_connection') + def test_device_with_high_cpu_appears_in_high_cpu_list(self, mock_get_es, test_device): + mock_es = MagicMock() + mock_es.search.return_value = { + 'aggregations': { + 'devices': { + 'buckets': [ + { + 'key': '10.0.0.1', + 'latest_cpu': { + 'hits': { + 'hits': [ + {'_source': {'system': {'cpu': {'total': {'norm': {'pct': 0.95}}}}}} + ] + } + }, + 'latest_memory': {'hits': {'hits': []}} + } + ] + } + } + } + mock_get_es.return_value = mock_es + + result = get_high_resource_usage() + assert result['success'] is True + assert len(result['high_cpu']) == 1 + assert result['high_cpu'][0]['cpu_pct'] == 95.0 + assert result['high_cpu'][0]['ip_address'] == '10.0.0.1' + + @patch('SNMP.overview.get_elastic_connection') + def test_device_with_high_memory_appears_in_high_memory_list(self, mock_get_es, test_device): + mock_es = MagicMock() + mock_es.search.return_value = { + 'aggregations': { + 'devices': { + 'buckets': [ + { + 'key': '10.0.0.1', + 'latest_cpu': {'hits': {'hits': []}}, + 'latest_memory': { + 'hits': { + 'hits': [ + {'_source': {'system': {'memory': {'actual': {'used': {'pct': 0.87}}}}}} + ] + } + } + } + ] + } + } + } + mock_get_es.return_value = mock_es + + result = get_high_resource_usage() + assert result['success'] is True + assert len(result['high_memory']) == 1 + assert result['high_memory'][0]['memory_pct'] == 87.0 + + @patch('SNMP.overview.get_elastic_connection') + def test_device_below_threshold_not_included(self, mock_get_es, test_device): + mock_es = MagicMock() + mock_es.search.return_value = { + 'aggregations': { + 'devices': { + 'buckets': [ + { + 'key': '10.0.0.1', + 'latest_cpu': { + 'hits': { + 'hits': [ + {'_source': {'system': {'cpu': {'total': {'norm': {'pct': 0.5}}}}}} + ] + } + }, + 'latest_memory': {'hits': {'hits': []}} + } + ] + } + } + } + mock_get_es.return_value = mock_es + + result = get_high_resource_usage() + assert result['high_cpu'] == [] + + @patch('SNMP.overview.get_elastic_connection') + def test_high_cpu_sorted_highest_first(self, mock_get_es, db, test_network, test_credential): + Device.objects.create( + name='device_b', ip_address='10.0.0.2', port=161, + retries=1, timeout=500, credential=test_credential, network=test_network + ) + + mock_es = MagicMock() + mock_es.search.return_value = { + 'aggregations': { + 'devices': { + 'buckets': [ + { + 'key': '10.0.0.1', + 'latest_cpu': { + 'hits': { + 'hits': [{'_source': {'system': {'cpu': {'total': {'norm': {'pct': 0.85}}}}}}] + } + }, + 'latest_memory': {'hits': {'hits': []}} + }, + { + 'key': '10.0.0.2', + 'latest_cpu': { + 'hits': { + 'hits': [{'_source': {'system': {'cpu': {'total': {'norm': {'pct': 0.95}}}}}}] + } + }, + 'latest_memory': {'hits': {'hits': []}} + } + ] + } + } + } + mock_get_es.return_value = mock_es + + result = get_high_resource_usage() + assert result['high_cpu'][0]['cpu_pct'] == 95.0 + assert result['high_cpu'][1]['cpu_pct'] == 85.0 + + @patch('SNMP.overview.get_elastic_connection') + def test_es_error_tracked_in_errors_list(self, mock_get_es, test_device): + mock_get_es.side_effect = Exception('Connection refused') + + result = get_high_resource_usage() + assert result['success'] is True + assert result['errors'] is not None + + @patch('SNMP.overview.get_elastic_connection') + def test_no_errors_returns_none_for_errors_key(self, mock_get_es, test_device): + mock_es = MagicMock() + mock_es.search.return_value = {'aggregations': {'devices': {'buckets': []}}} + mock_get_es.return_value = mock_es + + result = get_high_resource_usage() + assert result['errors'] is None + + +# =========================================================================== +# get_template_data_categories +# =========================================================================== + +@pytest.mark.django_db +class TestGetTemplateDataCategories: + + def test_no_networks_returns_empty_templates(self): + result = get_template_data_categories() + assert result['success'] is True + assert result['templates'] == [] + + @patch('SNMP.overview.get_elastic_connection') + def test_returns_templates_with_categories(self, mock_get_es, test_network): + mock_es = MagicMock() + mock_es.search.return_value = { + 'aggregations': { + 'templates': { + 'buckets': [ + { + 'key': 'dell_idrac', + 'categories': { + 'buckets': [ + {'key': 'system'}, + {'key': 'interface'}, + ] + } + } + ] + } + } + } + mock_get_es.return_value = mock_es + + result = get_template_data_categories() + assert result['success'] is True + assert len(result['templates']) == 1 + assert result['templates'][0]['template_name'] == 'dell_idrac' + assert 'system' in result['templates'][0]['categories'] + assert 'interface' in result['templates'][0]['categories'] + + @patch('SNMP.overview.get_elastic_connection') + def test_categories_sorted_alphabetically(self, mock_get_es, test_network): + mock_es = MagicMock() + mock_es.search.return_value = { + 'aggregations': { + 'templates': { + 'buckets': [ + { + 'key': 'generic', + 'categories': { + 'buckets': [ + {'key': 'system'}, + {'key': 'interface'}, + {'key': 'entity_sensor'}, + ] + } + } + ] + } + } + } + mock_get_es.return_value = mock_es + + result = get_template_data_categories() + cats = result['templates'][0]['categories'] + assert cats == sorted(cats) + + @patch('SNMP.overview.get_elastic_connection') + def test_merges_categories_across_connections(self, mock_get_es, db, test_credential): + conn1 = Connection.objects.create( + name='Cat Conn 1', connection_type='CENTRALIZED', + host='https://es1:9200', username='e', password='p' + ) + conn2 = Connection.objects.create( + name='Cat Conn 2', connection_type='CENTRALIZED', + host='https://es2:9200', username='e', password='p' + ) + Network.objects.create( + name='CatNet1', network_range='10.10.0.0/24', + connection=conn1, discovery_credential=test_credential, interval=30 + ) + Network.objects.create( + name='CatNet2', network_range='10.11.0.0/24', + connection=conn2, discovery_credential=test_credential, interval=30 + ) + + def side_effect(conn_id): + mock_es = MagicMock() + if conn_id == conn1.id: + mock_es.search.return_value = { + 'aggregations': { + 'templates': { + 'buckets': [{'key': 'cisco', 'categories': {'buckets': [{'key': 'system'}]}}] + } + } + } + else: + mock_es.search.return_value = { + 'aggregations': { + 'templates': { + 'buckets': [{'key': 'cisco', 'categories': {'buckets': [{'key': 'interface'}]}}] + } + } + } + return mock_es + + mock_get_es.side_effect = side_effect + + result = get_template_data_categories() + assert result['success'] is True + cisco_template = next(t for t in result['templates'] if t['template_name'] == 'cisco') + assert 'system' in cisco_template['categories'] + assert 'interface' in cisco_template['categories'] + + @patch('SNMP.overview.get_elastic_connection') + def test_es_error_tracked_continues(self, mock_get_es, test_network): + mock_get_es.side_effect = Exception('ES down') + + result = get_template_data_categories() + assert result['success'] is True + assert result['errors'] is not None + + @patch('SNMP.overview.get_elastic_connection') + def test_response_without_aggregations_returns_empty(self, mock_get_es, test_network): + mock_es = MagicMock() + mock_es.search.return_value = {} + mock_get_es.return_value = mock_es + + result = get_template_data_categories() + assert result['success'] is True + assert result['templates'] == [] diff --git a/tests/SNMP/unit/test_snmp_crud.py b/tests/SNMP/unit/test_snmp_crud.py new file mode 100644 index 0000000..a1e4e09 --- /dev/null +++ b/tests/SNMP/unit/test_snmp_crud.py @@ -0,0 +1,2485 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +import pytest +from django.contrib.auth.models import User +from django.test import Client +from unittest.mock import patch, MagicMock, Mock +import json + +from SNMP.models import Network, Device, Credential, Profile, DeviceTemplate +from PipelineManager.models import Connection +from Management.models import UserProfile + + +@pytest.fixture +def admin_user(db): + """Create a uer with admin profile""" + user = User.objects.create_user( + username='admin_user', + password='testpass123', + email='admin@example.com' + ) + profile, created = UserProfile.objects.get_or_create(user=user, defaults={'role': 'admin'}) + if not created: + profile.role = 'admin' + profile.save() + return user + + +@pytest.fixture +def readonly_user(db): + """Create a user with readonly profile""" + user = User.objects.create_user( + username='readonly_user', + password='testpass123', + email='readonly@example.com' + ) + profile = UserProfile.objects.get(user=user) + profile.role = 'readonly' + profile.save() + user.refresh_from_db() + return user + + +@pytest.fixture +def authenticated_client(admin_user): + """Create an authenticated client with admin user""" + client = Client() + client.force_login(admin_user) + return client + + +@pytest.fixture +def readonly_client(readonly_user): + """Create an authenticated client with readonly user""" + client = Client() + client.force_login(readonly_user) + return client + + +@pytest.fixture +def test_connection(db): + """Create a test Elasticsearch connection""" + return Connection.objects.create( + name='Test Connection', + connection_type='CENTRALIZED', + host='https://localhost:9200', + username='elastic', + password='changeme' + ) + + +@pytest.fixture +def test_credential_v2c(db): + """Create a test SNMP v2c credential""" + return Credential.objects.create( + name='Test Credential v2c', + version='2c', + community='public', + description='Test SNMP v2c credential' + ) + + +@pytest.fixture +def test_credential_v3(db): + """Create a test SNMP v3 credential""" + return Credential.objects.create( + name='Test Credential v3', + version='3', + security_name='snmpuser', + security_level='authPriv', + auth_protocol='sha', + auth_pass='authpassword', + priv_protocol='aes', + priv_pass='privpassword', + description='Test SNMP v3 credential' + ) + + +@pytest.fixture +def test_network(db, test_connection, test_credential_v2c): + """Create a test SNMP network""" + return Network.objects.create( + name='Test Network', + network_range='192.168.1.0/24', + connection=test_connection, + discovery_credential=test_credential_v2c, + discovery_enabled=True, + traps_enabled=False, + interval=30 + ) + + +@pytest.fixture +def test_device(db, test_network, test_credential_v2c): + """Create a test SNMP device""" + return Device.objects.create( + name='Test Device', + ip_address='192.168.1.100', + port=161, + retries=2, + timeout=1000, + credential=test_credential_v2c, + network=test_network + ) + + +@pytest.fixture +def test_profile(db): + """Create a test user profile""" + return Profile.objects.create( + name='custom_profile', + description='Custom test profile', + vendor='Generic', + profile_data={ + 'get': { + 'test.metric': '1.3.6.1.2.1.1.1.0' + }, + 'walk': {}, + 'table': {} + } + ) + + +# ============================================================================ +# Credential CRUD Tests +# ============================================================================ + +@pytest.mark.django_db +class TestCredentialCRUD: + """Test Credential Create, Read, Update, Delete operations""" + + def test_get_credentials(self, authenticated_client, test_credential_v2c): + """Test getting all credentials""" + response = authenticated_client.get('/SNMP/GetCredentials/') + assert response.status_code == 200 + data = json.loads(response.content) + assert isinstance(data, list) + assert len(data) >= 1 + assert any(c['name'] == 'Test Credential v2c' for c in data) + + def test_get_credential_by_id(self, authenticated_client, test_credential_v2c): + """Test getting a single credential by ID""" + response = authenticated_client.get(f'/SNMP/GetCredential/{test_credential_v2c.id}/') + assert response.status_code == 200 + data = json.loads(response.content) + assert data['name'] == 'Test Credential v2c' + assert data['version'] == '2c' + # Community should be masked + assert data['community'] == '***' + + def test_get_credential_not_found(self, authenticated_client): + """Test getting a non-existent credential""" + response = authenticated_client.get('/SNMP/GetCredential/99999/') + assert response.status_code == 404 + + def test_add_credential_v2c_requires_admin(self, readonly_client): + """Test that adding a credential requires admin role""" + response = readonly_client.post('/SNMP/AddCredential/', { + 'name': 'New Credential', + 'version': '2c', + 'community': 'public' + }) + assert response.status_code == 403 + assert b'Admin role required' in response.content + + def test_add_credential_v2c_success(self, authenticated_client): + """Test successfully adding a v2c credential""" + response = authenticated_client.post('/SNMP/AddCredential/', { + 'name': 'New v2c Credential', + 'version': '2c', + 'community': 'private', + 'description': 'Test description' + }) + assert response.status_code == 200 + data = json.loads(response.content) + assert 'id' in data + assert 'Credential created successfully!' in data['message'] + + # Verify credential was created + credential = Credential.objects.get(name='New v2c Credential') + assert credential.version == '2c' + assert credential.get_community() == 'private' + + def test_add_credential_v3_success(self, authenticated_client): + """Test successfully adding a v3 credential""" + response = authenticated_client.post('/SNMP/AddCredential/', { + 'name': 'New v3 Credential', + 'version': '3', + 'security_name': 'testuser', + 'security_level': 'authPriv', + 'auth_protocol': 'sha', + 'auth_pass': 'authpass123', + 'priv_protocol': 'aes', + 'priv_pass': 'privpass123' + }) + assert response.status_code == 200 + data = json.loads(response.content) + assert 'id' in data + + # Verify credential was created + credential = Credential.objects.get(name='New v3 Credential') + assert credential.version == '3' + assert credential.security_name == 'testuser' + assert credential.get_auth_pass() == 'authpass123' + assert credential.get_priv_pass() == 'privpass123' + + def test_add_credential_validation_error(self, authenticated_client): + """Test adding a credential with validation errors""" + response = authenticated_client.post('/SNMP/AddCredential/', { + 'name': 'Invalid Credential', + 'version': '2c', + 'community': '' # Empty community should fail + }) + assert response.status_code == 400 + assert b'Community string is required' in response.content + + def test_update_credential_requires_admin(self, readonly_client, test_credential_v2c): + """Test that updating a credential requires admin role""" + response = readonly_client.post(f'/SNMP/UpdateCredential/{test_credential_v2c.id}/', { + 'name': 'Updated Name', + 'version': '2c', + 'community': 'newcommunity' + }) + assert response.status_code == 403 + + def test_update_credential_success(self, authenticated_client, test_credential_v2c): + """Test successfully updating a credential""" + response = authenticated_client.post(f'/SNMP/UpdateCredential/{test_credential_v2c.id}/', { + 'name': 'Updated Credential', + 'version': '2c', + 'community': 'newcommunity', + 'description': 'Updated description' + }) + assert response.status_code == 200 + + # Verify credential was updated + test_credential_v2c.refresh_from_db() + assert test_credential_v2c.name == 'Updated Credential' + assert test_credential_v2c.get_community() == 'newcommunity' + + def test_update_credential_not_found(self, authenticated_client): + """Test updating a non-existent credential""" + response = authenticated_client.post('/SNMP/UpdateCredential/99999/', { + 'name': 'Test', + 'version': '2c', + 'community': 'public' + }) + assert response.status_code == 404 + + def test_delete_credential_requires_admin(self, readonly_client, test_credential_v2c): + """Test that deleting a credential requires admin role""" + response = readonly_client.post(f'/SNMP/DeleteCredential/{test_credential_v2c.id}/') + assert response.status_code == 403 + + def test_delete_credential_success(self, authenticated_client, test_credential_v2c): + """Test successfully deleting a credential""" + credential_id = test_credential_v2c.id + response = authenticated_client.post(f'/SNMP/DeleteCredential/{credential_id}/') + assert response.status_code == 200 + assert b'Credential deleted successfully!' in response.content + + # Verify credential was deleted + assert not Credential.objects.filter(id=credential_id).exists() + + def test_delete_credential_not_found(self, authenticated_client): + """Test deleting a non-existent credential""" + response = authenticated_client.post('/SNMP/DeleteCredential/99999/') + assert response.status_code == 404 + + +# ============================================================================ +# Network CRUD Tests +# ============================================================================ + +@pytest.mark.django_db +class TestNetworkCRUD: + """Test Network Create, Read, Update, Delete operations""" + + def test_get_networks(self, authenticated_client, test_network): + """Test getting all networks""" + response = authenticated_client.get('/SNMP/GetNetworks/') + assert response.status_code == 200 + data = json.loads(response.content) + assert isinstance(data, list) + assert len(data) >= 1 + assert any(n['name'] == 'Test Network' for n in data) + + def test_get_network_by_id(self, authenticated_client, test_network): + """Test getting a single network by ID""" + response = authenticated_client.get(f'/SNMP/GetNetwork/{test_network.id}/') + assert response.status_code == 200 + data = json.loads(response.content) + assert data['name'] == 'Test Network' + assert data['network_range'] == '192.168.1.0/24' + + def test_add_network_requires_admin(self, readonly_client, test_connection, test_credential_v2c): + """Test that adding a network requires admin role""" + response = readonly_client.post('/SNMP/AddNetwork/', { + 'name': 'New Network', + 'network_range': '10.0.0.0/24', + 'connection': test_connection.id, + 'discovery_credential': test_credential_v2c.id + }) + assert response.status_code == 403 + + def test_add_network_success(self, authenticated_client, test_connection, test_credential_v2c): + """Test successfully adding a network""" + response = authenticated_client.post('/SNMP/AddNetwork/', { + 'name': 'New Network', + 'network_range': '10.0.0.0/24', + 'connection': test_connection.id, + 'discovery_credential': test_credential_v2c.id, + 'discovery_enabled': 'true', + 'traps_enabled': 'false', + 'interval': '60' + }) + assert response.status_code == 200 + data = json.loads(response.content) + assert 'id' in data + assert 'Network created successfully!' in data['message'] + + # Verify network was created + network = Network.objects.get(name='New Network') + assert network.network_range == '10.0.0.0/24' + assert network.interval == 60 + + def test_add_network_invalid_cidr(self, authenticated_client, test_connection): + """Test adding a network with invalid CIDR notation""" + response = authenticated_client.post('/SNMP/AddNetwork/', { + 'name': 'Invalid Network', + 'network_range': 'not-a-valid-cidr', + }) + assert response.status_code == 400 + assert b'Invalid CIDR notation' in response.content + + def test_update_network_requires_admin(self, readonly_client, test_network): + """Test that updating a network requires admin role""" + response = readonly_client.post(f'/SNMP/UpdateNetwork/{test_network.id}/', { + 'name': 'Updated Network', + 'network_range': '192.168.1.0/24', + }) + assert response.status_code == 403 + + def test_update_network_success(self, authenticated_client, test_network): + """Test successfully updating a network""" + response = authenticated_client.post(f'/SNMP/UpdateNetwork/{test_network.id}/', { + 'name': 'Updated Network', + 'network_range': '192.168.2.0/24', + 'interval': '120' + }) + assert response.status_code == 200 + + # Verify network was updated + test_network.refresh_from_db() + assert test_network.name == 'Updated Network' + assert test_network.network_range == '192.168.2.0/24' + assert test_network.interval == 120 + + def test_delete_network_requires_admin(self, readonly_client, test_network): + """Test that deleting a network requires admin role""" + response = readonly_client.post(f'/SNMP/DeleteNetwork/{test_network.id}/') + assert response.status_code == 403 + + @patch('SNMP.snmp_crud.get_elastic_connection') + def test_delete_network_success(self, mock_es_conn, authenticated_client, test_network): + """Test successfully deleting a network""" + # Mock Elasticsearch connection + mock_es = MagicMock() + mock_es.logstash.get_pipeline.return_value = {} + mock_es_conn.return_value = mock_es + + network_id = test_network.id + response = authenticated_client.post(f'/SNMP/DeleteNetwork/{network_id}/') + assert response.status_code == 200 + data = json.loads(response.content) + assert data['success'] is True + + # Verify network was deleted + assert not Network.objects.filter(id=network_id).exists() + + def test_get_network_pipeline_name(self, authenticated_client, test_network): + """Test getting the pipeline name for a network""" + response = authenticated_client.get(f'/SNMP/GetNetworkPipelineName/{test_network.id}/') + assert response.status_code == 200 + data = json.loads(response.content) + assert data['success'] is True + assert 'pipeline_name' in data + assert 'snmp-' in data['pipeline_name'] + + +# ============================================================================ +# Device CRUD Tests +# ============================================================================ + +@pytest.mark.django_db +class TestDeviceCRUD: + """Test Device Create, Read, Update, Delete operations""" + + def test_get_devices_paginated(self, authenticated_client, test_device): + """Test getting paginated devices""" + response = authenticated_client.get('/SNMP/GetDevices/?page=1&page_size=25') + assert response.status_code == 200 + data = json.loads(response.content) + assert 'devices' in data + assert 'total' in data + assert 'page' in data + assert len(data['devices']) >= 1 + + def test_get_devices_with_search(self, authenticated_client, test_device): + """Test getting devices with search filter""" + response = authenticated_client.get('/SNMP/GetDevices/?search=Test') + assert response.status_code == 200 + data = json.loads(response.content) + assert len(data['devices']) >= 1 + assert any(d['name'] == 'Test Device' for d in data['devices']) + + def test_get_devices_with_network_filter(self, authenticated_client, test_device, test_network): + """Test getting devices filtered by network""" + response = authenticated_client.get(f'/SNMP/GetDevices/?network={test_network.id}') + assert response.status_code == 200 + data = json.loads(response.content) + assert all(d['network_id'] == test_network.id for d in data['devices']) + + def test_get_device_by_id(self, authenticated_client, test_device): + """Test getting a single device by ID""" + response = authenticated_client.get(f'/SNMP/GetDevice/{test_device.id}/') + assert response.status_code == 200 + data = json.loads(response.content) + assert data['name'] == 'Test Device' + assert data['ip_address'] == '192.168.1.100' + assert 'device_template' in data + + def test_add_device_requires_admin(self, readonly_client, test_network, test_credential_v2c): + """Test that adding a device requires admin role""" + response = readonly_client.post('/SNMP/AddDevice/', { + 'name': 'New Device', + 'ip_address': '192.168.1.101', + 'network': test_network.id, + 'credential': test_credential_v2c.id + }) + assert response.status_code == 403 + + def test_add_device_success(self, authenticated_client, test_network, test_credential_v2c): + """Test successfully adding a device""" + response = authenticated_client.post('/SNMP/AddDevice/', { + 'name': 'New Device', + 'ip_address': '192.168.1.101', + 'port': '161', + 'retries': '3', + 'timeout': '2000', + 'network': test_network.id, + 'credential': test_credential_v2c.id, + 'profiles': ['system'] + }) + assert response.status_code == 200 + data = json.loads(response.content) + assert 'id' in data + assert 'Device created successfully!' in data['message'] + + # Verify device was created + device = Device.objects.get(name='New Device') + assert device.ip_address == '192.168.1.101' + + def test_add_device_auto_adds_system_profile(self, authenticated_client, test_network, test_credential_v2c): + """Test that system profile is automatically added to devices""" + response = authenticated_client.post('/SNMP/AddDevice/', { + 'name': 'Device Without Profiles', + 'ip_address': '192.168.1.102', + 'network': test_network.id, + 'credential': test_credential_v2c.id + }) + assert response.status_code == 200 + + # Verify device was created successfully + device = Device.objects.get(name='Device Without Profiles') + assert device.ip_address == '192.168.1.102' + + def test_add_device_invalid_ip(self, authenticated_client, test_network, test_credential_v2c): + """Test adding a device with invalid IP address""" + response = authenticated_client.post('/SNMP/AddDevice/', { + 'name': 'Invalid Device', + 'ip_address': 'not-an-ip!@#', + 'network': test_network.id, + 'credential': test_credential_v2c.id + }) + assert response.status_code == 400 + + def test_update_device_requires_admin(self, readonly_client, test_device): + """Test that updating a device requires admin role""" + response = readonly_client.post(f'/SNMP/UpdateDevice/{test_device.id}/', { + 'name': 'Updated Device', + 'ip_address': '192.168.1.100' + }) + assert response.status_code == 403 + + def test_update_device_success(self, authenticated_client, test_device): + """Test successfully updating a device""" + response = authenticated_client.post(f'/SNMP/UpdateDevice/{test_device.id}/', { + 'name': 'Updated Device', + 'ip_address': '192.168.1.200', + 'port': '162', + 'profiles': ['system'] + }) + assert response.status_code == 200 + + # Verify device was updated + test_device.refresh_from_db() + assert test_device.name == 'Updated Device' + assert test_device.ip_address == '192.168.1.200' + assert test_device.port == 162 + + def test_delete_device_requires_admin(self, readonly_client, test_device): + """Test that deleting a device requires admin role""" + response = readonly_client.post(f'/SNMP/DeleteDevice/{test_device.id}/') + assert response.status_code == 403 + + def test_delete_device_success(self, authenticated_client, test_device): + """Test successfully deleting a device""" + device_id = test_device.id + response = authenticated_client.post(f'/SNMP/DeleteDevice/{device_id}/') + assert response.status_code == 200 + assert b'Device deleted successfully!' in response.content + + # Verify device was deleted + assert not Device.objects.filter(id=device_id).exists() + + +# ============================================================================ +# Profile CRUD Tests +# ============================================================================ + +@pytest.mark.django_db +class TestProfileCRUD: + """Test Profile Create, Read, Update, Delete operations""" + + def test_get_all_profiles(self, authenticated_client, test_profile): + """Test getting all profiles""" + response = authenticated_client.get('/SNMP/GetAllProfiles/') + assert response.status_code == 200 + data = json.loads(response.content) + assert 'profiles' in data + # The custom profile created by the fixture should always appear + assert any(p['name'] == 'custom_profile' for p in data['profiles']) + + def test_get_official_profile(self, authenticated_client): + """Test getting an official profile (mocks filesystem)""" + fake_data = {'description': 'Generic system profile', 'vendor': 'Generic', 'get': {}} + with patch('SNMP.snmp_crud.os.path.exists', return_value=True), \ + patch('builtins.open', create=True) as mock_open, \ + patch('SNMP.snmp_crud.json.load', return_value=fake_data): + mock_open.return_value.__enter__ = lambda s: s + mock_open.return_value.__exit__ = MagicMock(return_value=False) + response = authenticated_client.get('/SNMP/GetOfficialProfile/system/') + assert response.status_code == 200 + data = json.loads(response.content) + assert data['success'] is True + assert 'profile_data' in data + + def test_get_user_profile(self, authenticated_client, test_profile): + """Test getting a user profile""" + response = authenticated_client.get('/SNMP/GetProfile/custom_profile/') + assert response.status_code == 200 + data = json.loads(response.content) + assert data['success'] is True + assert data['name'] == 'custom_profile' + + def test_add_profile_requires_admin(self, readonly_client): + """Test that adding a profile requires admin role""" + response = readonly_client.post('/SNMP/AddProfile/', + json.dumps({ + 'name': 'new_profile', + 'description': 'Test', + 'profile_data': {'get': {}} + }), + content_type='application/json' + ) + assert response.status_code == 403 + + def test_add_profile_success(self, authenticated_client): + """Test successfully adding a profile""" + response = authenticated_client.post('/SNMP/AddProfile/', + json.dumps({ + 'name': 'new_custom_profile', + 'description': 'New custom profile', + 'type': 'Network', + 'vendor': 'Cisco', + 'profile_data': { + 'get': { + 'custom.metric': '1.3.6.1.4.1.1.1.0' + }, + 'walk': {}, + 'table': {} + } + }), + content_type='application/json' + ) + assert response.status_code == 200 + data = json.loads(response.content) + assert data['success'] is True + + # Verify profile was created + profile = Profile.objects.get(name='new_custom_profile') + assert profile.vendor == 'Cisco' + + def test_add_profile_duplicate_name(self, authenticated_client, test_profile): + """Test adding a profile with duplicate name""" + response = authenticated_client.post('/SNMP/AddProfile/', + json.dumps({ + 'name': 'custom_profile', + 'profile_data': {'get': {}} + }), + content_type='application/json' + ) + assert response.status_code == 400 + data = json.loads(response.content) + assert 'already exists' in data['message'] + + def test_update_profile_requires_admin(self, readonly_client, test_profile): + """Test that updating a profile requires admin role""" + response = readonly_client.post(f'/SNMP/UpdateProfile/{test_profile.name}/', + json.dumps({ + 'name': 'updated_profile', + 'profile_data': {'get': {}} + }), + content_type='application/json' + ) + assert response.status_code == 403 + + def test_update_profile_success(self, authenticated_client, test_profile): + """Test successfully updating a profile""" + response = authenticated_client.post(f'/SNMP/UpdateProfile/{test_profile.name}/', + json.dumps({ + 'description': 'Updated description', + 'vendor': 'Updated Vendor', + 'profile_data': { + 'get': { + 'updated.metric': '1.3.6.1.2.1.1.2.0' + } + } + }), + content_type='application/json' + ) + assert response.status_code == 200 + + # Verify profile was updated + test_profile.refresh_from_db() + assert test_profile.description == 'Updated description' + assert test_profile.vendor == 'Updated Vendor' + + def test_delete_profile_requires_admin(self, readonly_client, test_profile): + """Test that deleting a profile requires admin role""" + response = readonly_client.post(f'/SNMP/DeleteProfile/{test_profile.name}/') + assert response.status_code == 403 + + def test_delete_profile_success(self, authenticated_client, test_profile): + """Test successfully deleting a profile""" + response = authenticated_client.post(f'/SNMP/DeleteProfile/{test_profile.name}/') + assert response.status_code == 200 + data = json.loads(response.content) + assert data['success'] is True + + # Verify profile was deleted + assert not Profile.objects.filter(name='custom_profile').exists() + + def test_delete_system_profile_forbidden(self, authenticated_client): + """Test that system profile cannot be deleted""" + response = authenticated_client.post('/SNMP/DeleteProfile/system/') + assert response.status_code == 403 + data = json.loads(response.content) + assert 'cannot be deleted' in data['message'] + + +# ============================================================================ +# Deploy Configuration Tests +# ============================================================================ + +@pytest.mark.django_db +class TestDeployConfiguration: + """Test configuration deployment operations""" + + def test_get_deploy_diff(self, authenticated_client, test_network, test_device): + """Test getting deploy diff""" + with patch('SNMP.snmp_crud.get_elastic_connection') as mock_es_conn: + mock_es = MagicMock() + mock_es.logstash.get_pipeline.return_value = {} + mock_es_conn.return_value = mock_es + + response = authenticated_client.get('/SNMP/GetDeployDiff/') + assert response.status_code == 200 + data = json.loads(response.content) + assert data['success'] is True + assert 'networks' in data + assert any(c['id'] == test_network.connection.id for c in data['connections']) + + def test_get_deploy_diff_includes_es_connection_for_agent_only_networks( + self, authenticated_client, test_network, test_device, test_connection + ): + """Agent-only setups still need the SNMP index template on their ES connection.""" + from PipelineManager.models import Policy, Connection as AgentConnection + + policy = Policy.objects.create( + name='Simulated SNMP Policy', + settings_path='/etc/logstash/', + logs_path='/var/log/logstash', + binary_path='/usr/share/logstash/bin', + logstash_yml='http.host: "0.0.0.0"', + jvm_options='-Xms1g', + log4j2_properties='logger.logstash.name = logstash', + keystore_password='test_password', + ) + agent = AgentConnection.objects.create( + name='SimulatedSNMP Agent', + connection_type='AGENT', + host='agent.example.com', + agent_id='sim-snmp-001', + is_active=True, + policy=policy, + ) + test_network.deployment_mode = 'AGENT' + test_network.agent_connection = agent + test_network.save(update_fields=['deployment_mode', 'agent_connection']) + + with patch('SNMP.snmp_crud.get_elastic_connection') as mock_es_conn: + mock_es = MagicMock() + mock_es.logstash.get_pipeline.return_value = {} + mock_es_conn.return_value = mock_es + + response = authenticated_client.get('/SNMP/GetDeployDiff/') + + assert response.status_code == 200 + data = json.loads(response.content) + assert data['success'] is True + assert data['connections'] == [ + {'id': test_connection.id, 'name': test_connection.name} + ] + + # GetDeployDiff caches a 60s deploy plan; don't leak an Agent-mode plan + # into later DeployConfiguration tests in this class. + from django.core.cache import cache + cache.delete('snmp_deployment_plan') + + def test_deploy_configuration_requires_admin(self, readonly_client): + """Test that deploying configuration requires admin role""" + response = readonly_client.post('/SNMP/DeployConfiguration/') + assert response.status_code == 403 + + @patch('SNMP.snmp_crud.get_elastic_connection') + def test_deploy_configuration_success(self, mock_es_conn, authenticated_client, test_network, test_device): + """Test successfully deploying configuration""" + mock_es = MagicMock() + mock_es.logstash.get_pipeline.return_value = {} + mock_es.logstash.put_pipeline.return_value = {'acknowledged': True} + mock_es_conn.return_value = mock_es + + response = authenticated_client.post('/SNMP/DeployConfiguration/') + assert response.status_code == 200 + data = json.loads(response.content) + assert data['success'] is True + + @patch('SNMP.snmp_crud.get_elastic_connection') + def test_deploy_configuration_no_networks(self, mock_es_conn, authenticated_client): + """Test deploying with no networks configured""" + response = authenticated_client.post('/SNMP/DeployConfiguration/') + assert response.status_code == 400 + data = json.loads(response.content) + assert 'No networks configured' in data['error'] + + +# ============================================================================ +# Device Status and Visualization Tests +# ============================================================================ + +@pytest.mark.django_db +class TestDeviceStatusAndVisualization: + """Test device status checking and visualization endpoints""" + + @patch('SNMP.snmp_crud.get_elastic_connection') + def test_get_devices_status(self, mock_es_conn, authenticated_client, test_device): + """Test getting device status in batch""" + mock_es = MagicMock() + mock_es.search.return_value = { + 'aggregations': { + 'online_devices': { + 'buckets': [ + {'key': '192.168.1.100', 'doc_count': 10} + ] + } + } + } + mock_es_conn.return_value = mock_es + + response = authenticated_client.get(f'/SNMP/GetDevicesStatus/?device_ids={test_device.id}') + assert response.status_code == 200 + data = json.loads(response.content) + assert data['success'] is True + assert 'statuses' in data + assert data['statuses'][str(test_device.id)]['is_online'] is True + + search_kwargs = mock_es.search.call_args.kwargs + assert search_kwargs['index'] == 'metrics-snmp*' + assert search_kwargs['aggregations']['online_devices']['terms']['field'] == 'host.polled_address' + + @patch('SNMP.snmp_crud.get_elastic_connection') + def test_get_devices_status_hostname_only_device( + self, mock_es_conn, authenticated_client, test_network, test_credential_v2c + ): + """Hostname-only devices are matched on host.polled_address, not IP.""" + device = Device.objects.create( + name='Linux', + hostname='linux_host.lab', + ip_address=None, + port=1161, + credential=test_credential_v2c, + network=test_network, + ) + mock_es = MagicMock() + mock_es.search.return_value = { + 'aggregations': { + 'online_devices': { + 'buckets': [ + {'key': 'linux_host.lab', 'doc_count': 4} + ] + } + } + } + mock_es_conn.return_value = mock_es + + response = authenticated_client.get(f'/SNMP/GetDevicesStatus/?device_ids={device.id}') + assert response.status_code == 200 + data = json.loads(response.content) + assert data['statuses'][str(device.id)]['is_online'] is True + + terms_filter = mock_es.search.call_args.kwargs['query']['bool']['filter'][1] + assert terms_filter == {'terms': {'host.polled_address': ['linux_host.lab']}} + + def test_get_devices_status_invalid_ids(self, authenticated_client): + """Test getting device status with invalid IDs""" + response = authenticated_client.get('/SNMP/GetDevicesStatus/?device_ids=invalid') + assert response.status_code == 400 + + @patch('SNMP.snmp_crud.generate_visualizations') + @patch('SNMP.snmp_crud.get_elastic_connection') + def test_get_device_visualization(self, mock_es_conn, mock_gen_viz, authenticated_client, test_device): + """Test getting device visualization data""" + mock_es = MagicMock() + mock_es.search.return_value = { + 'aggregations': { + 'data_kinds': { + 'buckets': [ + {'key': 'metric', 'doc_count': 100} + ] + } + } + } + mock_es_conn.return_value = mock_es + + # Mock the visualization generation to return simple data + mock_gen_viz.return_value = { + 'charts': [], + 'has_data': True + } + + response = authenticated_client.get(f'/SNMP/GetDeviceVisualization/{test_device.id}/') + assert response.status_code == 200 + data = json.loads(response.content) + assert data['success'] is True + assert 'device' in data + assert 'visualizations' in data + + @patch('SNMP.snmp_crud.get_elastic_connection') + def test_get_discovered_devices(self, mock_es_conn, authenticated_client, test_connection, test_network): + """Test getting discovered devices from Elasticsearch""" + mock_es = MagicMock() + mock_es.search.return_value = { + 'aggregations': { + 'devices_by_host': { + 'buckets': [ + { + 'key': 'device1', + 'latest_doc': { + 'hits': { + 'hits': [ + { + '_source': { + 'host': {'name': 'device1', 'hostname': '192.168.1.50'}, + 'network': {'name': 'Test Network'}, + '@timestamp': '2024-01-01T00:00:00Z' + } + } + ] + } + } + } + ] + } + } + } + mock_es_conn.return_value = mock_es + + response = authenticated_client.get('/SNMP/DiscoveredDevices/') + assert response.status_code == 200 + data = json.loads(response.content) + assert data['success'] is True + assert 'devices' in data + + +# ============================================================================ +# Edge Cases and Error Handling +# ============================================================================ + +@pytest.mark.django_db +class TestEdgeCasesAndErrors: + """Test edge cases and error handling""" + + def test_unauthenticated_access_denied(self, client): + """Test that unauthenticated requests are denied""" + response = client.get('/SNMP/GetCredentials/') + assert response.status_code == 302 + assert '/Management/Login/' in response.url + + def test_credential_encryption(self, authenticated_client): + """Test that credentials are encrypted when saved""" + response = authenticated_client.post('/SNMP/AddCredential/', { + 'name': 'Encryption Test', + 'version': '2c', + 'community': 'secret' + }) + assert response.status_code == 200 + + # Verify community is encrypted in database + credential = Credential.objects.get(name='Encryption Test') + # Encrypted value should start with 'gAAAAA' (Fernet token) + assert credential.community.startswith('gAAAAA') + # But decrypted value should be original + assert credential.get_community() == 'secret' + + def test_network_cidr_validation(self, authenticated_client): + """Test CIDR validation for networks""" + # Valid CIDR within the /20 size limit + response = authenticated_client.post('/SNMP/AddNetwork/', { + 'name': 'Valid CIDR', + 'network_range': '10.0.0.0/24', + }) + assert response.status_code == 200 + + # Networks larger than /20 are rejected (would OOM during discovery) + response = authenticated_client.post('/SNMP/AddNetwork/', { + 'name': 'Too Large CIDR', + 'network_range': '10.0.0.0/8', + }) + assert response.status_code == 400 + data = response.json() + assert not data['success'] + assert 'too large' in data['message'].lower() + + # Invalid CIDR is rejected by model validation + response = authenticated_client.post('/SNMP/AddNetwork/', { + 'name': 'Invalid CIDR', + 'network_range': '999.999.999.999/99', + }) + assert response.status_code == 400 + + def test_device_ip_validation(self, authenticated_client, test_network, test_credential_v2c): + """Test IP address validation for devices""" + # Valid IP address + response = authenticated_client.post('/SNMP/AddDevice/', { + 'name': 'Valid IP Device', + 'ip_address': '192.168.1.1', + 'network': test_network.id, + 'credential': test_credential_v2c.id + }) + assert response.status_code == 200 + + # Hostname in ip_address is no longer valid; ip_address must be a valid IP. + # Hostnames should be set in the 'hostname' field instead. + response = authenticated_client.post('/SNMP/AddDevice/', { + 'name': 'Hostname Device', + 'ip_address': 'router.example.com', + 'network': test_network.id, + 'credential': test_credential_v2c.id + }) + assert response.status_code == 400 + + def test_profile_json_validation(self, authenticated_client): + """Test that profile_data must be valid JSON object""" + # Valid JSON object — vendor is now required + response = authenticated_client.post('/SNMP/AddProfile/', + json.dumps({ + 'name': 'valid_json_profile', + 'vendor': 'Generic', + 'profile_data': {'get': {}, 'walk': {}} + }), + content_type='application/json' + ) + assert response.status_code == 200 + + @patch('SNMP.snmp_crud.get_elastic_connection') + def test_deploy_handles_elasticsearch_errors(self, mock_es_conn, authenticated_client, test_network, test_device): + """Test that deploy handles Elasticsearch errors gracefully""" + mock_es_conn.side_effect = Exception("Connection failed") + + response = authenticated_client.post('/SNMP/DeployConfiguration/') + # Should return error but not crash + assert response.status_code in [400, 500] + + +# ============================================================================ +# Pure-function unit tests for snmp_crud.py helpers +# ============================================================================ + +@pytest.mark.django_db +class TestGetPipelineName: + """Tests for _get_pipeline_name() helper""" + + def test_basic_name_generation(self, test_network): + from SNMP.snmp_crud import _get_pipeline_name + name = _get_pipeline_name(test_network) + assert name.startswith('snmp-') + # network name is 'Test Network' — spaces become underscores via sanitizer + assert 'test_network' in name + + def test_special_chars_sanitized(self, test_connection, test_credential_v2c): + """Special chars in network name are sanitized""" + from SNMP.snmp_crud import _get_pipeline_name + network = Network.objects.create( + name='My Network (prod)!', + network_range='10.0.0.0/24', + connection=test_connection, + ) + name = _get_pipeline_name(network) + # Pipeline names must not contain special chars + import re + assert re.match(r'^[a-z0-9_\-]+$', name), f"Bad pipeline name: {name}" + + +@pytest.mark.django_db +class TestCreateOrUpdatePipeline: + """Tests for _create_or_update_pipeline() helper""" + + def test_creates_new_pipeline(self): + from SNMP.snmp_crud import _create_or_update_pipeline + mock_es = MagicMock() + mock_es.logstash.get_pipeline.side_effect = Exception("not found") + mock_es.logstash.put_pipeline.return_value = {} + + success, is_new, error, was_updated = _create_or_update_pipeline( + mock_es, 'test-pipe', 'input {} filter {} output {}' + ) + assert success is True + assert is_new is True + assert error is None + assert was_updated is True + mock_es.logstash.put_pipeline.assert_called_once() + + def test_updates_existing_pipeline_when_content_changed(self): + from SNMP.snmp_crud import _create_or_update_pipeline + mock_es = MagicMock() + mock_es.logstash.get_pipeline.return_value = { + 'test-pipe': { + 'pipeline': 'input {} filter {} output { old_output }', + 'pipeline_settings': {'queue.type': 'memory'}, + 'pipeline_metadata': {'version': 2, 'type': 'logstash_pipeline'}, + } + } + mock_es.logstash.put_pipeline.return_value = {} + + success, is_new, error, was_updated = _create_or_update_pipeline( + mock_es, 'test-pipe', 'input {} filter {} output { new_output }' + ) + assert success is True + assert is_new is False + assert was_updated is True + mock_es.logstash.put_pipeline.assert_called_once() + + def test_skips_update_when_content_identical(self): + from SNMP.snmp_crud import _create_or_update_pipeline + content = 'input {} filter {} output {}' + mock_es = MagicMock() + mock_es.logstash.get_pipeline.return_value = { + 'test-pipe': { + 'pipeline': content, + 'pipeline_settings': {}, + 'pipeline_metadata': {}, + } + } + + success, is_new, error, was_updated = _create_or_update_pipeline( + mock_es, 'test-pipe', content + ) + assert success is True + assert is_new is False + assert was_updated is False + mock_es.logstash.put_pipeline.assert_not_called() + + def test_returns_false_on_put_exception(self): + from SNMP.snmp_crud import _create_or_update_pipeline + mock_es = MagicMock() + mock_es.logstash.get_pipeline.side_effect = Exception("not found") + mock_es.logstash.put_pipeline.side_effect = Exception("ES write error") + + success, is_new, error, was_updated = _create_or_update_pipeline( + mock_es, 'test-pipe', 'input {} filter {} output {}' + ) + assert success is False + assert error is not None + assert 'ES write error' in error + + def test_new_pipeline_uses_default_settings(self): + from SNMP.snmp_crud import _create_or_update_pipeline + mock_es = MagicMock() + mock_es.logstash.get_pipeline.side_effect = Exception("not found") + mock_es.logstash.put_pipeline.return_value = {} + + _create_or_update_pipeline(mock_es, 'new-pipe', 'input {}') + call_body = mock_es.logstash.put_pipeline.call_args[1]['body'] + assert 'pipeline_settings' in call_body + assert call_body['pipeline_settings']['queue.type'] == 'memory' + + def test_existing_pipeline_preserves_settings(self): + from SNMP.snmp_crud import _create_or_update_pipeline + custom_settings = {'queue.type': 'persisted', 'pipeline.workers': 4} + mock_es = MagicMock() + mock_es.logstash.get_pipeline.return_value = { + 'test-pipe': { + 'pipeline': 'old content', + 'pipeline_settings': custom_settings, + 'pipeline_metadata': {'version': 5}, + } + } + mock_es.logstash.put_pipeline.return_value = {} + + _create_or_update_pipeline(mock_es, 'test-pipe', 'new content') + call_body = mock_es.logstash.put_pipeline.call_args[1]['body'] + assert call_body['pipeline_settings'] == custom_settings + + +@pytest.mark.django_db +class TestGetDeviceProfiles: + """Tests for _get_device_profiles() helper (lives in snmp_pipeline_generator)""" + + def test_no_template_returns_empty(self, test_network, test_credential_v2c): + from SNMP.snmp_pipeline_generator import _get_device_profiles + device = Device.objects.create( + name='No Template Device', ip_address='10.0.0.1', + credential=test_credential_v2c, network=test_network + ) + profile_ids, merged, normalizers = _get_device_profiles(device, {}) + assert profile_ids == tuple() + assert merged == {'get': {}, 'walk': {}, 'table': {}} + assert normalizers == [] + + def test_custom_profile_oids_merged(self, test_network, test_credential_v2c): + from SNMP.snmp_pipeline_generator import _get_device_profiles + profile = Profile.objects.create( + name='custom_test', + vendor='Generic', + profile_data={ + 'get': {'system.name': '1.3.6.1.2.1.1.5.0'}, + 'walk': {}, + 'table': {} + } + ) + template = DeviceTemplate.objects.create(name='Test Template', vendor='Generic') + template.profiles.add(profile) + device = Device.objects.create( + name='Profile Device', ip_address='10.0.0.2', + credential=test_credential_v2c, network=test_network, + device_template=template + ) + + profile_ids, merged, normalizers = _get_device_profiles(device, {}) + assert len(profile_ids) == 1 + assert '1.3.6.1.2.1.1.5.0' in merged['get'].values() + + def test_official_placeholder_loaded_from_file(self, test_network, test_credential_v2c): + from SNMP.snmp_pipeline_generator import _get_device_profiles + profile = Profile.objects.create( + name='test_official.json', + vendor='Generic', + profile_data={'is_official_placeholder': True}, + ) + template = DeviceTemplate.objects.create(name='Official Template', vendor='Generic') + template.profiles.add(profile) + device = Device.objects.create( + name='Official Device', ip_address='10.0.0.3', + credential=test_credential_v2c, network=test_network, + device_template=template + ) + + fake_data = {'get': {'system.desc': '1.3.6.1.2.1.1.1.0'}, 'walk': {}, 'table': {}} + with patch('SNMP.snmp_pipeline_generator.os.path.exists', return_value=True), \ + patch('builtins.open', create=True) as mock_open, \ + patch('SNMP.snmp_pipeline_generator.json.load', return_value=fake_data): + mock_open.return_value.__enter__ = lambda s: s + mock_open.return_value.__exit__ = MagicMock(return_value=False) + profile_ids, merged, normalizers = _get_device_profiles(device, {}) + + assert '1.3.6.1.2.1.1.1.0' in merged['get'].values() + + def test_official_placeholder_file_missing_skipped(self, test_network, test_credential_v2c): + from SNMP.snmp_pipeline_generator import _get_device_profiles + profile = Profile.objects.create( + name='missing_official.json', + vendor='Generic', + profile_data={'is_official_placeholder': True}, + ) + template = DeviceTemplate.objects.create(name='Missing File Template', vendor='Generic') + template.profiles.add(profile) + device = Device.objects.create( + name='Missing File Device', ip_address='10.0.0.4', + credential=test_credential_v2c, network=test_network, + device_template=template + ) + + with patch('SNMP.snmp_pipeline_generator.os.path.exists', return_value=False): + profile_ids, merged, normalizers = _get_device_profiles(device, {}) + + assert merged == {'get': {}, 'walk': {}, 'table': {}} + + def test_oid_conflict_gets_suffixed(self, test_network, test_credential_v2c): + """When two profiles define the same OID key with different values, a suffix is added""" + from SNMP.snmp_pipeline_generator import _get_device_profiles + profile_a = Profile.objects.create( + name='profile_a', vendor='Generic', + profile_data={'get': {'metric': 'oid.1'}, 'walk': {}, 'table': {}} + ) + profile_b = Profile.objects.create( + name='profile_b', vendor='Generic', + profile_data={'get': {'metric': 'oid.2'}, 'walk': {}, 'table': {}} + ) + template = DeviceTemplate.objects.create(name='Conflict Template', vendor='Generic') + template.profiles.add(profile_a, profile_b) + device = Device.objects.create( + name='Conflict Device', ip_address='10.0.0.5', + credential=test_credential_v2c, network=test_network, + device_template=template + ) + + _, merged, _ = _get_device_profiles(device, {}) + assert len(merged['get']) == 2 + + +@pytest.mark.django_db +class TestFormatFieldName: + """Tests for _format_field_name() pure function""" + + def test_already_bracket_notation_unchanged(self): + from SNMP.snmp_pipeline_generator import _format_field_name + assert _format_field_name('[system][cpu]') == '[system][cpu]' + + def test_dotted_name_converted(self): + from SNMP.snmp_pipeline_generator import _format_field_name + assert _format_field_name('system.cpu.load') == '[system][cpu][load]' + + def test_plain_name_wrapped_in_brackets(self): + # In snmp_pipeline_generator, plain names without dots are wrapped in [brackets] + from SNMP.snmp_pipeline_generator import _format_field_name + assert _format_field_name('hostname') == '[hostname]' + + def test_single_dot(self): + from SNMP.snmp_pipeline_generator import _format_field_name + assert _format_field_name('a.b') == '[a][b]' + + +@pytest.mark.django_db +class TestGetDiscoveryIpAddresses: + """Tests for _get_discovery_ip_addresses() helper""" + + def test_returns_all_hosts_in_range(self, test_network): + from SNMP.snmp_pipeline_generator import _get_discovery_ip_addresses + # /30 has 2 usable hosts + test_network.network_range = '192.168.100.0/30' + test_network.save() + ips = _get_discovery_ip_addresses(test_network) + assert '192.168.100.1' in ips + assert '192.168.100.2' in ips + assert '192.168.100.0' not in ips # network address + assert '192.168.100.3' not in ips # broadcast + + def test_excludes_existing_device_ips(self, test_network, test_credential_v2c): + from SNMP.snmp_pipeline_generator import _get_discovery_ip_addresses + test_network.network_range = '10.0.0.0/30' + test_network.save() + Device.objects.create( + name='Existing', ip_address='10.0.0.1', + credential=test_credential_v2c, network=test_network + ) + ips = _get_discovery_ip_addresses(test_network) + assert '10.0.0.1' not in ips + assert '10.0.0.2' in ips + + def test_legacy_hostname_ip_values_not_excluded(self, test_network): + """If legacy data has a non-IP value in ip_address, the function skips it gracefully. + + The Device model now validates that ip_address must be a valid IP, so this + scenario can only occur with legacy data. We test it using a mocked queryset. + """ + from SNMP.snmp_pipeline_generator import _get_discovery_ip_addresses + test_network.network_range = '10.0.1.0/30' + test_network.save() + + with patch('SNMP.snmp_pipeline_generator.Device.objects') as mock_objs: + mock_objs.filter.return_value.values_list.return_value = ['router.example.com'] + ips = _get_discovery_ip_addresses(test_network) + + # Hostname in ip_address should be skipped; both usable IPs remain + assert '10.0.1.1' in ips + assert '10.0.1.2' in ips + + def test_invalid_cidr_returns_empty(self): + """Invalid CIDR can't be saved to DB (model validates it), so use a Mock.""" + from SNMP.snmp_pipeline_generator import _get_discovery_ip_addresses + from unittest.mock import MagicMock + fake_network = MagicMock() + fake_network.network_range = 'not-a-cidr' + fake_network.name = 'Fake' + ips = _get_discovery_ip_addresses(fake_network) + assert ips == [] + + +@pytest.mark.django_db +class TestGetCredentialEndpointV3: + """Additional GetCredential tests for v3 fields""" + + def test_get_credential_v3_returns_security_fields(self, authenticated_client, test_credential_v3): + """v3 credential response includes security_name, security_level, auth_protocol""" + response = authenticated_client.get(f'/SNMP/GetCredential/{test_credential_v3.id}/') + assert response.status_code == 200 + data = json.loads(response.content) + assert data['version'] == '3' + assert data['security_name'] == 'snmpuser' + assert data['security_level'] == 'authPriv' + assert data['auth_protocol'] == 'sha' + # Passwords must be masked + assert data['auth_pass'] == '***' + assert data['priv_pass'] == '***' + + def test_get_credential_v3_authnopriv_no_priv_fields(self, authenticated_client): + """authNoPriv credential has no priv fields in response""" + cred = Credential.objects.create( + name='v3_authNoPriv', + version='3', + security_name='user', + security_level='authNoPriv', + auth_protocol='sha', + auth_pass='authpass', + ) + response = authenticated_client.get(f'/SNMP/GetCredential/{cred.id}/') + data = json.loads(response.content) + assert 'priv_pass' not in data + assert 'auth_protocol' in data + + +@pytest.mark.django_db +class TestGetNetworkEndpointEdgeCases: + """Tests for GetNetwork, UpdateNetwork, GetNetworkPipelineName error paths""" + + def test_get_network_not_found(self, authenticated_client): + response = authenticated_client.get('/SNMP/GetNetwork/99999/') + assert response.status_code == 404 + assert 'error' in json.loads(response.content) + + def test_update_network_not_found(self, authenticated_client): + response = authenticated_client.post('/SNMP/UpdateNetwork/99999/', { + 'name': 'Ghost', 'network_range': '10.0.0.0/24' + }) + assert response.status_code == 404 + + def test_get_network_pipeline_name_not_found(self, authenticated_client): + response = authenticated_client.get('/SNMP/GetNetworkPipelineName/99999/') + assert response.status_code == 404 + data = json.loads(response.content) + assert data['success'] is False + + def test_update_network_clears_optional_fields_when_empty(self, authenticated_client, test_network): + """Passing empty connection/credential nullifies those FK fields""" + response = authenticated_client.post(f'/SNMP/UpdateNetwork/{test_network.id}/', { + 'name': test_network.name, + 'network_range': test_network.network_range, + 'connection': '', + 'discovery_credential': '', + 'credential': '', + }) + assert response.status_code == 200 + test_network.refresh_from_db() + assert test_network.connection is None + assert test_network.discovery_credential is None + assert test_network.credential is None + + +@pytest.mark.django_db +class TestDeleteNetworkPipelinePaths: + """Tests for DeleteNetwork pipeline deletion branches""" + + @patch('SNMP.snmp_crud.get_elastic_connection') + def test_delete_network_with_pipeline_deleted_reports_it(self, mock_get_es, authenticated_client, test_network): + """When both pipelines exist and are deleted, success message mentions them""" + mock_es = MagicMock() + # make get_pipeline return the pipeline as existing + def get_pipeline_side_effect(id): + return {id: {'pipeline': 'content'}} + mock_es.logstash.get_pipeline.side_effect = get_pipeline_side_effect + mock_es.logstash.delete_pipeline.return_value = {} + mock_get_es.return_value = mock_es + + response = authenticated_client.post(f'/SNMP/DeleteNetwork/{test_network.id}/') + assert response.status_code == 200 + data = json.loads(response.content) + assert data['success'] is True + assert 'pipeline' in data['message'].lower() + + @patch('SNMP.snmp_crud.get_elastic_connection') + def test_delete_network_connection_error_still_deletes_db_record( + self, mock_get_es, authenticated_client, test_network): + """Even if ES connection fails, the DB record is deleted and success=True returned""" + mock_get_es.side_effect = Exception("ES connection failed") + network_id = test_network.id + + response = authenticated_client.post(f'/SNMP/DeleteNetwork/{network_id}/') + assert response.status_code == 200 + data = json.loads(response.content) + assert data['success'] is True + # DB record should be gone + assert not Network.objects.filter(id=network_id).exists() + + def test_delete_network_without_connection_skips_es(self, authenticated_client, test_credential_v2c): + """Network with no connection skips ES interaction and deletes cleanly""" + network = Network.objects.create( + name='No Conn Network', + network_range='172.16.0.0/24', + ) + network_id = network.id + response = authenticated_client.post(f'/SNMP/DeleteNetwork/{network_id}/') + assert response.status_code == 200 + data = json.loads(response.content) + assert data['success'] is True + assert not Network.objects.filter(id=network_id).exists() + + +class TestDefaultTemplateAssignment: + """Tests for automatic Default template assignment (Device.save / DeviceTemplate.delete)""" + + @pytest.fixture + def default_template(self, db): + """Create the official default template as synced from default.json""" + return DeviceTemplate.objects.create( + name='default', + description='Fallback template applied when no other template matches.', + vendor='Any', + official=True + ) + + def test_device_without_template_gets_default(self, default_template, test_network, test_credential_v2c): + """A device saved with no template is auto-assigned the official default template""" + device = Device.objects.create( + name='No Template Device', + ip_address='192.168.1.150', + credential=test_credential_v2c, + network=test_network + ) + assert device.device_template == default_template + + def test_device_keeps_explicit_template(self, default_template, test_network, test_credential_v2c): + """A device saved with an explicit template is not reassigned to default""" + other_template = DeviceTemplate.objects.create( + name='custom_template', + vendor='Any', + official=False + ) + device = Device.objects.create( + name='Templated Device', + ip_address='192.168.1.151', + credential=test_credential_v2c, + network=test_network, + device_template=other_template + ) + assert device.device_template == other_template + + def test_default_template_cannot_be_deleted(self, default_template): + """The official default template is protected from deletion""" + from django.core.exceptions import ValidationError + with pytest.raises(ValidationError): + default_template.delete() + + def test_deleting_template_reassigns_devices_to_default(self, default_template, test_network, test_credential_v2c): + """Deleting a template moves its devices onto the default template""" + doomed_template = DeviceTemplate.objects.create( + name='doomed_template', + vendor='Any', + official=False + ) + device = Device.objects.create( + name='Orphaned Device', + ip_address='192.168.1.152', + credential=test_credential_v2c, + network=test_network, + device_template=doomed_template + ) + doomed_template.delete() + device.refresh_from_db() + assert device.device_template == default_template + + +# ============================================================================ +# DeviceTemplate CRUD Endpoint Tests +# ============================================================================ + +@pytest.fixture +def test_device_template(db): + """Create a custom (non-official) device template.""" + return DeviceTemplate.objects.create( + name='custom_template', + description='A custom test template', + vendor='Cisco', + model='9300', + product='Catalyst', + official=False, + matching_rules=['cisco', 'catalyst'], + ) + + +@pytest.fixture +def official_template(db): + """Create an official (read-only) device template.""" + return DeviceTemplate.objects.create( + name='official_template', + description='Official template', + vendor='Dell', + official=True, + ) + + +@pytest.mark.django_db +class TestDeviceTemplateCRUD: + """Test DeviceTemplate Create, Read, Update, Delete operations via API endpoints.""" + + # ── GetDeviceTemplates ──────────────────────────────────────────────────── + + def test_get_device_templates_returns_list(self, authenticated_client, test_device_template): + response = authenticated_client.get('/SNMP/GetDeviceTemplates/') + assert response.status_code == 200 + data = json.loads(response.content) + assert 'templates' in data + names = [t['name'] for t in data['templates']] + assert 'custom_template' in names + + def test_get_device_templates_includes_required_fields(self, authenticated_client, test_device_template): + response = authenticated_client.get('/SNMP/GetDeviceTemplates/') + data = json.loads(response.content) + template = next(t for t in data['templates'] if t['name'] == 'custom_template') + for field in ('id', 'name', 'display_name', 'vendor', 'model', 'product', 'official'): + assert field in template + + def test_get_device_templates_display_name_formatted(self, authenticated_client, test_device_template): + response = authenticated_client.get('/SNMP/GetDeviceTemplates/') + data = json.loads(response.content) + template = next(t for t in data['templates'] if t['name'] == 'custom_template') + assert template['display_name'] == 'Custom Template' + + # ── GetDeviceTemplate ──────────────────────────────────────────────────── + + def test_get_device_template_by_id(self, authenticated_client, test_device_template): + response = authenticated_client.get(f'/SNMP/GetDeviceTemplate/{test_device_template.id}/') + assert response.status_code == 200 + data = json.loads(response.content) + assert data['name'] == 'custom_template' + assert data['vendor'] == 'Cisco' + assert 'profiles' in data + assert 'matching_rules' in data + + def test_get_device_template_not_found(self, authenticated_client): + response = authenticated_client.get('/SNMP/GetDeviceTemplate/99999/') + # Falls back to GetOfficialDeviceTemplate which returns 404 for unknown names + assert response.status_code in (404, 200) + + def test_get_device_template_includes_profiles(self, authenticated_client, test_device_template, test_profile): + test_device_template.profiles.add(test_profile) + response = authenticated_client.get(f'/SNMP/GetDeviceTemplate/{test_device_template.id}/') + data = json.loads(response.content) + assert any(p['name'] == test_profile.name for p in data['profiles']) + + # ── AddDeviceTemplate ──────────────────────────────────────────────────── + + def test_add_device_template_requires_admin(self, readonly_client): + response = readonly_client.post('/SNMP/AddDeviceTemplate/', { + 'name': 'new_tmpl', + 'vendor': 'Cisco', + }) + assert response.status_code == 403 + + def test_add_device_template_success(self, authenticated_client): + import json as _json + response = authenticated_client.post('/SNMP/AddDeviceTemplate/', { + 'name': 'brand_new_template', + 'description': 'Test', + 'vendor': 'Juniper', + 'model': 'EX2300', + 'product': 'EX', + 'matching_rules': _json.dumps(['juniper', 'ex']), + 'profiles': _json.dumps([]), + }) + assert response.status_code == 200 + data = json.loads(response.content) + assert 'template_id' in data + assert DeviceTemplate.objects.filter(name='brand_new_template').exists() + + def test_add_device_template_missing_name(self, authenticated_client): + import json as _json + response = authenticated_client.post('/SNMP/AddDeviceTemplate/', { + 'vendor': 'Cisco', + 'matching_rules': _json.dumps([]), + 'profiles': _json.dumps([]), + }) + assert response.status_code == 400 + + def test_add_device_template_missing_vendor(self, authenticated_client): + import json as _json + response = authenticated_client.post('/SNMP/AddDeviceTemplate/', { + 'name': 'no_vendor_tmpl', + 'matching_rules': _json.dumps([]), + 'profiles': _json.dumps([]), + }) + assert response.status_code == 400 + + def test_add_device_template_with_profile_ids(self, authenticated_client, test_profile): + import json as _json + response = authenticated_client.post('/SNMP/AddDeviceTemplate/', { + 'name': 'with_profiles', + 'vendor': 'Generic', + 'matching_rules': _json.dumps([]), + 'profiles': _json.dumps([test_profile.id]), + }) + assert response.status_code == 200 + tmpl = DeviceTemplate.objects.get(name='with_profiles') + assert tmpl.profiles.filter(id=test_profile.id).exists() + + # ── UpdateDeviceTemplate ───────────────────────────────────────────────── + + def test_update_device_template_requires_admin(self, readonly_client, test_device_template): + import json as _json + response = readonly_client.post( + f'/SNMP/UpdateDeviceTemplate/{test_device_template.id}/', + { + 'name': 'hacked_name', + 'vendor': 'X', + 'matching_rules': _json.dumps([]), + 'profiles': _json.dumps([]), + }, + ) + assert response.status_code == 403 + + def test_update_device_template_success(self, authenticated_client, test_device_template): + import json as _json + response = authenticated_client.post( + f'/SNMP/UpdateDeviceTemplate/{test_device_template.id}/', + { + 'name': 'updated_template', + 'vendor': 'HPE', + 'model': 'ProLiant', + 'description': 'Updated desc', + 'matching_rules': _json.dumps(['hpe']), + 'profiles': _json.dumps([]), + }, + ) + assert response.status_code == 200 + test_device_template.refresh_from_db() + assert test_device_template.name == 'updated_template' + assert test_device_template.vendor == 'HPE' + + def test_update_official_template_rejected(self, authenticated_client, official_template): + import json as _json + response = authenticated_client.post( + f'/SNMP/UpdateDeviceTemplate/{official_template.id}/', + { + 'name': 'hacked_official', + 'vendor': 'X', + 'matching_rules': _json.dumps([]), + 'profiles': _json.dumps([]), + }, + ) + assert response.status_code == 403 + + def test_update_device_template_not_found(self, authenticated_client): + import json as _json + response = authenticated_client.post( + '/SNMP/UpdateDeviceTemplate/99999/', + { + 'name': 'ghost', + 'vendor': 'X', + 'matching_rules': _json.dumps([]), + 'profiles': _json.dumps([]), + }, + ) + assert response.status_code == 404 + + # ── DeleteDeviceTemplate ────────────────────────────────────────────────── + + def test_delete_device_template_requires_admin(self, readonly_client, test_device_template): + response = readonly_client.post(f'/SNMP/DeleteDeviceTemplate/{test_device_template.id}/') + assert response.status_code == 403 + + def test_delete_device_template_success(self, authenticated_client, test_device_template): + tmpl_id = test_device_template.id + response = authenticated_client.post(f'/SNMP/DeleteDeviceTemplate/{tmpl_id}/') + assert response.status_code == 200 + assert not DeviceTemplate.objects.filter(id=tmpl_id).exists() + + def test_delete_official_template_rejected(self, authenticated_client, official_template): + response = authenticated_client.post(f'/SNMP/DeleteDeviceTemplate/{official_template.id}/') + assert response.status_code == 403 + + def test_delete_device_template_not_found(self, authenticated_client): + response = authenticated_client.post('/SNMP/DeleteDeviceTemplate/99999/') + assert response.status_code == 404 + + +# ============================================================================ +# suggest_device_template +# ============================================================================ + +@pytest.mark.django_db +class TestSuggestDeviceTemplate: + """Test the suggest_device_template pure function in snmp_crud.""" + + @pytest.fixture(autouse=True) + def clear_templates(self, db): + """Ensure no leftover templates pollute suggestion results.""" + DeviceTemplate.objects.all().delete() + + def _make_template(self, name, rules): + return DeviceTemplate.objects.create( + name=name, vendor='Any', matching_rules=rules + ) + + def test_empty_device_info_returns_empty(self): + from SNMP.snmp_crud import suggest_device_template + assert suggest_device_template('') == [] + assert suggest_device_template(None) == [] + + def test_all_rules_match_returns_full_match(self): + from SNMP.snmp_crud import suggest_device_template + tmpl = self._make_template('cisco_cat', ['cisco', 'catalyst']) + result = suggest_device_template('Cisco Catalyst 9300 switch') + assert tmpl.id in result + assert result.index(tmpl.id) == 0 # full match first + + def test_partial_match_returned(self): + from SNMP.snmp_crud import suggest_device_template + tmpl = self._make_template('cisco_any', ['cisco', 'nexus']) + result = suggest_device_template('Cisco Catalyst switch') # 'cisco' matches, 'nexus' doesn't + assert tmpl.id in result + + def test_no_match_excluded(self): + from SNMP.snmp_crud import suggest_device_template + self._make_template('juniper_tmpl', ['juniper', 'ex']) + result = suggest_device_template('Cisco Catalyst 9300') + assert result == [] + + def test_full_match_ranked_before_partial(self): + from SNMP.snmp_crud import suggest_device_template + full = self._make_template('full_match', ['cisco', 'catalyst']) + partial = self._make_template('partial_match', ['cisco', 'nexus']) + result = suggest_device_template('Cisco Catalyst switch') + assert result.index(full.id) < result.index(partial.id) + + def test_case_insensitive_matching(self): + from SNMP.snmp_crud import suggest_device_template + tmpl = self._make_template('caps_tmpl', ['CISCO']) + result = suggest_device_template('cisco ios router') + assert tmpl.id in result + + def test_template_without_rules_excluded(self): + from SNMP.snmp_crud import suggest_device_template + self._make_template('no_rules', []) + result = suggest_device_template('cisco ios') + assert result == [] + + +# ============================================================================ +# GetDeviceLocationData +# ============================================================================ + +@pytest.mark.django_db +class TestGetDeviceLocationData: + """Test the /SNMP/GetDeviceLocationData/ endpoint.""" + + def test_requires_authentication(self, client): + response = client.get('/SNMP/GetDeviceLocationData/') + assert response.status_code == 302 + + def test_returns_empty_lists_when_no_devices(self, authenticated_client): + Device.objects.all().delete() + response = authenticated_client.get('/SNMP/GetDeviceLocationData/') + assert response.status_code == 200 + data = json.loads(response.content) + assert data['sites'] == [] + assert data['site_building'] == [] + assert data['full'] == [] + + def test_returns_sites(self, authenticated_client, test_credential_v2c, test_network): + Device.objects.create( + name='dev_site1', + ip_address='10.0.0.1', + credential=test_credential_v2c, + network=test_network, + site='HQ', + ) + response = authenticated_client.get('/SNMP/GetDeviceLocationData/') + data = json.loads(response.content) + assert 'HQ' in data['sites'] + + def test_site_building_pairs(self, authenticated_client, test_credential_v2c, test_network): + Device.objects.create( + name='dev_sb', + ip_address='10.0.0.2', + credential=test_credential_v2c, + network=test_network, + site='Campus A', + building='Bldg 1', + ) + response = authenticated_client.get('/SNMP/GetDeviceLocationData/') + data = json.loads(response.content) + assert any( + sb['site'] == 'Campus A' and sb['building'] == 'Bldg 1' + for sb in data['site_building'] + ) + + def test_full_entries_with_coordinates(self, authenticated_client, test_credential_v2c, test_network): + Device.objects.create( + name='dev_full', + ip_address='10.0.0.3', + credential=test_credential_v2c, + network=test_network, + site='Site B', + building='Bldg 2', + room='Room 101', + latitude='37.774929', + longitude='-122.419418', + ) + response = authenticated_client.get('/SNMP/GetDeviceLocationData/') + data = json.loads(response.content) + entry = next((e for e in data['full'] if e['room'] == 'Room 101'), None) + assert entry is not None + # lat/lon must be serialised as strings (Decimal-safe) + assert isinstance(entry['latitude'], str) + assert isinstance(entry['longitude'], str) + + def test_devices_without_site_excluded_from_sites(self, authenticated_client, test_credential_v2c, test_network): + Device.objects.create( + name='dev_no_site', + ip_address='10.0.0.4', + credential=test_credential_v2c, + network=test_network, + site=None, + ) + response = authenticated_client.get('/SNMP/GetDeviceLocationData/') + data = json.loads(response.content) + assert None not in data['sites'] + + +# ============================================================================ +# GetDevices – additional coverage (pagination, sorting) +# ============================================================================ + +@pytest.mark.django_db +class TestGetDevicesAdditional: + """Additional GetDevices tests not covered by TestDeviceCRUD.""" + + def test_sort_by_name(self, authenticated_client, test_network, test_credential_v2c): + Device.objects.create(name='Zebra Device', ip_address='10.1.1.1', + credential=test_credential_v2c, network=test_network) + Device.objects.create(name='Alpha Device', ip_address='10.1.1.2', + credential=test_credential_v2c, network=test_network) + response = authenticated_client.get('/SNMP/GetDevices/?sort_by=name') + data = json.loads(response.content) + names = [d['name'] for d in data['devices']] + assert names == sorted(names) + + def test_sort_by_name_descending(self, authenticated_client, test_network, test_credential_v2c): + Device.objects.create(name='ZZZ Device', ip_address='10.1.2.1', + credential=test_credential_v2c, network=test_network) + Device.objects.create(name='AAA Device', ip_address='10.1.2.2', + credential=test_credential_v2c, network=test_network) + response = authenticated_client.get('/SNMP/GetDevices/?sort_by=-name') + data = json.loads(response.content) + names = [d['name'] for d in data['devices']] + assert names == sorted(names, reverse=True) + + def test_pagination_has_next(self, authenticated_client, test_network, test_credential_v2c): + """When more devices than page_size exist, has_next is True.""" + for i in range(5): + Device.objects.create( + name=f'Paged Device {i}', + ip_address=f'10.2.0.{i + 1}', + credential=test_credential_v2c, + network=test_network, + ) + response = authenticated_client.get('/SNMP/GetDevices/?page=1&page_size=2') + data = json.loads(response.content) + assert data['has_next'] is True + assert len(data['devices']) == 2 + + def test_pagination_page_2(self, authenticated_client, test_network, test_credential_v2c): + """Page 2 returns the next slice and has_previous is True.""" + for i in range(4): + Device.objects.create( + name=f'Page2 Device {i}', + ip_address=f'10.3.0.{i + 1}', + credential=test_credential_v2c, + network=test_network, + ) + response = authenticated_client.get('/SNMP/GetDevices/?page=2&page_size=2') + data = json.loads(response.content) + assert data['has_previous'] is True + + def test_search_by_ip(self, authenticated_client, test_network, test_credential_v2c): + Device.objects.create(name='IP Search Device', ip_address='172.16.100.1', + credential=test_credential_v2c, network=test_network) + response = authenticated_client.get('/SNMP/GetDevices/?search=172.16.100') + data = json.loads(response.content) + assert any(d['name'] == 'IP Search Device' for d in data['devices']) + + def test_get_device_returns_location_fields(self, authenticated_client, test_network, test_credential_v2c): + """GetDevice includes location and metadata fields.""" + device = Device.objects.create( + name='Location Device', + ip_address='10.5.5.5', + credential=test_credential_v2c, + network=test_network, + site='HQ', + building='Main', + room='A1', + metadata={'rack': '12'}, + ) + response = authenticated_client.get(f'/SNMP/GetDevice/{device.id}/') + data = json.loads(response.content) + assert data['site'] == 'HQ' + assert data['building'] == 'Main' + assert data['room'] == 'A1' + assert data['metadata'] == {'rack': '12'} + + def test_add_device_with_location_fields(self, authenticated_client, test_network, test_credential_v2c): + """AddDevice persists location and metadata fields.""" + import json as _json + response = authenticated_client.post('/SNMP/AddDevice/', { + 'name': 'Located Device', + 'ip_address': '10.6.6.6', + 'network': test_network.id, + 'credential': test_credential_v2c.id, + 'site': 'West Campus', + 'building': 'B1', + 'room': 'R2', + 'metadata': _json.dumps({'owner': 'infra'}), + }) + assert response.status_code == 200 + device = Device.objects.get(name='Located Device') + assert device.site == 'West Campus' + assert device.metadata == {'owner': 'infra'} + + def test_add_device_with_hostname(self, authenticated_client, test_network, test_credential_v2c): + """AddDevice accepts hostname-only devices (no IP).""" + response = authenticated_client.post('/SNMP/AddDevice/', { + 'name': 'Hostname Only Device', + 'hostname': 'myswitch.example.com', + 'network': test_network.id, + 'credential': test_credential_v2c.id, + }) + assert response.status_code == 200 + device = Device.objects.get(name='Hostname Only Device') + assert device.hostname == 'myswitch.example.com' + assert device.ip_address is None + + +# ============================================================================ +# Unit tests for _build_trap_components helper +# ============================================================================ + +@pytest.mark.django_db +class TestBuildTrapComponents: + """Direct unit tests for the _build_trap_components() internal helper""" + + def test_v2c_trap_components_have_correct_structure( + self, test_connection, test_credential_v2c + ): + from SNMP.snmp_crud import _build_trap_components + network = Network.objects.create( + name='Trap V2c Network', + network_range='10.10.0.0/24', + connection=test_connection, + credential=test_credential_v2c, + traps_enabled=True, + credential_mode='PLAINTEXT', + ) + result = _build_trap_components(network) + assert 'input' in result + assert 'filter' in result + assert 'output' in result + assert len(result['input']) == 1 + assert result['input'][0]['plugin'] == 'snmptrap' + trap_cfg = result['input'][0]['config'] + assert '2c' in trap_cfg.get('supported_versions', []) + + def test_v1_trap_components_include_v1_version( + self, test_connection, test_credential_v2c + ): + from SNMP.snmp_crud import _build_trap_components + cred = Credential.objects.create(name='V1 Trap Cred', version='1', community='public') + network = Network.objects.create( + name='Trap V1 Network', + network_range='10.11.0.0/24', + connection=test_connection, + credential=cred, + traps_enabled=True, + credential_mode='PLAINTEXT', + ) + result = _build_trap_components(network) + trap_cfg = result['input'][0]['config'] + assert '1' in trap_cfg.get('supported_versions', []) + + def test_v3_trap_components_include_security_fields( + self, test_connection, test_credential_v3 + ): + from SNMP.snmp_crud import _build_trap_components + network = Network.objects.create( + name='Trap V3 Network', + network_range='10.12.0.0/24', + connection=test_connection, + credential=test_credential_v3, + traps_enabled=True, + credential_mode='PLAINTEXT', + ) + result = _build_trap_components(network) + trap_cfg = result['input'][0]['config'] + assert '3' in trap_cfg.get('supported_versions', []) + assert 'security_name' in trap_cfg + + def test_keystore_mode_emits_keystore_references( + self, test_connection, test_credential_v2c + ): + from SNMP.snmp_crud import _build_trap_components + network = Network.objects.create( + name='Trap Keystore Network', + network_range='10.13.0.0/24', + connection=test_connection, + credential=test_credential_v2c, + traps_enabled=True, + credential_mode='KEYSTORE', + ) + result = _build_trap_components(network) + trap_cfg = result['input'][0]['config'] + # In KEYSTORE mode the community string should be a ${...} reference + community = trap_cfg.get('community', []) + assert community and community[0].startswith('${') + + def test_plaintext_mode_emits_decrypted_community( + self, test_connection, test_credential_v2c + ): + from SNMP.snmp_crud import _build_trap_components + network = Network.objects.create( + name='Trap Plaintext Network', + network_range='10.14.0.0/24', + connection=test_connection, + credential=test_credential_v2c, + traps_enabled=True, + credential_mode='PLAINTEXT', + ) + result = _build_trap_components(network) + trap_cfg = result['input'][0]['config'] + community = trap_cfg.get('community', []) + assert community and not community[0].startswith('${') + + def test_filter_adds_event_category_traps(self, test_connection, test_credential_v2c): + from SNMP.snmp_crud import _build_trap_components + network = Network.objects.create( + name='Trap Filter Check', + network_range='10.15.0.0/24', + connection=test_connection, + credential=test_credential_v2c, + traps_enabled=True, + credential_mode='PLAINTEXT', + ) + result = _build_trap_components(network) + mutate_filter = next( + (f for f in result['filter'] if f.get('plugin') == 'mutate'), None + ) + assert mutate_filter is not None + add_field = mutate_filter['config'].get('add_field', {}) + assert add_field.get('[event][category]') == 'traps' + + +# ============================================================================ +# Unit tests for _build_network_pipeline_configs helper +# ============================================================================ + +@pytest.mark.django_db +class TestBuildNetworkPipelineConfigs: + """Direct unit tests for the _build_network_pipeline_configs() internal helper""" + + def test_returns_empty_when_no_devices(self, test_connection, test_credential_v2c): + from SNMP.snmp_crud import _build_network_pipeline_configs + network = Network.objects.create( + name='Empty Network Configs', + network_range='10.20.0.0/24', + connection=test_connection, + traps_enabled=False, + discovery_enabled=False, + ) + results = _build_network_pipeline_configs(network) + assert results == [] + + def test_returns_polling_pipeline_for_v2c_device( + self, test_connection, test_credential_v2c + ): + from SNMP.snmp_crud import _build_network_pipeline_configs + network = Network.objects.create( + name='Polling V2c Configs', + network_range='10.21.0.0/24', + connection=test_connection, + traps_enabled=False, + discovery_enabled=False, + ) + Device.objects.create( + name='Config Test Device', + ip_address='10.21.0.10', + credential=test_credential_v2c, + network=network, + ) + results = _build_network_pipeline_configs(network) + assert len(results) >= 1 + pipeline_types = [r['pipeline_type'] for r in results] + assert 'polling' in pipeline_types + + def test_trap_pipeline_included_when_enabled( + self, test_connection, test_credential_v2c + ): + from SNMP.snmp_crud import _build_network_pipeline_configs + network = Network.objects.create( + name='Trap Enabled Configs', + network_range='10.22.0.0/24', + connection=test_connection, + credential=test_credential_v2c, + traps_enabled=True, + discovery_enabled=False, + credential_mode='PLAINTEXT', + ) + Device.objects.create( + name='Trap Config Device', + ip_address='10.22.0.10', + credential=test_credential_v2c, + network=network, + ) + results = _build_network_pipeline_configs(network) + pipeline_types = [r['pipeline_type'] for r in results] + assert 'trap' in pipeline_types + + def test_discovery_pipeline_included_when_enabled( + self, test_connection, test_credential_v2c + ): + from SNMP.snmp_crud import _build_network_pipeline_configs + network = Network.objects.create( + name='Discovery Enabled Configs', + network_range='10.23.0.0/24', + connection=test_connection, + discovery_credential=test_credential_v2c, + traps_enabled=False, + discovery_enabled=True, + credential_mode='PLAINTEXT', + ) + Device.objects.create( + name='Discovery Config Device', + ip_address='10.23.0.10', + credential=test_credential_v2c, + network=network, + ) + results = _build_network_pipeline_configs(network) + pipeline_types = [r['pipeline_type'] for r in results] + assert 'discovery' in pipeline_types + + def test_pipeline_name_contains_network_name( + self, test_connection, test_credential_v2c + ): + from SNMP.snmp_crud import _build_network_pipeline_configs + network = Network.objects.create( + name='Name Check Network', + network_range='10.24.0.0/24', + connection=test_connection, + traps_enabled=False, + discovery_enabled=False, + ) + Device.objects.create( + name='Name Check Device', + ip_address='10.24.0.10', + credential=test_credential_v2c, + network=network, + ) + results = _build_network_pipeline_configs(network) + for r in results: + assert 'name_check_network' in r['pipeline_name'] + + def test_config_is_valid_logstash_syntax( + self, test_connection, test_credential_v2c + ): + from SNMP.snmp_crud import _build_network_pipeline_configs + network = Network.objects.create( + name='Syntax Check Network', + network_range='10.25.0.0/24', + connection=test_connection, + traps_enabled=False, + discovery_enabled=False, + ) + Device.objects.create( + name='Syntax Device', + ip_address='10.25.0.10', + credential=test_credential_v2c, + network=network, + ) + results = _build_network_pipeline_configs(network) + for r in results: + config = r['config'] + assert 'input {' in config + assert 'output {' in config + + +@pytest.mark.django_db +class TestDeviceVisualizationData: + """Regression tests for device visualization data shaping. + + Covers the defects that left the device detail panel blank or wrong while the + underlying data was present: interface values nested under OpenConfig's + ``state``, and memory being sourced from the wrong place. + """ + + def _search_stub(self, by_category, captured=None): + """Build an es.search side_effect that dispatches on event.category.""" + def search(**kwargs): + filters = kwargs['query']['bool']['filter'] + if captured is not None: + captured.append(filters) + categories = [f['term']['event.category'] for f in filters + if 'term' in f and 'event.category' in f['term']] + for category in categories: + if category in by_category: + return {'hits': {'hits': by_category[category]}} + return {'hits': {'hits': []}} + return search + + # ---- interface shaping ---- + + def test_interfaces_flatten_openconfig_state_and_counters(self, test_device): + """state.* AND state.counters.* are lifted to where the UI reads them.""" + from SNMP.snmp_crud import _get_device_interfaces + + mock_es = MagicMock() + mock_es.search.return_value = { + 'aggregations': {'fans': {'buckets': [ + {'top_if_doc': {'hits': {'hits': [{'_source': {'interface': { + 'name': 'Ethernet1', + 'index': 1, + 'state': { + 'admin_status': 'UP', + 'oper_status': 'DOWN', + 'speed': 1000000000.0, + 'counters': {'in_octets': 42, 'out_errors': 7}, + }, + }}}]}}}, + ]}} + } + + iface = _get_device_interfaces(test_device, mock_es)['interfaces'][0] + + assert iface['admin_status'] == 'UP' + assert iface['oper_status'] == 'DOWN' + assert iface['speed'] == 1000000000.0 + # Counters must be flat — createInterfaceCard reads iface.in_octets, so + # leaving them at iface.counters.in_octets renders 0 B on a busy link. + assert iface['in_octets'] == 42 + assert iface['out_errors'] == 7 + assert iface['name'] == 'Ethernet1' + assert iface['index'] == 1 + assert 'state' not in iface + + def test_normalized_top_level_status_wins_over_raw_state(self, test_device): + """A raw state value must not clobber the pipeline-normalized one.""" + from SNMP.snmp_crud import _get_device_interfaces + + mock_es = MagicMock() + mock_es.search.return_value = { + 'aggregations': {'fans': {'buckets': [ + {'top_if_doc': {'hits': {'hits': [{'_source': {'interface': { + 'name': 'Ethernet1', + # Translate normalizer already decoded 1 -> UP here. + 'oper_status': 'UP', + # ...while the raw enum survives under state. + 'state': {'oper_status': 1, 'admin_status': 'UP'}, + }}}]}}}, + ]}} + } + + iface = _get_device_interfaces(test_device, mock_es)['interfaces'][0] + + # The UI compares strictly against 'UP'; the raw 1 would render "Unknown". + assert iface['oper_status'] == 'UP' + # Keys only present under state still come through. + assert iface['admin_status'] == 'UP' + + def test_interfaces_without_state_are_passed_through(self, test_device): + """A device that already reports flat status is left alone.""" + from SNMP.snmp_crud import _get_device_interfaces + + mock_es = MagicMock() + mock_es.search.return_value = { + 'aggregations': {'fans': {'buckets': [ + {'top_if_doc': {'hits': {'hits': [{'_source': {'interface': { + 'name': 'Ethernet1', 'admin_status': 'UP', 'oper_status': 'UP', + }}}]}}}, + ]}} + } + + iface = _get_device_interfaces(test_device, mock_es)['interfaces'][0] + assert iface['admin_status'] == 'UP' + assert iface['oper_status'] == 'UP' + + # ---- metrics sourcing ---- + + def test_canonical_memory_field_is_preferred(self, test_device): + """Profiles deriving system.memory.actual.used.pct keep working.""" + from SNMP.snmp_crud import _get_device_metrics + + captured = [] + mock_es = MagicMock() + mock_es.search.side_effect = self._search_stub({ + 'metrics': [{'_source': { + '@timestamp': '2026-08-04T01:00:00Z', + 'system': { + 'cpu': {'total': {'norm': {'pct': 0.25}}}, + 'memory': {'actual': {'used': {'pct': 0.19}}}, + }, + 'host': {'uptime': 12345}, + }}], + }, captured) + + metrics = _get_device_metrics(test_device, mock_es) + + assert metrics['CPU'] == [0.25] + assert metrics['Memory'] == [0.19] + assert metrics['MemorySource'] == 'system.memory.actual.used.pct' + assert metrics['Uptime'] == 12345 + # The storage-table fallback must not be queried when the canonical + # field is present — that query is pure overhead here. + queried = [f['term']['event.category'] for fl in captured for f in fl + if 'term' in f and 'event.category' in f['term']] + assert 'system.filesystem' not in queried + + def test_cpu_returned_when_memory_is_absent(self, test_device): + """CPU must survive on its own — it used to be dropped with memory.""" + from SNMP.snmp_crud import _get_device_metrics + + mock_es = MagicMock() + mock_es.search.side_effect = self._search_stub({ + 'metrics': [{'_source': { + '@timestamp': '2026-08-04T01:00:00Z', + 'system': {'cpu': {'total': {'norm': {'pct': 0.25}}}}, + 'host': {'uptime': 12345}, + }}], + }) + + metrics = _get_device_metrics(test_device, mock_es) + + assert metrics['CPU'] == [0.25] + assert metrics['CPUTime'] == ['2026-08-04T01:00:00Z'] + assert metrics['Memory'] == [] + assert metrics['MemorySource'] is None + + def test_memory_falls_back_to_physical_hrstorage_ram_row(self, test_device): + """Without the canonical field, use hrStorageRam — physical, not cache.""" + from SNMP.snmp_crud import _get_device_metrics + + captured = [] + + def search(**kwargs): + filters = kwargs['query']['bool']['filter'] + captured.append(filters) + categories = [f['term']['event.category'] for f in filters + if 'term' in f and 'event.category' in f['term']] + if 'metrics' in categories: + return {'hits': {'hits': [{'_source': { + '@timestamp': '2026-08-04T01:00:00Z', + 'system': {'cpu': {'total': {'norm': {'pct': 0.25}}}}, + }}]}} + # The aggregation asks for the lowest hrStorageIndex per poll, so the + # physical row is what comes back — cache/buffers rows are ranked out + # by the sort, which is the behaviour being pinned here. + return {'aggregations': {'by_poll': {'buckets': [ + {'key': 1, 'physical': {'hits': {'hits': [{'_source': { + '@timestamp': '2026-08-04T01:00:00Z', + 'system': {'filesystem': {'used': {'pct': 0.98}}}, + }}]}}}, + ]}}} + + mock_es = MagicMock() + mock_es.search.side_effect = search + + metrics = _get_device_metrics(test_device, mock_es) + + assert metrics['Memory'] == [0.98] + assert metrics['MemoryTime'] == ['2026-08-04T01:00:00Z'] + assert metrics['MemorySource'] == 'hrStorageRam' + + fs_filters = [fl for fl in captured + if any('term' in f and f['term'].get('event.category') == 'system.filesystem' + for f in fl)] + assert fs_filters, "expected a system.filesystem query" + + # Rows are selected by hrStorageType, not by a locale-specific description + # ("RAM" on EOS vs "Physical memory" on net-snmp), and matched under both + # possible mappings of that field. + shoulds = [f['bool']['should'] for f in fs_filters[0] if 'bool' in f] + assert shoulds, "expected a bool/should type filter" + fields = {list(clause['term'].keys())[0] for clause in shoulds[0]} + assert fields == {'system.filesystem.type', 'system.filesystem.type.keyword'} + assert all(list(c['term'].values())[0] == '1.3.6.1.2.1.25.2.1.2' for c in shoulds[0]) + assert not any('mount_point' in str(f) for f in fs_filters[0]) + + # All RAM rows report the same total, so the row must be disambiguated by + # lowest hrStorageIndex — ranking by total silently picks an arbitrary row. + agg = mock_es.search.call_args_list[-1].kwargs['aggregations'] + top_hits = agg['by_poll']['aggregations']['physical']['top_hits'] + assert top_hits['sort'] == [{'system.filesystem.index': {'order': 'asc'}}] + assert top_hits['size'] == 1 + + def test_metrics_queries_are_scoped_to_the_device(self, test_device): + """Every metrics query must filter to this device, not the whole fleet.""" + from SNMP.snmp_crud import _get_device_metrics + + captured = [] + mock_es = MagicMock() + mock_es.search.side_effect = self._search_stub({ + 'metrics': [{'_source': { + '@timestamp': '2026-08-04T01:00:00Z', + 'system': {'cpu': {'total': {'norm': {'pct': 0.25}}}}, + }}], + 'system.filesystem': [], + }, captured) + + _get_device_metrics(test_device, mock_es) + + assert captured, "expected at least one query" + for filters in captured: + assert {"term": {"host.polled_address": test_device.ip_address}} in filters + assert {"range": {"@timestamp": {"gte": "now-6h"}}} in filters + + def test_uptime_defaults_to_zero_without_metrics_docs(self, test_device): + """No metrics documents must not raise.""" + from SNMP.snmp_crud import _get_device_metrics + + mock_es = MagicMock() + mock_es.search.side_effect = self._search_stub({}) + + metrics = _get_device_metrics(test_device, mock_es) + + assert metrics['Uptime'] == 0 + assert metrics['CPU'] == [] + assert metrics['Memory'] == [] diff --git a/tests/SNMP/unit/test_snmp_grounding.py b/tests/SNMP/unit/test_snmp_grounding.py new file mode 100644 index 0000000..961682a --- /dev/null +++ b/tests/SNMP/unit/test_snmp_grounding.py @@ -0,0 +1,263 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +""" +Tests for SNMP.snmp_grounding — all pure functions, no DB or network required. +""" + +import json +import os +import tempfile +import pytest + +from SNMP.snmp_grounding import ( + build_grounding, + load_grounding, + reduce_and_ground, +) + + +# =========================================================================== +# Helpers +# =========================================================================== + +def _write_mib(directory, filename, data): + path = os.path.join(directory, filename) + with open(path, 'w') as f: + json.dump(data, f) + return path + + +def _minimal_mib(obj_name, oid, nodetype='scalar', typ='Integer32', enum=None, units=None): + """ + Return a minimal pysmi-style MIB dict for one object. + build_grounding iterates the file dict directly as {obj_name: obj_body}, + so the file should contain {obj_name: {...}} at the top level. + """ + syntax = {'type': typ} + if enum: + syntax['constraints'] = {'enumeration': enum} + obj = { + 'oid': oid, + 'nodetype': nodetype, + 'syntax': syntax, + 'maxaccess': 'read-only', + } + if units: + obj['units'] = units + return {obj_name: obj} + + +# =========================================================================== +# build_grounding +# =========================================================================== + +class TestBuildGrounding: + + def test_empty_directory_returns_empty_dict(self): + with tempfile.TemporaryDirectory() as d: + result = build_grounding(d) + assert result == {} + + def test_scalar_object_included(self): + with tempfile.TemporaryDirectory() as d: + mib = _minimal_mib('sysDescr', '1.3.6.1.2.1.1.1', nodetype='scalar') + _write_mib(d, 'RFC1213-MIB.json', mib) + result = build_grounding(d) + assert '1.3.6.1.2.1.1.1' in result + entry = result['1.3.6.1.2.1.1.1'] + assert entry['name'] == 'sysDescr' + assert entry['nodetype'] == 'scalar' + + def test_column_object_included(self): + with tempfile.TemporaryDirectory() as d: + mib = _minimal_mib('ifDescr', '1.3.6.1.2.1.2.2.1.2', nodetype='column') + _write_mib(d, 'IF-MIB.json', mib) + result = build_grounding(d) + assert '1.3.6.1.2.1.2.2.1.2' in result + + def test_enum_is_inverted(self): + """pysmi stores enums as {name: int}; build_grounding inverts to {int: name}.""" + with tempfile.TemporaryDirectory() as d: + mib = _minimal_mib( + 'ifOperStatus', '1.3.6.1.2.1.2.2.1.8', + nodetype='scalar', + enum={'up': 1, 'down': 2}, + ) + _write_mib(d, 'IF-MIB.json', mib) + result = build_grounding(d) + assert result['1.3.6.1.2.1.2.2.1.8']['enum'] == {1: 'up', 2: 'down'} + + def test_invalid_json_file_skipped(self): + with tempfile.TemporaryDirectory() as d: + bad_path = os.path.join(d, 'bad.json') + with open(bad_path, 'w') as f: + f.write('this is not json {{{') + result = build_grounding(d) + assert result == {} + + def test_non_dict_json_file_skipped(self): + with tempfile.TemporaryDirectory() as d: + _write_mib(d, 'list.json', [1, 2, 3]) + result = build_grounding(d) + assert result == {} + + def test_non_scalar_column_nodetype_excluded(self): + with tempfile.TemporaryDirectory() as d: + mib = {'someRow': {'oid': '1.2.3', 'nodetype': 'row', 'syntax': {}}} + _write_mib(d, 'TEST-MIB.json', mib) + result = build_grounding(d) + assert '1.2.3' not in result + + def test_units_included_when_defined(self): + with tempfile.TemporaryDirectory() as d: + mib = _minimal_mib('sysUpTime', '1.3.6.1.2.1.1.3', + nodetype='scalar', typ='TimeTicks', + units='hundredths of a second') + _write_mib(d, 'MIB.json', mib) + result = build_grounding(d) + assert result['1.3.6.1.2.1.1.3']['units'] == 'hundredths of a second' + + def test_multiple_files_merged(self): + with tempfile.TemporaryDirectory() as d: + mib1 = _minimal_mib('sysDescr', '1.3.6.1.2.1.1.1', nodetype='scalar') + mib2 = _minimal_mib('ifDescr', '1.3.6.1.2.1.2.2.1.2', nodetype='column') + _write_mib(d, 'MIB1.json', mib1) + _write_mib(d, 'MIB2.json', mib2) + result = build_grounding(d) + assert '1.3.6.1.2.1.1.1' in result + assert '1.3.6.1.2.1.2.2.1.2' in result + + +# =========================================================================== +# load_grounding +# =========================================================================== + +class TestLoadGrounding: + + def test_valid_json_file_loaded(self): + with tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False) as f: + json.dump({'1.2.3': {'name': 'sysDescr'}}, f) + path = f.name + try: + result = load_grounding(path) + assert '1.2.3' in result + finally: + os.unlink(path) + + def test_missing_file_returns_empty_dict(self): + result = load_grounding('/nonexistent/path/grounding.json') + assert result == {} + + def test_invalid_json_returns_empty_dict(self): + with tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False) as f: + f.write('not valid json !!!') + path = f.name + try: + result = load_grounding(path) + assert result == {} + finally: + os.unlink(path) + + +# =========================================================================== +# reduce_and_ground +# =========================================================================== + +class TestReduceAndGround: + """ + reduce_and_ground is pure; we supply a synthetic grounding dict so tests + don't depend on the compiled grounding.json file. + """ + + GROUNDING = { + '1.3.6.1.2.1.1.1': { + 'name': 'sysDescr', 'mib': 'RFC1213-MIB', + 'type': 'OctetString', 'enum': None, + 'nodetype': 'scalar', 'access': 'read-only', + }, + '1.3.6.1.2.1.2.2.1.2': { + 'name': 'ifDescr', 'mib': 'IF-MIB', + 'type': 'DisplayString', 'enum': None, + 'nodetype': 'column', 'access': 'read-only', + }, + '1.3.6.1.2.1.2.2.1.8': { + 'name': 'ifOperStatus', 'mib': 'IF-MIB', + 'type': 'Integer32', + 'enum': {1: 'up', 2: 'down'}, + 'nodetype': 'column', 'access': 'read-only', + }, + } + + def test_empty_walk_returns_empty_results(self): + grounded, ungrounded = reduce_and_ground('', self.GROUNDING) + assert grounded == [] + assert ungrounded == [] + + def test_known_scalar_oid_grounded(self): + walk = '1.3.6.1.2.1.1.1.0 = Linux router' + grounded, ungrounded = reduce_and_ground(walk, self.GROUNDING) + assert len(grounded) == 1 + assert grounded[0]['name'] == 'sysDescr' + + def test_known_table_oid_with_instance_grounded(self): + # ifDescr.1 — the '.1' is the instance arc (row index) + walk = '1.3.6.1.2.1.2.2.1.2.1 = GigabitEthernet0/0' + grounded, ungrounded = reduce_and_ground(walk, self.GROUNDING) + names = [r['name'] for r in grounded] + assert 'ifDescr' in names + + def test_multiple_rows_of_same_column_counted(self): + walk = ( + '1.3.6.1.2.1.2.2.1.2.1 = GigabitEthernet0/0\n' + '1.3.6.1.2.1.2.2.1.2.2 = GigabitEthernet0/1\n' + ) + grounded, _ = reduce_and_ground(walk, self.GROUNDING) + ifdescr = next(r for r in grounded if r['name'] == 'ifDescr') + assert ifdescr['instances'] == 2 + + def test_unknown_oid_goes_to_ungrounded(self): + walk = '9.9.9.9.9.9 = SomeValue' + _, ungrounded = reduce_and_ground(walk, self.GROUNDING) + assert len(ungrounded) > 0 + + def test_grounded_sorted_by_oid_numerically(self): + walk = ( + '1.3.6.1.2.1.2.2.1.8.1 = 1\n' + '1.3.6.1.2.1.1.1.0 = router\n' + ) + grounded, _ = reduce_and_ground(walk, self.GROUNDING) + oids = [r['oid'] for r in grounded] + assert oids == sorted(oids, key=lambda o: [int(x) for x in o.split('.')]) + + def test_sample_value_truncated_to_50_chars(self): + long_value = 'X' * 200 + walk = f'1.3.6.1.2.1.1.1.0 = {long_value}' + grounded, _ = reduce_and_ground(walk, self.GROUNDING) + assert len(grounded[0]['sample']) <= 50 + + def test_malformed_lines_skipped(self): + walk = 'this is not a valid walk line\n1.3.6.1.2.1.1.1.0 = Linux' + grounded, _ = reduce_and_ground(walk, self.GROUNDING) + assert len(grounded) == 1 + + def test_enum_propagated_to_grounded_entry(self): + walk = '1.3.6.1.2.1.2.2.1.8.1 = 1' + grounded, _ = reduce_and_ground(walk, self.GROUNDING) + entry = next(r for r in grounded if r['name'] == 'ifOperStatus') + assert entry['enum'] == {1: 'up', 2: 'down'} + + def test_uses_module_level_grounding_when_none_passed(self): + # Should not raise even when the module-level GROUNDING may be empty. + walk = '1.3.6.1.2.1.1.1.0 = test' + grounded, ungrounded = reduce_and_ground(walk) # no grounding arg + # Result depends on the compiled file; just verify it doesn't crash + assert isinstance(grounded, list) + assert isinstance(ungrounded, list) + + def test_tab_separated_walk_format(self): + walk = '1.3.6.1.2.1.1.1.0\tLinux router' + grounded, _ = reduce_and_ground(walk, self.GROUNDING) + assert len(grounded) == 1 + assert grounded[0]['name'] == 'sysDescr' diff --git a/tests/SNMP/unit/test_snmp_normalizers.py b/tests/SNMP/unit/test_snmp_normalizers.py new file mode 100644 index 0000000..5945aa8 --- /dev/null +++ b/tests/SNMP/unit/test_snmp_normalizers.py @@ -0,0 +1,545 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +""" +Tests for SNMP.snmp_normalizers — all pure functions, no DB or network required. +""" + +import pytest + +from SNMP.snmp_normalizers import ( + _apply_normalizers, + _generate_multiply_get_filter, + _generate_ratio_get_filter, + _generate_translate_filter, +) + + +# =========================================================================== +# _generate_multiply_get_filter +# =========================================================================== + +class TestGenerateMultiplyGetFilter: + + def test_returns_none_for_empty_list(self): + assert _generate_multiply_get_filter([]) is None + + def test_returns_comment_and_ruby_filter(self): + normalizers = [ + { + 'operation': 'multiply', + 'target': {'scope': 'get', 'field': 'system.cpu.total.norm.pct'}, + 'params': {'multiply_value': 0.01} + } + ] + result = _generate_multiply_get_filter(normalizers) + assert isinstance(result, list) + assert len(result) == 2 + comment, ruby = result + assert comment['plugin'] == 'comment' + assert ruby['plugin'] == 'ruby' + + def test_ruby_code_contains_field_path(self): + normalizers = [ + { + 'operation': 'multiply', + 'target': {'scope': 'get', 'field': 'system.cpu.total.norm.pct'}, + 'params': {'multiply_value': 0.01} + } + ] + result = _generate_multiply_get_filter(normalizers) + ruby_code = result[1]['config']['code'] + assert '[system][cpu][total][norm][pct]' in ruby_code + + def test_ruby_code_contains_multiply_value(self): + normalizers = [ + { + 'operation': 'multiply', + 'target': {'scope': 'get', 'field': 'some.metric'}, + 'params': {'multiply_value': 100} + } + ] + result = _generate_multiply_get_filter(normalizers) + ruby_code = result[1]['config']['code'] + assert '100' in ruby_code + + def test_skips_normalizer_missing_field(self): + normalizers = [ + { + 'operation': 'multiply', + 'target': {'scope': 'get'}, # no 'field' + 'params': {'multiply_value': 0.01} + } + ] + result = _generate_multiply_get_filter(normalizers) + assert result is None + + def test_skips_normalizer_missing_multiply_value(self): + normalizers = [ + { + 'operation': 'multiply', + 'target': {'scope': 'get', 'field': 'some.metric'}, + 'params': {} # no 'multiply_value' + } + ] + result = _generate_multiply_get_filter(normalizers) + assert result is None + + def test_multiple_normalizers_in_single_filter(self): + normalizers = [ + { + 'operation': 'multiply', + 'target': {'scope': 'get', 'field': 'metric.a'}, + 'params': {'multiply_value': 2} + }, + { + 'operation': 'multiply', + 'target': {'scope': 'get', 'field': 'metric.b'}, + 'params': {'multiply_value': 3} + } + ] + result = _generate_multiply_get_filter(normalizers) + # Both fields consolidated into single ruby filter + assert isinstance(result, list) + ruby_code = result[1]['config']['code'] + assert '[metric][a]' in ruby_code + assert '[metric][b]' in ruby_code + + def test_comment_mentions_multiply(self): + normalizers = [ + { + 'operation': 'multiply', + 'target': {'scope': 'get', 'field': 'metric.a'}, + 'params': {'multiply_value': 0.5} + } + ] + result = _generate_multiply_get_filter(normalizers) + comment_text = result[0]['config']['text'] + assert 'Multiply' in comment_text + + +# =========================================================================== +# _generate_ratio_get_filter +# =========================================================================== + +class TestGenerateRatioGetFilter: + + def test_returns_none_for_empty_list(self): + assert _generate_ratio_get_filter([]) is None + + def test_returns_comment_and_ruby_filter(self): + normalizers = [ + { + 'operation': 'ratio', + 'target': {'scope': 'get'}, + 'params': { + 'value1_field': 'memory.used', + 'value2_field': 'memory.free', + 'total_output_field': 'memory.total', + } + } + ] + result = _generate_ratio_get_filter(normalizers) + assert isinstance(result, list) + assert len(result) == 2 + comment, ruby = result + assert comment['plugin'] == 'comment' + assert ruby['plugin'] == 'ruby' + + def test_skips_normalizer_missing_value_fields(self): + normalizers = [ + { + 'operation': 'ratio', + 'target': {'scope': 'get'}, + 'params': { + 'value1_field': 'memory.used', + # no value2_field + } + } + ] + result = _generate_ratio_get_filter(normalizers) + assert result is None + + def test_ruby_code_contains_field_paths(self): + normalizers = [ + { + 'operation': 'ratio', + 'target': {'scope': 'get'}, + 'params': { + 'value1_field': 'memory.used', + 'value2_field': 'memory.free', + } + } + ] + result = _generate_ratio_get_filter(normalizers) + ruby_code = result[1]['config']['code'] + assert '[memory][used]' in ruby_code + assert '[memory][free]' in ruby_code + + def test_optional_output_fields_included_when_specified(self): + normalizers = [ + { + 'operation': 'ratio', + 'target': {'scope': 'get'}, + 'params': { + 'value1_field': 'mem.used', + 'value2_field': 'mem.free', + 'total_output_field': 'mem.total', + 'ratio1_output_field': 'mem.used_pct', + 'ratio2_output_field': 'mem.free_pct', + 'complement_ratio_output_field': 'mem.complement', + 'divide_output_field': 'mem.divided', + } + } + ] + result = _generate_ratio_get_filter(normalizers) + ruby_code = result[1]['config']['code'] + assert '[mem][total]' in ruby_code + assert '[mem][used_pct]' in ruby_code + assert '[mem][free_pct]' in ruby_code + assert '[mem][complement]' in ruby_code + assert '[mem][divided]' in ruby_code + + def test_multiple_ratio_normalizers_use_unique_variable_names(self): + normalizers = [ + { + 'operation': 'ratio', + 'target': {'scope': 'get'}, + 'params': {'value1_field': 'a.used', 'value2_field': 'a.free'} + }, + { + 'operation': 'ratio', + 'target': {'scope': 'get'}, + 'params': {'value1_field': 'b.used', 'value2_field': 'b.free'} + } + ] + result = _generate_ratio_get_filter(normalizers) + ruby_code = result[1]['config']['code'] + # With multiple normalizers, suffixes are added to variable names + assert 'value1_0' in ruby_code + assert 'value1_1' in ruby_code + + def test_comment_mentions_ratio(self): + normalizers = [ + { + 'operation': 'ratio', + 'target': {'scope': 'get'}, + 'params': {'value1_field': 'a.used', 'value2_field': 'a.free'} + } + ] + result = _generate_ratio_get_filter(normalizers) + comment_text = result[0]['config']['text'] + assert 'Ratio' in comment_text + + +# =========================================================================== +# _generate_translate_filter +# =========================================================================== + +class TestGenerateTranslateFilter: + + def test_returns_none_for_empty_list(self): + assert _generate_translate_filter([]) is None + + def test_returns_comment_and_translate_filter(self): + normalizers = [ + { + 'operation': 'translate', + 'target': {'scope': 'table', 'field': 'interface.admin_status'}, + 'params': { + 'mapping': {'1': 'UP', '2': 'DOWN', '3': 'TESTING'} + } + } + ] + result = _generate_translate_filter(normalizers) + assert isinstance(result, list) + assert len(result) == 2 + comment, translate = result + assert comment['plugin'] == 'comment' + assert translate['plugin'] == 'translate' + + def test_translate_config_has_correct_source_and_destination(self): + normalizers = [ + { + 'operation': 'translate', + 'target': {'scope': 'table', 'field': 'interface.oper_status'}, + 'params': {'mapping': {'1': 'UP', '2': 'DOWN'}} + } + ] + result = _generate_translate_filter(normalizers) + translate_config = result[1]['config'] + assert translate_config['source'] == '[interface][oper_status]' + assert translate_config['destination'] == '[interface][oper_status]' + + def test_translate_config_has_override_true(self): + normalizers = [ + { + 'operation': 'translate', + 'target': {'scope': 'table', 'field': 'interface.admin_status'}, + 'params': {'mapping': {'1': 'UP'}} + } + ] + result = _generate_translate_filter(normalizers) + assert result[1]['config']['override'] is True + + def test_translate_config_contains_mapping(self): + mapping = {'1': 'UP', '2': 'DOWN', '3': 'TESTING'} + normalizers = [ + { + 'operation': 'translate', + 'target': {'scope': 'table', 'field': 'interface.admin_status'}, + 'params': {'mapping': mapping} + } + ] + result = _generate_translate_filter(normalizers) + assert result[1]['config']['dictionary'] == mapping + + def test_skips_normalizer_without_field(self): + normalizers = [ + { + 'operation': 'translate', + 'target': {'scope': 'table'}, # no 'field' + 'params': {'mapping': {'1': 'UP'}} + } + ] + result = _generate_translate_filter(normalizers) + assert result is None + + def test_skips_normalizer_without_mapping(self): + normalizers = [ + { + 'operation': 'translate', + 'target': {'scope': 'table', 'field': 'interface.admin_status'}, + 'params': {} # no 'mapping' + } + ] + result = _generate_translate_filter(normalizers) + assert result is None + + def test_multiple_translate_normalizers_each_get_own_filter(self): + normalizers = [ + { + 'operation': 'translate', + 'target': {'scope': 'table', 'field': 'interface.admin_status'}, + 'params': {'mapping': {'1': 'UP', '2': 'DOWN'}} + }, + { + 'operation': 'translate', + 'target': {'scope': 'table', 'field': 'interface.oper_status'}, + 'params': {'mapping': {'1': 'UP', '2': 'DOWN', '3': 'TESTING'}} + } + ] + result = _generate_translate_filter(normalizers) + # Two normalizers → 2 comment + 2 translate = 4 total + assert len(result) == 4 + + +# =========================================================================== +# _apply_normalizers +# =========================================================================== + +class TestApplyNormalizers: + + def test_returns_empty_list_for_none(self): + assert _apply_normalizers(None) == [] + + def test_returns_empty_list_for_empty_list(self): + assert _apply_normalizers([]) == [] + + def test_skips_normalizer_missing_operation(self): + normalizers = [ + { + 'target': {'scope': 'get', 'field': 'some.field'}, + 'params': {'multiply_value': 2} + } + ] + result = _apply_normalizers(normalizers) + assert result == [] + + def test_skips_normalizer_missing_scope(self): + normalizers = [ + { + 'operation': 'multiply', + 'target': {'field': 'some.field'}, # no 'scope' + 'params': {'multiply_value': 2} + } + ] + result = _apply_normalizers(normalizers) + assert result == [] + + def test_applies_multiply_normalizer(self): + normalizers = [ + { + 'operation': 'multiply', + 'target': {'scope': 'get', 'field': 'system.cpu.total.norm.pct'}, + 'params': {'multiply_value': 0.01} + } + ] + result = _apply_normalizers(normalizers) + assert len(result) > 0 + plugin_types = [c['plugin'] for c in result] + assert 'ruby' in plugin_types + + def test_applies_ratio_normalizer(self): + normalizers = [ + { + 'operation': 'ratio', + 'target': {'scope': 'get'}, + 'params': { + 'value1_field': 'memory.used', + 'value2_field': 'memory.free', + } + } + ] + result = _apply_normalizers(normalizers) + assert len(result) > 0 + plugin_types = [c['plugin'] for c in result] + assert 'ruby' in plugin_types + + def test_applies_translate_normalizer(self): + normalizers = [ + { + 'operation': 'translate', + 'target': {'scope': 'table', 'field': 'interface.admin_status'}, + 'params': {'mapping': {'1': 'UP', '2': 'DOWN'}} + } + ] + result = _apply_normalizers(normalizers) + assert len(result) > 0 + plugin_types = [c['plugin'] for c in result] + assert 'translate' in plugin_types + + def test_applies_multiple_normalizer_types(self): + normalizers = [ + { + 'operation': 'multiply', + 'target': {'scope': 'get', 'field': 'metric.a'}, + 'params': {'multiply_value': 0.01} + }, + { + 'operation': 'translate', + 'target': {'scope': 'table', 'field': 'interface.admin_status'}, + 'params': {'mapping': {'1': 'UP'}} + } + ] + result = _apply_normalizers(normalizers) + plugin_types = [c['plugin'] for c in result] + assert 'ruby' in plugin_types + assert 'translate' in plugin_types + + def test_ignores_unknown_operation(self): + normalizers = [ + { + 'operation': 'unknown_op', + 'target': {'scope': 'get', 'field': 'some.field'}, + 'params': {} + } + ] + result = _apply_normalizers(normalizers) + assert result == [] + + def test_table_scope_multiply_is_handled(self): + normalizers = [ + { + 'operation': 'multiply', + 'target': {'scope': 'table', 'field': 'interface.speed'}, + 'params': {'multiply_value': 1000} + } + ] + result = _apply_normalizers(normalizers) + assert len(result) > 0 + + +# =========================================================================== +# Scope-qualified filter IDs (regression: duplicate Logstash plugin IDs) +# =========================================================================== + +class TestScopeQualifiedFilterIds: + """ + _generate_multiply_get_filter / _generate_ratio_get_filter run for BOTH + 'get' and 'table' scope normalizers, so their generated Logstash filter + component IDs must be scope-qualified. Otherwise a profile pairing a + get-scope op with a table-scope op of the same type emits two components + with an identical plugin ID, and Logstash rejects pipelines with duplicate + plugin IDs at compile time (the merged pipeline never builds). + """ + + # Mirrors cisco_system_metrics.json (get scope) combined with a Cisco + # OpenConfig interface-table profile (table scope). + GET_MULTIPLY = { + 'operation': 'multiply', + 'target': {'scope': 'get', 'field': 'system.cpu.total.norm.pct'}, + 'params': {'multiply_value': 0.01}, + } + TABLE_MULTIPLY = { + 'operation': 'multiply', + 'target': {'scope': 'table', 'field': 'interface.state.speed'}, + 'params': {'multiply_value': 1000000}, + } + GET_RATIO = { + 'operation': 'ratio', + 'target': {'scope': 'get'}, + 'params': { + 'value1_field': 'system.memory.actual.used.bytes', + 'value2_field': 'system.memory.actual.free.bytes', + 'total_output_field': 'system.memory.total.bytes', + 'ratio1_output_field': 'system.memory.actual.used.pct', + }, + } + TABLE_RATIO = { + 'operation': 'ratio', + 'target': {'scope': 'table'}, + 'params': { + 'value1_field': 'interface.state.counters.in_octets', + 'value2_field': 'interface.state.counters.out_octets', + 'total_output_field': 'interface.state.counters.total_octets', + }, + } + + @staticmethod + def _assert_unique(components): + ids = [c['id'] for c in components] + dupes = sorted({i for i in ids if ids.count(i) > 1}) + assert not dupes, f"duplicate filter component IDs: {dupes}" + + def test_multiply_get_and_table_scope_ids_are_scope_qualified(self): + components = _apply_normalizers([self.GET_MULTIPLY, self.TABLE_MULTIPLY]) + self._assert_unique(components) + ids = [c['id'] for c in components] + assert 'normalizer_multiply_get_1' in ids + assert 'normalizer_multiply_table_1' in ids + assert 'normalizer_multiply_get_comment_1' in ids + assert 'normalizer_multiply_table_comment_1' in ids + + def test_ratio_get_and_table_scope_ids_are_scope_qualified(self): + components = _apply_normalizers([self.GET_RATIO, self.TABLE_RATIO]) + self._assert_unique(components) + ids = [c['id'] for c in components] + assert 'normalizer_ratio_get_1' in ids + assert 'normalizer_ratio_table_1' in ids + assert 'normalizer_ratio_get_comment_1' in ids + assert 'normalizer_ratio_table_comment_1' in ids + + def test_combined_profile_all_component_ids_unique(self): + # The exact scenario that triggered the bug: cisco_system_metrics + # (get-scope CPU multiply + memory ratio) merged with a Cisco + # OpenConfig interface-table profile (table-scope multiply + ratio). + components = _apply_normalizers( + [self.GET_MULTIPLY, self.GET_RATIO, self.TABLE_MULTIPLY, self.TABLE_RATIO] + ) + self._assert_unique(components) + ids = [c['id'] for c in components] + for expected in ( + 'normalizer_multiply_get_1', + 'normalizer_multiply_table_1', + 'normalizer_ratio_get_1', + 'normalizer_ratio_table_1', + ): + assert expected in ids, f"missing generated component: {expected}" + + def test_single_get_scope_multiply_keeps_stable_id(self): + # IDs are always suffixed with _N by _next_id, even on the first call. + components = _apply_normalizers([self.GET_MULTIPLY]) + ids = [c['id'] for c in components] + assert ids == ['normalizer_multiply_get_comment_1', 'normalizer_multiply_get_1'] diff --git a/tests/SNMP/unit/test_snmp_pipeline_generator.py b/tests/SNMP/unit/test_snmp_pipeline_generator.py new file mode 100644 index 0000000..7ce51b5 --- /dev/null +++ b/tests/SNMP/unit/test_snmp_pipeline_generator.py @@ -0,0 +1,789 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +""" +Tests for SNMP.snmp_pipeline_generator — pure utility functions that require +no database or network access. +""" + +import pytest +from unittest.mock import MagicMock + +from SNMP.snmp_pipeline_generator import ( + _normalize_template_name, + _deduplicate_normalizers, + _uses_keystore, + _community_key_name, + _auth_pass_key_name, + _priv_pass_key_name, + _es_api_key_name, + _es_user_key_name, + _es_password_key_name, + _ref, + snmp_credential_keystore_entries, + snmp_credential_keystore_key_names, + es_connection_keystore_entries, + es_connection_keystore_key_names, + _ruby_table_nested_entry, + _ruby_row_rename_statements, + _ruby_row_value_expr, + _ruby_keep_when_statements, + _avg_var_name, + _ruby_avg_pre_loop, + _ruby_avg_in_loop, + _ruby_avg_post_loop, + _generate_table_split_filters, + _generate_snmp_error_cleanup_filter, +) + + +# =========================================================================== +# _normalize_template_name +# =========================================================================== + +class TestNormalizeTemplateName: + """Tests for the ES-index-safe name normalizer.""" + + def test_empty_string_returns_fallback(self): + assert _normalize_template_name('') == 'unknown_template' + + def test_none_returns_fallback(self): + assert _normalize_template_name(None) == 'unknown_template' + + def test_plain_name_lowercased(self): + assert _normalize_template_name('Cisco') == 'cisco' + + def test_spaces_replaced_with_underscore(self): + assert _normalize_template_name('My Template') == 'my_template' + + def test_multiple_spaces_collapsed(self): + assert _normalize_template_name('My Template') == 'my_template' + + def test_tabs_and_newlines_replaced(self): + result = _normalize_template_name('a\tb\nc') + assert result == 'a_b_c' + + def test_es_illegal_chars_replaced(self): + # Characters: * : / \ ? " < > | , # space + for char in ['*', ':', '/', '\\', '?', '"', '<', '>', '|', ',', '#']: + result = _normalize_template_name(f'name{char}value') + assert result == 'name_value', f"Failed for char {char!r}" + + def test_consecutive_underscores_collapsed(self): + assert _normalize_template_name('a__b') == 'a_b' + + def test_consecutive_hyphens_collapsed(self): + assert _normalize_template_name('a--b') == 'a_b' + + def test_mixed_separators_collapsed(self): + assert _normalize_template_name('a-_b') == 'a_b' + + def test_leading_hyphen_stripped(self): + assert _normalize_template_name('-name') == 'name' + + def test_leading_underscore_stripped(self): + assert _normalize_template_name('_name') == 'name' + + def test_leading_plus_stripped(self): + assert _normalize_template_name('+name') == 'name' + + def test_leading_dot_stripped(self): + assert _normalize_template_name('.name') == 'name' + + def test_already_clean_name_unchanged(self): + assert _normalize_template_name('dell_idrac') == 'dell_idrac' + + def test_alphanumeric_with_single_hyphen_preserved(self): + # Single hyphens are NOT replaced — only runs of 2+ hyphens/underscores are collapsed + assert _normalize_template_name('cisco-catalyst-9300') == 'cisco-catalyst-9300' + + def test_numeric_only_name(self): + assert _normalize_template_name('1234') == '1234' + + def test_surrounding_whitespace_stripped(self): + assert _normalize_template_name(' cisco ') == 'cisco' + + def test_all_forbidden_leading_chars_stripped(self): + # Multiple leading forbidden chars + result = _normalize_template_name('---___name') + assert result == 'name' + + def test_result_is_255_bytes_max(self): + # Build a name that would exceed 255 bytes when encoded + long_name = 'a' * 300 + result = _normalize_template_name(long_name) + assert len(result.encode('utf-8')) <= 255 + + def test_name_that_becomes_empty_after_stripping_returns_fallback(self): + # A name consisting only of forbidden leading chars + result = _normalize_template_name('---') + assert result == 'unknown_template' + + def test_real_world_dell_idrac(self): + assert _normalize_template_name('Dell iDRAC') == 'dell_idrac' + + def test_real_world_ubiquiti(self): + assert _normalize_template_name('Ubiquiti UniFi AP') == 'ubiquiti_unifi_ap' + + def test_real_world_brocade(self): + assert _normalize_template_name('Brocade FC Switch') == 'brocade_fc_switch' + + +# =========================================================================== +# _deduplicate_normalizers +# =========================================================================== + +class TestDeduplicateNormalizers: + + def _make_normalizer(self, operation, field, param_value): + return { + 'operation': operation, + 'target': {'scope': 'get', 'field': field}, + 'params': {'multiply_value': param_value} + } + + def test_returns_empty_for_none(self): + assert _deduplicate_normalizers(None) == [] + + def test_returns_empty_for_empty_list(self): + assert _deduplicate_normalizers([]) == [] + + def test_single_normalizer_returned_unchanged(self): + n = self._make_normalizer('multiply', 'metric.a', 0.01) + result = _deduplicate_normalizers([n]) + assert result == [n] + + def test_identical_normalizers_deduplicated(self): + n = self._make_normalizer('multiply', 'metric.a', 0.01) + result = _deduplicate_normalizers([n, n]) + assert len(result) == 1 + + def test_different_normalizers_both_kept(self): + n1 = self._make_normalizer('multiply', 'metric.a', 0.01) + n2 = self._make_normalizer('multiply', 'metric.b', 0.01) + result = _deduplicate_normalizers([n1, n2]) + assert len(result) == 2 + + def test_different_operations_both_kept(self): + n1 = { + 'operation': 'multiply', + 'target': {'scope': 'get', 'field': 'metric.a'}, + 'params': {'multiply_value': 0.01} + } + n2 = { + 'operation': 'ratio', + 'target': {'scope': 'get'}, + 'params': {'value1_field': 'metric.a', 'value2_field': 'metric.b'} + } + result = _deduplicate_normalizers([n1, n2]) + assert len(result) == 2 + + def test_duplicate_with_different_param_value_both_kept(self): + n1 = self._make_normalizer('multiply', 'metric.a', 0.01) + n2 = self._make_normalizer('multiply', 'metric.a', 100) + result = _deduplicate_normalizers([n1, n2]) + assert len(result) == 2 + + def test_three_duplicates_only_one_kept(self): + n = self._make_normalizer('multiply', 'metric.a', 0.01) + result = _deduplicate_normalizers([n, n, n]) + assert len(result) == 1 + + def test_mixed_duplicates_and_unique(self): + n1 = self._make_normalizer('multiply', 'metric.a', 0.01) + n2 = self._make_normalizer('multiply', 'metric.b', 0.01) + result = _deduplicate_normalizers([n1, n1, n2]) + assert len(result) == 2 + + def test_order_preserved_for_unique_normalizers(self): + n1 = self._make_normalizer('multiply', 'metric.a', 0.01) + n2 = self._make_normalizer('multiply', 'metric.b', 0.01) + n3 = self._make_normalizer('multiply', 'metric.c', 0.01) + result = _deduplicate_normalizers([n1, n2, n3]) + assert result[0] == n1 + assert result[1] == n2 + assert result[2] == n3 + + def test_first_occurrence_kept_on_duplicate(self): + n1 = self._make_normalizer('multiply', 'metric.a', 0.01) + n2 = self._make_normalizer('multiply', 'metric.a', 0.01) + result = _deduplicate_normalizers([n1, n2]) + assert result[0] is n1 + + +# =========================================================================== +# _uses_keystore +# =========================================================================== + +class TestUsesKeystore: + + def _network(self, deployment_mode='CENTRALIZED', credential_mode='KEYSTORE'): + n = MagicMock() + n.deployment_mode = deployment_mode + n.credential_mode = credential_mode + return n + + def test_agent_mode_always_uses_keystore(self): + assert _uses_keystore(self._network(deployment_mode='AGENT')) is True + + def test_agent_mode_ignores_credential_mode(self): + assert _uses_keystore(self._network(deployment_mode='AGENT', credential_mode='PLAINTEXT')) is True + + def test_centralized_keystore_mode_uses_keystore(self): + assert _uses_keystore(self._network(deployment_mode='CENTRALIZED', credential_mode='KEYSTORE')) is True + + def test_centralized_plaintext_mode_no_keystore(self): + assert _uses_keystore(self._network(deployment_mode='CENTRALIZED', credential_mode='PLAINTEXT')) is False + + def test_missing_deployment_mode_attr_defaults_to_centralized(self): + n = MagicMock(spec=[]) # no attributes at all → getattr returns default + n.credential_mode = 'KEYSTORE' + assert _uses_keystore(n) is True + + def test_missing_credential_mode_defaults_to_keystore(self): + n = MagicMock(spec=[]) + n.deployment_mode = 'CENTRALIZED' + assert _uses_keystore(n) is True + + +# =========================================================================== +# Keystore key-name helpers +# =========================================================================== + +class TestKeystoreKeyNameHelpers: + + def _cred(self, cred_id, version): + c = MagicMock() + c.id = cred_id + c.version = version + return c + + def _conn(self, conn_id): + c = MagicMock() + c.id = conn_id + return c + + # _community_key_name + def test_community_key_v1(self): + assert _community_key_name(self._cred(5, '1')) == 'snmp_5_v1' + + def test_community_key_v2c(self): + assert _community_key_name(self._cred(7, '2c')) == 'snmp_7_v2' + + # _auth_pass_key_name + def test_auth_pass_key_name(self): + assert _auth_pass_key_name(self._cred(3, '3')) == 'snmp_3_v3_auth' + + # _priv_pass_key_name + def test_priv_pass_key_name(self): + assert _priv_pass_key_name(self._cred(3, '3')) == 'snmp_3_v3_priv' + + # _es_api_key_name + def test_es_api_key_name(self): + assert _es_api_key_name(self._conn(10)) == 'snmp_es_10_api_key' + + # _es_user_key_name + def test_es_user_key_name(self): + assert _es_user_key_name(self._conn(10)) == 'snmp_es_10_user' + + # _es_password_key_name + def test_es_password_key_name(self): + assert _es_password_key_name(self._conn(10)) == 'snmp_es_10_password' + + # _ref + def test_ref_wraps_in_dollar_braces(self): + assert _ref('my_key') == '${my_key}' + + def test_ref_preserves_underscores(self): + assert _ref('snmp_5_v2') == '${snmp_5_v2}' + + +# =========================================================================== +# snmp_credential_keystore_entries +# =========================================================================== + +class TestSnmpCredentialKeystoreEntries: + + def _cred(self, cred_id, version, **kwargs): + c = MagicMock() + c.id = cred_id + c.version = version + for k, v in kwargs.items(): + setattr(c, k, v) + return c + + def test_none_credential_returns_empty(self): + assert snmp_credential_keystore_entries(None) == {} + + def test_v2c_community_included(self): + cred = self._cred(1, '2c') + cred.get_community.return_value = 'public' + entries = snmp_credential_keystore_entries(cred) + assert 'snmp_1_v2' in entries + assert entries['snmp_1_v2'] == 'public' + + def test_v1_community_uses_v1_suffix(self): + cred = self._cred(2, '1') + cred.get_community.return_value = 'private' + entries = snmp_credential_keystore_entries(cred) + assert 'snmp_2_v1' in entries + + def test_v2c_empty_community_not_included(self): + cred = self._cred(3, '2c') + cred.get_community.return_value = None + entries = snmp_credential_keystore_entries(cred) + assert entries == {} + + def test_v3_authpriv_includes_auth_and_priv(self): + cred = self._cred(4, '3', security_level='authPriv') + cred.get_auth_pass.return_value = 'authsecret' + cred.get_priv_pass.return_value = 'privsecret' + entries = snmp_credential_keystore_entries(cred) + assert 'snmp_4_v3_auth' in entries + assert 'snmp_4_v3_priv' in entries + assert entries['snmp_4_v3_auth'] == 'authsecret' + assert entries['snmp_4_v3_priv'] == 'privsecret' + + def test_v3_authnopriv_includes_only_auth(self): + cred = self._cred(5, '3', security_level='authNoPriv') + cred.get_auth_pass.return_value = 'authsecret' + entries = snmp_credential_keystore_entries(cred) + assert 'snmp_5_v3_auth' in entries + assert 'snmp_5_v3_priv' not in entries + + def test_v3_noauthnopriv_returns_empty(self): + cred = self._cred(6, '3', security_level='noAuthNoPriv') + entries = snmp_credential_keystore_entries(cred) + assert entries == {} + + +# =========================================================================== +# snmp_credential_keystore_key_names +# =========================================================================== + +class TestSnmpCredentialKeystoreKeyNames: + + def _cred(self, cred_id, version, **kwargs): + c = MagicMock() + c.id = cred_id + c.version = version + for k, v in kwargs.items(): + setattr(c, k, v) + return c + + def test_none_returns_empty_set(self): + assert snmp_credential_keystore_key_names(None) == set() + + def test_v2c_with_community_returns_one_key(self): + cred = self._cred(1, '2c', community='public') + names = snmp_credential_keystore_key_names(cred) + assert names == {'snmp_1_v2'} + + def test_v2c_empty_community_returns_empty(self): + cred = self._cred(2, '2c', community='') + names = snmp_credential_keystore_key_names(cred) + assert names == set() + + def test_v3_authpriv_returns_auth_and_priv_keys(self): + cred = self._cred(3, '3', security_level='authPriv', auth_pass='x', priv_pass='y') + names = snmp_credential_keystore_key_names(cred) + assert 'snmp_3_v3_auth' in names + assert 'snmp_3_v3_priv' in names + + def test_v3_authnopriv_returns_only_auth_key(self): + cred = self._cred(4, '3', security_level='authNoPriv', auth_pass='x', priv_pass='') + names = snmp_credential_keystore_key_names(cred) + assert 'snmp_4_v3_auth' in names + assert 'snmp_4_v3_priv' not in names + + +# =========================================================================== +# es_connection_keystore_entries +# =========================================================================== + +class TestEsConnectionKeystoreEntries: + + def _conn(self, conn_id, **kwargs): + c = MagicMock() + c.id = conn_id + for k, v in kwargs.items(): + setattr(c, k, v) + return c + + def test_none_returns_empty(self): + assert es_connection_keystore_entries(None) == {} + + def test_api_key_preferred(self): + conn = self._conn(1, api_key='encrypted_key', username='user', password='pass') + conn.get_api_key.return_value = 'myapikey' + entries = es_connection_keystore_entries(conn) + assert 'snmp_es_1_api_key' in entries + assert entries['snmp_es_1_api_key'] == 'myapikey' + assert 'snmp_es_1_user' not in entries + + def test_username_password_used_when_no_api_key(self): + conn = self._conn(2, api_key=None, username='elastic', password='encrypted_pass') + conn.get_password.return_value = 'secret' + entries = es_connection_keystore_entries(conn) + assert 'snmp_es_2_user' in entries + assert 'snmp_es_2_password' in entries + assert entries['snmp_es_2_user'] == 'elastic' + assert entries['snmp_es_2_password'] == 'secret' + + def test_no_credentials_returns_empty(self): + conn = self._conn(3, api_key=None, username='', password='') + entries = es_connection_keystore_entries(conn) + assert entries == {} + + +# =========================================================================== +# es_connection_keystore_key_names +# =========================================================================== + +class TestEsConnectionKeystoreKeyNames: + + def _conn(self, conn_id, **kwargs): + c = MagicMock() + c.id = conn_id + for k, v in kwargs.items(): + setattr(c, k, v) + return c + + def test_none_returns_empty_set(self): + assert es_connection_keystore_key_names(None) == set() + + def test_api_key_returns_api_key_name(self): + conn = self._conn(1, api_key='something', username='user', password='pass') + names = es_connection_keystore_key_names(conn) + assert names == {'snmp_es_1_api_key'} + + def test_username_password_returns_user_and_password_names(self): + conn = self._conn(2, api_key=None, username='elastic', password='secret') + names = es_connection_keystore_key_names(conn) + assert 'snmp_es_2_user' in names + assert 'snmp_es_2_password' in names + + def test_no_credentials_returns_empty_set(self): + conn = self._conn(3, api_key=None, username='', password='') + names = es_connection_keystore_key_names(conn) + assert names == set() + + +# =========================================================================== +# _ruby_table_nested_entry +# =========================================================================== + +class TestRubyTableNestedEntry: + + def test_flat_table_name(self): + result = _ruby_table_nested_entry('ifTable', 'row') + assert result == '"ifTable" => row' + + def test_dotted_table_name_two_levels(self): + result = _ruby_table_nested_entry('component.fan', 'row') + assert result == '"component" => { "fan" => row }' + + def test_dotted_table_name_three_levels(self): + result = _ruby_table_nested_entry('a.b.c', 'val') + assert result == '"a" => { "b" => { "c" => val } }' + + def test_value_expr_is_preserved(self): + result = _ruby_table_nested_entry('ifTable', 'event.get("[myfield]")') + assert 'event.get("[myfield]")' in result + + +# =========================================================================== +# _ruby_row_rename_statements +# =========================================================================== + +class TestRubyRowRenameStatements: + + def test_empty_columns_returns_empty_string(self): + assert _ruby_row_rename_statements({}) == '' + + def test_flat_column_rename(self): + result = _ruby_row_rename_statements({'col_a': 'oid1'}) + assert 'row["col_a"] = row.delete("oid1")' in result + + def test_dotted_column_initializes_parent(self): + result = _ruby_row_rename_statements({'component.speed': 'oid2'}) + assert 'row["component"] ||= {}' in result + assert 'row["component"]["speed"] = row.delete("oid2")' in result + + def test_two_columns_sharing_parent_initializes_parent_once(self): + result = _ruby_row_rename_statements({ + 'iface.in_octets': 'oid1', + 'iface.out_octets': 'oid2', + }) + assert result.count('row["iface"] ||= {}') == 1 + + def test_multiple_flat_columns(self): + result = _ruby_row_rename_statements({'a': '1', 'b': '2'}) + assert 'row["a"] = row.delete("1")' in result + assert 'row["b"] = row.delete("2")' in result + + +# =========================================================================== +# _avg_var_name +# =========================================================================== + +class TestAvgVarName: + + def _normalizer(self, output_field='', target_field='unknown'): + return { + 'params': {'output_field': output_field}, + 'target': {'field': target_field}, + } + + def test_output_field_used_when_set(self): + n = self._normalizer(output_field='interface.avg_in_octets') + assert _avg_var_name(n) == 'avg_interface_avg_in_octets' + + def test_dots_replaced_with_underscores(self): + n = self._normalizer(output_field='a.b.c') + assert _avg_var_name(n) == 'avg_a_b_c' + + def test_hyphens_replaced_with_underscores(self): + n = self._normalizer(output_field='some-field') + assert _avg_var_name(n) == 'avg_some_field' + + def test_fallback_to_target_field_when_no_output_field(self): + n = self._normalizer(output_field='', target_field='interface.load') + result = _avg_var_name(n) + assert result.startswith('avg_') + assert 'interface' in result + + +# =========================================================================== +# _ruby_avg_pre_loop / _ruby_avg_in_loop / _ruby_avg_post_loop +# =========================================================================== + +class TestRubyAvgLoops: + + def _avg_normalizer(self, output_field, target_field='interface.in_octets'): + return { + 'operation': 'average', + 'target': {'scope': 'table', 'field': target_field}, + 'params': {'output_field': output_field}, + } + + def test_pre_loop_empty_returns_empty_string(self): + assert _ruby_avg_pre_loop([]) == '' + + def test_pre_loop_declares_sum_and_count(self): + n = self._avg_normalizer('interface.avg_in_octets', 'interface.in_octets') + result = _ruby_avg_pre_loop([n]) + assert '_sum = 0.0' in result + assert '_count = 0' in result + + def test_pre_loop_multiple_normalizers(self): + n1 = self._avg_normalizer('interface.avg_in', 'interface.in_octets') + n2 = self._avg_normalizer('interface.avg_out', 'interface.out_octets') + result = _ruby_avg_pre_loop([n1, n2]) + assert result.count('_sum = 0.0') == 2 + + def test_in_loop_empty_returns_empty_string(self): + assert _ruby_avg_in_loop([], 'interface') == '' + + def test_in_loop_generates_accumulation_statements(self): + n = self._avg_normalizer('interface.avg_in_octets', 'interface.in_octets') + result = _ruby_avg_in_loop([n], 'interface') + assert 'Float(_avg_v)' in result + assert '_count +=' in result + assert 'row["in_octets"]' in result + assert 'is_a?(Numeric)' not in result + + def test_in_loop_strips_dotted_table_prefix(self): + n = self._avg_normalizer('system.cpu.total.norm.pct', 'component.cpu.load_pct') + result = _ruby_avg_in_loop([n], 'component.cpu') + assert 'row["load_pct"]' in result + + def test_post_loop_empty_returns_empty_string(self): + assert _ruby_avg_post_loop([]) == '' + + def test_post_loop_generates_event_set(self): + n = self._avg_normalizer('interface.avg_in_octets', 'interface.in_octets') + result = _ruby_avg_post_loop([n]) + assert 'event.set(' in result + assert '_count > 0' in result + + def test_post_loop_skips_normalizer_without_output_field(self): + n = {'operation': 'average', 'target': {}, 'params': {'output_field': ''}} + assert _ruby_avg_post_loop([n]) == '' + + def test_post_loop_includes_multiply_when_set(self): + n = self._avg_normalizer('interface.avg_in_octets', 'interface.in_octets') + n['params']['multiply_value'] = 8 + result = _ruby_avg_post_loop([n]) + assert '* 8' in result + + +class TestRubyRowValueExpr: + + def test_single_level_column(self): + assert _ruby_row_value_expr('component.cpu', 'component.cpu.load_pct') == 'row["load_pct"]' + + def test_nested_column(self): + assert _ruby_row_value_expr( + 'system.filesystem', 'system.filesystem.total.bytes' + ) == 'row.dig("total", "bytes")' + + +class TestHostSystemMetricsSplit: + + def test_cpu_average_writes_ecs_field_on_metrics_doc(self): + oid_mappings = { + 'table': { + 'component.cpu': { + 'columns': {'load_pct': '1.3.6.1.2.1.25.3.3.1.2'} + }, + } + } + averages = [{ + 'operation': 'average', + 'target': { + 'scope': 'table', + 'table': 'component.cpu', + 'field': 'component.cpu.load_pct', + }, + 'params': { + 'output_field': 'system.cpu.total.norm.pct', + 'multiply_value': 0.01, + }, + }] + filters = _generate_table_split_filters(oid_mappings, averages) + cpu = filters[0]['config']['code'] + assert '[system][cpu][total][norm][pct]' in cpu + assert 'Float(_avg_v)' in cpu + assert '* 0.01' in cpu + + def test_official_profile_averages_cores_to_ecs_cpu_field(self): + import json + from pathlib import Path + + path = ( + Path(__file__).resolve().parents[1] + / 'data' / 'official_profiles' / 'generic_host_system_metrics.json' + ) + profile = json.loads(path.read_text(encoding='utf-8')) + average = next(n for n in profile['normalizers'] if n['operation'] == 'average') + assert average['params']['output_field'] == 'system.cpu.total.norm.pct' + assert average['params']['multiply_value'] == 0.01 + assert 'promote_output_field' not in average.get('params', {}) + ratio = next(n for n in profile['normalizers'] if n['operation'] == 'ratio') + assert 'promote_output_field' not in ratio['params'] + assert 'promote_when_type' not in ratio['params'] + + +class TestRubyKeepWhenStatements: + + def test_empty_table_returns_empty_string(self): + assert _ruby_keep_when_statements({}) == '' + assert _ruby_keep_when_statements(None) == '' + + def test_emits_delete_and_include_check(self): + result = _ruby_keep_when_statements({ + 'keep_when': {'column': 'type', 'equals': ['10']}, + }) + assert 'row.delete("type")' in result + assert '["10"].include?(_keep_v.to_s)' in result + + def test_rejects_unsafe_column_names(self): + result = _ruby_keep_when_statements({ + 'keep_when': {'column': 'type"; exit', 'equals': ['10']}, + }) + assert result == '' + + def test_table_split_injects_keep_when_before_event_emit(self): + oid_mappings = { + 'table': { + 'component.fan': { + 'columns': { + 'description': '1.3.6.1.2.1.47.1.1.1.1.2', + 'type': '1.3.6.1.2.1.99.1.1.1.1', + }, + 'keep_when': {'column': 'type', 'equals': ['10']}, + } + } + } + filters = _generate_table_split_filters(oid_mappings) + code = filters[0]['config']['code'] + assert 'row.delete("type")' in code + assert '["10"].include?(_keep_v.to_s)' in code + assert code.index('row.delete("type")') < code.index('LogStash::Event.new') + + def test_tables_without_keep_when_are_unchanged(self): + oid_mappings = { + 'table': { + 'component.cpu': { + 'columns': {'load_pct': '1.3.6.1.2.1.25.3.3.1.2'} + } + } + } + filters = _generate_table_split_filters(oid_mappings) + code = filters[0]['config']['code'] + assert 'row.delete("type")' not in code + assert '_keep_v' not in code + + +class TestPaloaltoComponentsProfile: + + def _profile(self): + import json + from pathlib import Path + path = ( + Path(__file__).resolve().parents[1] + / 'data' / 'official_profiles' / 'paloalto_components.json' + ) + return json.loads(path.read_text(encoding='utf-8')) + + def test_maps_entity_sensor_onto_cisco_schema(self): + profile = self._profile() + fans = profile['table']['component.fan'] + sensors = profile['table']['component.sensor'] + assert fans['columns']['description'] == '1.3.6.1.2.1.47.1.1.1.1.2' + assert fans['columns']['state'] == '1.3.6.1.2.1.99.1.1.1.5' + assert fans['columns']['rpm'] == '1.3.6.1.2.1.99.1.1.1.4' + assert fans['keep_when'] == {'column': 'type', 'equals': ['10']} + assert sensors['columns']['description'] == '1.3.6.1.2.1.47.1.1.1.1.2' + assert sensors['columns']['temp.celsius'] == '1.3.6.1.2.1.99.1.1.1.4' + assert sensors['columns']['state'] == '1.3.6.1.2.1.99.1.1.1.5' + assert sensors['keep_when'] == {'column': 'type', 'equals': ['8']} + assert 'temp.threshold' not in sensors['columns'] + assert 'temp.last_shutdown' not in sensors['columns'] + + def test_firewall_template_uses_components_not_generic_entity_sensor(self): + import json + from pathlib import Path + path = ( + Path(__file__).resolve().parents[1] + / 'data' / 'official_device_templates' / 'palo_alto_firewall.json' + ) + template = json.loads(path.read_text(encoding='utf-8')) + assert 'paloalto_components' in template['profiles'] + assert 'generic_entity_sensor' not in template['profiles'] + + +# =========================================================================== +# _generate_snmp_error_cleanup_filter +# =========================================================================== + +class TestGenerateSnmpErrorCleanupFilter: + + def test_returns_a_dict(self): + result = _generate_snmp_error_cleanup_filter() + assert isinstance(result, dict) + + def test_plugin_is_ruby(self): + result = _generate_snmp_error_cleanup_filter() + assert result['plugin'] == 'ruby' + + def test_has_id(self): + result = _generate_snmp_error_cleanup_filter() + assert 'id' in result and result['id'] + + def test_code_contains_error_cleanup_logic(self): + result = _generate_snmp_error_cleanup_filter() + code = result['config']['code'] + assert 'error' in code.lower() diff --git a/tests/SNMP/unit/test_snmp_test.py b/tests/SNMP/unit/test_snmp_test.py new file mode 100644 index 0000000..f9c32c1 --- /dev/null +++ b/tests/SNMP/unit/test_snmp_test.py @@ -0,0 +1,920 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +""" +Tests for SNMP.snmp_test — format helpers, auth data creation, and the +RunSNMPTest / RunSNMPWalk view endpoints (SNMP network I/O mocked). +""" + +import json +import socket +import pytest +from unittest.mock import patch, MagicMock +from django.test import Client +from django.contrib.auth.models import User + +from SNMP.snmp_test import ( + _create_auth_data, + _device_poll_address, + _device_response_data, + _format_snmp_value, + _load_profile_data, + _merge_profile_oids, + _resolve_device_poll_address, +) +from SNMP.models import Device, DeviceTemplate, Profile, Credential, Network +from PipelineManager.models import Connection +from Management.models import UserProfile + + +# =========================================================================== +# Fixtures +# =========================================================================== + +@pytest.fixture +def admin_user(db): + user = User.objects.create_user( + username='snmp_test_admin', + password='testpass123', + email='snmp_test_admin@example.com' + ) + profile, created = UserProfile.objects.get_or_create(user=user, defaults={'role': 'admin'}) + if not created: + profile.role = 'admin' + profile.save() + return user + + +@pytest.fixture +def authenticated_client(admin_user): + client = Client() + client.force_login(admin_user) + return client + + +@pytest.fixture +def test_connection(db): + return Connection.objects.create( + name='SNMP Test Connection', + connection_type='CENTRALIZED', + host='https://localhost:9200', + username='elastic', + password='changeme' + ) + + +@pytest.fixture +def test_credential_v2c(db): + return Credential.objects.create( + name='snmp_test_cred_v2c', + version='2c', + community='public', + description='Test v2c credential for snmp_test tests' + ) + + +@pytest.fixture +def test_credential_v3_auth_priv(db): + return Credential.objects.create( + name='snmp_test_cred_v3', + version='3', + security_name='testuser', + security_level='authPriv', + auth_protocol='sha', + auth_pass='authpass123', + priv_protocol='aes', + priv_pass='privpass123', + ) + + +@pytest.fixture +def test_credential_v3_auth_no_priv(db): + return Credential.objects.create( + name='snmp_test_cred_v3_anp', + version='3', + security_name='testuser2', + security_level='authNoPriv', + auth_protocol='md5', + auth_pass='authpass123', + ) + + +@pytest.fixture +def test_credential_v3_no_auth(db): + return Credential.objects.create( + name='snmp_test_cred_v3_noanp', + version='3', + security_name='testuser3', + security_level='noAuthNoPriv', + ) + + +@pytest.fixture +def test_network(db, test_connection, test_credential_v2c): + return Network.objects.create( + name='SNMP Test Network', + network_range='10.0.0.0/24', + connection=test_connection, + discovery_credential=test_credential_v2c, + interval=30 + ) + + +@pytest.fixture +def test_custom_profile(db): + return Profile.objects.create( + name='snmp_test_custom_profile', + description='Custom profile for snmp_test tests', + vendor='Generic', + profile_data={ + 'get': {'sysDescr': '1.3.6.1.2.1.1.1.0'}, + 'walk': {}, + 'table': {} + } + ) + + +@pytest.fixture +def test_device_template(db, test_custom_profile): + template = DeviceTemplate.objects.create( + name='snmp_test_template', + description='Test template', + vendor='Generic', + ) + template.profiles.add(test_custom_profile) + return template + + +@pytest.fixture +def test_device(db, test_network, test_credential_v2c, test_device_template): + return Device.objects.create( + name='snmp_test_device', + ip_address='10.0.0.1', + port=161, + retries=1, + timeout=500, + credential=test_credential_v2c, + network=test_network, + device_template=test_device_template, + ) + + +# =========================================================================== +# _format_snmp_value — pure function +# =========================================================================== + +class TestFormatSnmpValue: + + def test_printable_string_returned_as_is(self): + assert _format_snmp_value('Hello World') == 'Hello World' + + def test_empty_string_returned_as_is(self): + assert _format_snmp_value('') == '' + + def test_numeric_string_returned_as_is(self): + assert _format_snmp_value('12345') == '12345' + + def test_mostly_printable_string_returned_as_is(self): + # All printable ASCII + value = 'Linux router 2.6.32' + assert _format_snmp_value(value) == value + + def test_six_byte_binary_value_formatted_as_mac(self): + # Simulate a 6-character string with non-printable bytes → MAC-like hex + binary = '\x00\x11\x22\x33\x44\x55' + result = _format_snmp_value(binary) + # Should be hex-formatted + assert ':' in result + + def test_four_byte_binary_value_formatted_as_hex(self): + binary = '\xc0\xa8\x01\x01' # 192.168.1.1 as binary + result = _format_snmp_value(binary) + assert ':' in result + + +# =========================================================================== +# _load_profile_data +# =========================================================================== + +class TestLoadProfileData: + + def test_returns_profile_data_for_custom_profile(self, test_custom_profile): + data = _load_profile_data(test_custom_profile) + assert data == test_custom_profile.profile_data + + def test_official_placeholder_loads_from_file(self, settings, tmp_path): + import os, json as jsonlib + # Point BASE_DIR at tmp_path and create the official profile file + settings.BASE_DIR = str(tmp_path) + profile_dir = tmp_path / 'SNMP' / 'data' / 'official_profiles' + profile_dir.mkdir(parents=True) + + profile_content = { + 'get': {'sysDescr': '1.3.6.1.2.1.1.1.0'}, + 'walk': {}, + 'table': {} + } + profile_file = profile_dir / 'test_official.json' + profile_file.write_text(jsonlib.dumps(profile_content)) + + official_profile = Profile( + name='test_official.json', + profile_data={'is_official_placeholder': True}, + vendor='Generic' + ) + data = _load_profile_data(official_profile) + assert data['get']['sysDescr'] == '1.3.6.1.2.1.1.1.0' + + def test_official_placeholder_missing_file_returns_empty(self, settings, tmp_path): + settings.BASE_DIR = str(tmp_path) + (tmp_path / 'SNMP' / 'data' / 'official_profiles').mkdir(parents=True) + + official_profile = Profile( + name='nonexistent.json', + profile_data={'is_official_placeholder': True}, + vendor='Generic' + ) + data = _load_profile_data(official_profile) + assert data == {'get': {}, 'walk': {}, 'table': {}} + + +# =========================================================================== +# _merge_profile_oids +# =========================================================================== + +class TestMergeProfileOids: + + def test_empty_profiles_returns_empty_structure(self): + result = _merge_profile_oids([]) + assert result == {'get': {}, 'walk': {}, 'table': {}} + + def test_single_profile_merged_correctly(self, test_custom_profile): + result = _merge_profile_oids([test_custom_profile]) + assert 'sysDescr' in result['get'] + + def test_multiple_profiles_oids_merged(self, db): + p1 = Profile.objects.create( + name='merge_test_p1', + vendor='Generic', + profile_data={'get': {'oid_a': '1.3.6.1.2.1.1.1.0'}, 'walk': {}, 'table': {}} + ) + p2 = Profile.objects.create( + name='merge_test_p2', + vendor='Generic', + profile_data={'get': {'oid_b': '1.3.6.1.2.1.1.2.0'}, 'walk': {}, 'table': {}} + ) + result = _merge_profile_oids([p1, p2]) + assert 'oid_a' in result['get'] + assert 'oid_b' in result['get'] + + def test_later_profile_overwrites_duplicate_oid_key(self, db): + p1 = Profile.objects.create( + name='merge_test_dup1', + vendor='Generic', + profile_data={'get': {'oid_x': '1.3.6.1.2.1.1.1.0'}, 'walk': {}, 'table': {}} + ) + p2 = Profile.objects.create( + name='merge_test_dup2', + vendor='Generic', + profile_data={'get': {'oid_x': '1.3.6.1.2.1.1.2.0'}, 'walk': {}, 'table': {}} + ) + result = _merge_profile_oids([p1, p2]) + # p2's value wins + assert result['get']['oid_x'] == '1.3.6.1.2.1.1.2.0' + + +# =========================================================================== +# Device poll address +# =========================================================================== + +class TestDevicePollAddress: + + def test_hostname_is_preferred_over_ip_address(self): + device = MagicMock( + id=1, + name='switch-1', + hostname='switch-1.example.com', + ip_address='10.0.0.1', + port=161, + ) + + assert _device_poll_address(device) == 'switch-1.example.com' + assert _device_response_data(device)['address'] == 'switch-1.example.com' + + def test_ip_address_is_used_when_hostname_is_empty(self): + device = MagicMock(hostname=None, ip_address='10.0.0.1') + + assert _device_poll_address(device) == '10.0.0.1' + + @patch('SNMP.snmp_test.socket.getaddrinfo') + def test_resolvable_hostname_is_used(self, mock_getaddrinfo): + mock_getaddrinfo.return_value = [(socket.AF_INET, socket.SOCK_DGRAM, 17, '', ('10.0.0.1', 161))] + device = MagicMock( + hostname='switch-1.example.com', + ip_address='10.0.0.1', + port=161, + ) + + address, warning = _resolve_device_poll_address(device) + + assert address == 'switch-1.example.com' + assert warning is None + + @patch('SNMP.snmp_test.socket.getaddrinfo', side_effect=socket.gaierror) + def test_unresolvable_hostname_falls_back_to_ip(self, mock_getaddrinfo): + device = MagicMock( + hostname='switch-1.example.com', + ip_address='10.0.0.1', + port=161, + ) + + address, warning = _resolve_device_poll_address(device) + + assert address == '10.0.0.1' + assert 'cannot resolve' in warning + assert 'Falling back' in warning + + @patch('SNMP.snmp_test.socket.getaddrinfo', side_effect=socket.gaierror) + def test_unresolvable_hostname_without_ip_raises_clear_error(self, mock_getaddrinfo): + device = MagicMock( + hostname='switch-1.example.com', + ip_address=None, + port=161, + ) + + with pytest.raises(ValueError, match='No fallback IP address'): + _resolve_device_poll_address(device) + + +# =========================================================================== +# _create_auth_data +# =========================================================================== + +class TestCreateAuthData: + + def test_v2c_creates_community_data(self, test_credential_v2c): + from pysnmp.hlapi.v3arch.asyncio import CommunityData + auth = _create_auth_data(test_credential_v2c) + assert isinstance(auth, CommunityData) + + def test_v3_no_auth_no_priv_creates_usm(self, test_credential_v3_no_auth): + from pysnmp.hlapi.v3arch.asyncio import UsmUserData + auth = _create_auth_data(test_credential_v3_no_auth) + assert isinstance(auth, UsmUserData) + + def test_v3_auth_no_priv_creates_usm_with_auth(self, test_credential_v3_auth_no_priv): + from pysnmp.hlapi.v3arch.asyncio import UsmUserData + auth = _create_auth_data(test_credential_v3_auth_no_priv) + assert isinstance(auth, UsmUserData) + + def test_v3_auth_priv_creates_usm_with_auth_and_priv(self, test_credential_v3_auth_priv): + from pysnmp.hlapi.v3arch.asyncio import UsmUserData + auth = _create_auth_data(test_credential_v3_auth_priv) + assert isinstance(auth, UsmUserData) + + def test_v2c_missing_community_raises_value_error(self): + # Use an unsaved model instance to bypass model-level validation + # and test the _create_auth_data logic directly + cred = Credential( + name='snmp_test_cred_nocommunity', + version='2c', + community='', # explicitly empty, overriding default='public' + ) + with pytest.raises(ValueError, match='no community string'): + _create_auth_data(cred) + + def test_v3_auth_no_priv_missing_auth_protocol_raises(self): + # Use unsaved instance: model validates auth_protocol at save() time, + # but _create_auth_data validates it independently at runtime + cred = Credential( + version='3', + security_name='user', + security_level='authNoPriv', + auth_protocol='', # no protocol + auth_pass='', + ) + with pytest.raises(ValueError, match='auth protocol'): + _create_auth_data(cred) + + def test_v3_auth_priv_missing_priv_protocol_raises(self): + # Use unsaved instance to bypass model validation + cred = Credential( + version='3', + security_name='user', + security_level='authPriv', + auth_protocol='sha', + auth_pass='authpass123', + priv_protocol='', # no priv protocol + priv_pass='', + ) + with pytest.raises(ValueError, match='privacy protocol'): + _create_auth_data(cred) + + def test_unknown_version_raises_value_error(self): + cred = Credential( + name='bad_version_cred', + version='9', + security_name='user', + ) + with pytest.raises(ValueError, match='Unknown SNMP version'): + _create_auth_data(cred) + + +# =========================================================================== +# RunSNMPTest view +# =========================================================================== + +@pytest.mark.django_db +class TestRunSNMPTestView: + + def test_missing_device_id_returns_400(self, authenticated_client): + response = authenticated_client.post( + '/SNMP/RunSNMPTest/', + data=json.dumps({}), + content_type='application/json' + ) + assert response.status_code == 400 + data = json.loads(response.content) + assert data['success'] is False + assert 'device_id' in data['error'] + + def test_device_not_found_returns_404(self, authenticated_client): + response = authenticated_client.post( + '/SNMP/RunSNMPTest/', + data=json.dumps({'device_id': 999999}), + content_type='application/json' + ) + assert response.status_code == 404 + data = json.loads(response.content) + assert data['success'] is False + + def test_device_without_credential_returns_400(self, authenticated_client, test_network, test_device_template, db): + # Create device with no credential by bypassing model validation + # We'll use a credential initially then remove it + cred = Credential.objects.create( + name='temp_cred_for_removal', + version='2c', + community='public' + ) + device = Device.objects.create( + name='device_no_cred', + ip_address='10.0.0.99', + port=161, + retries=1, + timeout=500, + credential=cred, + network=test_network, + device_template=test_device_template, + ) + # Remove credential by setting it to None directly in DB + Device.objects.filter(pk=device.pk).update(credential=None) + + response = authenticated_client.post( + '/SNMP/RunSNMPTest/', + data=json.dumps({'device_id': device.pk}), + content_type='application/json' + ) + assert response.status_code == 400 + data = json.loads(response.content) + assert data['success'] is False + assert 'credential' in data['error'].lower() + + def test_template_not_found_returns_404(self, authenticated_client, test_device): + response = authenticated_client.post( + '/SNMP/RunSNMPTest/', + data=json.dumps({'device_id': test_device.pk, 'template_id': 999999}), + content_type='application/json' + ) + assert response.status_code == 404 + data = json.loads(response.content) + assert data['success'] is False + + def test_template_with_no_profiles_returns_400(self, authenticated_client, test_network, + test_credential_v2c, db): + empty_template = DeviceTemplate.objects.create( + name='snmp_test_empty_template', + vendor='Generic', + ) + device = Device.objects.create( + name='device_empty_template', + ip_address='10.0.0.50', + port=161, + retries=1, + timeout=500, + credential=test_credential_v2c, + network=test_network, + device_template=empty_template, + ) + + response = authenticated_client.post( + '/SNMP/RunSNMPTest/', + data=json.dumps({'device_id': device.pk, 'template_id': empty_template.pk}), + content_type='application/json' + ) + assert response.status_code == 400 + data = json.loads(response.content) + assert data['success'] is False + assert 'profiles' in data['error'].lower() + + @patch('SNMP.snmp_test._perform_snmp_get') + @patch('SNMP.snmp_test._perform_snmp_walk') + @patch('SNMP.snmp_test._perform_snmp_table') + def test_successful_snmp_test_returns_200_with_results( + self, mock_table, mock_walk, mock_get, + authenticated_client, test_device + ): + mock_get.return_value = {'sysDescr': 'Linux router'} + mock_walk.return_value = {} + mock_table.return_value = {} + + response = authenticated_client.post( + '/SNMP/RunSNMPTest/', + data=json.dumps({'device_id': test_device.pk}), + content_type='application/json' + ) + assert response.status_code == 200 + data = json.loads(response.content) + assert data['success'] is True + assert 'results' in data + assert data['device']['id'] == test_device.pk + assert data['device']['address'] == test_device.ip_address + assert data['template']['name'] == test_device.device_template.name + + @patch('SNMP.snmp_test._perform_snmp_get') + @patch('SNMP.snmp_test._perform_snmp_walk') + @patch('SNMP.snmp_test._perform_snmp_table') + def test_hostname_device_response_uses_hostname_as_poll_address( + self, mock_table, mock_walk, mock_get, + authenticated_client, test_device + ): + test_device.hostname = 'switch-1.example.com' + test_device.save(update_fields=['hostname']) + mock_get.return_value = {'sysDescr': 'Network switch'} + mock_walk.return_value = {} + mock_table.return_value = {} + + with patch( + 'SNMP.snmp_test._resolve_device_poll_address', + return_value=('switch-1.example.com', None), + ): + response = authenticated_client.post( + '/SNMP/RunSNMPTest/', + data=json.dumps({'device_id': test_device.pk}), + content_type='application/json' + ) + + assert response.status_code == 200 + data = json.loads(response.content) + assert data['success'] is True + assert data['device']['address'] == 'switch-1.example.com' + assert data['device']['hostname'] == 'switch-1.example.com' + assert data['device']['ip_address'] == '10.0.0.1' + + @patch('SNMP.snmp_test._perform_snmp_get') + @patch('SNMP.snmp_test._perform_snmp_walk') + @patch('SNMP.snmp_test._perform_snmp_table') + def test_unresolvable_hostname_falls_back_to_ip_and_returns_warning( + self, mock_table, mock_walk, mock_get, + authenticated_client, test_device + ): + test_device.hostname = 'switch-1.example.com' + test_device.save(update_fields=['hostname']) + mock_get.return_value = {'sysDescr': 'Network switch'} + mock_walk.return_value = {} + mock_table.return_value = {} + + warning = ( + "The machine running LogstashUI cannot resolve hostname " + "'switch-1.example.com'. Falling back to IP address 10.0.0.1." + ) + with patch( + 'SNMP.snmp_test._resolve_device_poll_address', + return_value=('10.0.0.1', warning), + ): + response = authenticated_client.post( + '/SNMP/RunSNMPTest/', + data=json.dumps({'device_id': test_device.pk}), + content_type='application/json' + ) + + data = json.loads(response.content) + assert data['success'] is True + assert data['device']['address'] == '10.0.0.1' + assert data['address_warning'] == warning + mock_get.assert_called_once_with( + test_device, test_device.credential, + test_device.device_template.profiles.first().profile_data['get'], + '10.0.0.1' + ) + + @patch('SNMP.snmp_test.socket.getaddrinfo', side_effect=socket.gaierror) + def test_unresolvable_hostname_only_device_returns_clear_error( + self, mock_getaddrinfo, authenticated_client, test_device + ): + test_device.hostname = 'switch-1.example.com' + test_device.ip_address = None + test_device.save(update_fields=['hostname', 'ip_address']) + + response = authenticated_client.post( + '/SNMP/RunSNMPTest/', + data=json.dumps({'device_id': test_device.pk}), + content_type='application/json' + ) + + assert response.status_code == 400 + data = json.loads(response.content) + assert data['success'] is False + assert data['error'] == ( + "The machine running LogstashUI cannot resolve hostname " + "'switch-1.example.com'. No fallback IP address is configured " + "for this device." + ) + + @patch('SNMP.snmp_test._perform_snmp_get') + @patch('SNMP.snmp_test._perform_snmp_walk') + @patch('SNMP.snmp_test._perform_snmp_table') + def test_auth_failure_returns_success_false_with_auth_error( + self, mock_table, mock_walk, mock_get, + authenticated_client, test_device + ): + auth_error = {'error': 'Unknown USM user'} + mock_get.return_value = {'sysDescr': auth_error} + mock_walk.return_value = {} + mock_table.return_value = {} + + response = authenticated_client.post( + '/SNMP/RunSNMPTest/', + data=json.dumps({'device_id': test_device.pk}), + content_type='application/json' + ) + assert response.status_code == 200 + data = json.loads(response.content) + assert data['success'] is False + assert 'authentication' in data['error'].lower() or 'auth' in data['error'].lower() + + @patch('SNMP.snmp_test._perform_snmp_get') + @patch('SNMP.snmp_test._perform_snmp_walk') + @patch('SNMP.snmp_test._perform_snmp_table') + def test_all_operations_fail_returns_success_false( + self, mock_table, mock_walk, mock_get, + authenticated_client, test_device + ): + mock_get.return_value = {'sysDescr': {'error': 'No response from device'}} + mock_walk.return_value = {} + mock_table.return_value = {} + + response = authenticated_client.post( + '/SNMP/RunSNMPTest/', + data=json.dumps({'device_id': test_device.pk}), + content_type='application/json' + ) + assert response.status_code == 200 + data = json.loads(response.content) + assert data['success'] is False + + @patch('SNMP.snmp_test._perform_snmp_get') + @patch('SNMP.snmp_test._perform_snmp_walk') + @patch('SNMP.snmp_test._perform_snmp_table') + def test_partial_success_returns_success_true_with_has_errors( + self, mock_table, mock_walk, mock_get, + authenticated_client, test_device, test_custom_profile, db + ): + # Profile has two GET OIDs so we can simulate one success and one failure + test_custom_profile.profile_data = { + 'get': {'sysDescr': '1.3.6.1.2.1.1.1.0', 'sysUpTime': '1.3.6.1.2.1.1.3.0'}, + 'walk': {}, + 'table': {} + } + test_custom_profile.save() + + mock_get.return_value = { + 'sysDescr': 'Linux router', + 'sysUpTime': {'error': 'No such object'} + } + mock_walk.return_value = {} + mock_table.return_value = {} + + response = authenticated_client.post( + '/SNMP/RunSNMPTest/', + data=json.dumps({'device_id': test_device.pk}), + content_type='application/json' + ) + assert response.status_code == 200 + data = json.loads(response.content) + assert data['success'] is True + assert data['has_errors'] is True + + def test_get_method_not_allowed(self, authenticated_client): + response = authenticated_client.get('/SNMP/RunSNMPTest/') + assert response.status_code == 405 + + @patch('SNMP.snmp_test._perform_snmp_get') + @patch('SNMP.snmp_test._perform_snmp_walk') + @patch('SNMP.snmp_test._perform_snmp_table') + def test_explicit_template_id_overrides_device_template( + self, mock_table, mock_walk, mock_get, + authenticated_client, test_device, db + ): + mock_get.return_value = {'sysDescr': 'Linux router'} + mock_walk.return_value = {} + mock_table.return_value = {} + + # Create a second template with a profile + extra_profile = Profile.objects.create( + name='snmp_test_extra_profile', + vendor='Generic', + profile_data={'get': {'sysContact': '1.3.6.1.2.1.1.4.0'}, 'walk': {}, 'table': {}} + ) + extra_template = DeviceTemplate.objects.create( + name='snmp_test_extra_template', + vendor='Generic', + ) + extra_template.profiles.add(extra_profile) + + response = authenticated_client.post( + '/SNMP/RunSNMPTest/', + data=json.dumps({'device_id': test_device.pk, 'template_id': extra_template.pk}), + content_type='application/json' + ) + assert response.status_code == 200 + data = json.loads(response.content) + assert data['template']['name'] == extra_template.name + + @patch('SNMP.snmp_test._perform_snmp_get') + @patch('SNMP.snmp_test._perform_snmp_walk') + @patch('SNMP.snmp_test._perform_snmp_table') + def test_response_contains_execution_time( + self, mock_table, mock_walk, mock_get, + authenticated_client, test_device + ): + mock_get.return_value = {'sysDescr': 'Linux router'} + mock_walk.return_value = {} + mock_table.return_value = {} + + response = authenticated_client.post( + '/SNMP/RunSNMPTest/', + data=json.dumps({'device_id': test_device.pk}), + content_type='application/json' + ) + data = json.loads(response.content) + assert 'execution_time' in data + + +# =========================================================================== +# RunSNMPWalk view +# =========================================================================== + +@pytest.mark.django_db +class TestRunSNMPWalkView: + + def test_missing_host_returns_400(self, authenticated_client, test_credential_v2c): + response = authenticated_client.post( + '/SNMP/RunSNMPWalk/', + data=json.dumps({'credential_id': test_credential_v2c.pk}), + content_type='application/json' + ) + assert response.status_code == 400 + data = json.loads(response.content) + assert data['success'] is False + assert 'host' in data['error'] + + def test_missing_credential_id_returns_400(self, authenticated_client): + response = authenticated_client.post( + '/SNMP/RunSNMPWalk/', + data=json.dumps({'host': '10.0.0.1'}), + content_type='application/json' + ) + assert response.status_code == 400 + data = json.loads(response.content) + assert data['success'] is False + assert 'credential_id' in data['error'] + + def test_credential_not_found_returns_404(self, authenticated_client): + response = authenticated_client.post( + '/SNMP/RunSNMPWalk/', + data=json.dumps({'host': '10.0.0.1', 'credential_id': 999999}), + content_type='application/json' + ) + assert response.status_code == 404 + data = json.loads(response.content) + assert data['success'] is False + + @patch('SNMP.snmp_test._perform_full_walk') + def test_successful_walk_returns_results(self, mock_walk, authenticated_client, test_credential_v2c): + mock_walk.return_value = { + 'results': [ + {'oid': '1.3.6.1.2.1.1.1.0', 'value': 'Linux router'}, + {'oid': '1.3.6.1.2.1.1.2.0', 'value': '1.3.6.1.4.1.8072.3.2.10'}, + ] + } + + response = authenticated_client.post( + '/SNMP/RunSNMPWalk/', + data=json.dumps({'host': '10.0.0.1', 'credential_id': test_credential_v2c.pk}), + content_type='application/json' + ) + assert response.status_code == 200 + data = json.loads(response.content) + assert data['success'] is True + assert data['oid_count'] == 2 + assert len(data['results']) == 2 + assert data['host'] == '10.0.0.1' + + @patch('SNMP.snmp_test._perform_full_walk') + def test_walk_error_without_results_returns_success_false( + self, mock_walk, authenticated_client, test_credential_v2c + ): + mock_walk.return_value = {'error': 'No response from device', 'results': []} + + response = authenticated_client.post( + '/SNMP/RunSNMPWalk/', + data=json.dumps({'host': '10.0.0.1', 'credential_id': test_credential_v2c.pk}), + content_type='application/json' + ) + assert response.status_code == 200 + data = json.loads(response.content) + assert data['success'] is False + assert 'error' in data + + @patch('SNMP.snmp_test._perform_full_walk') + def test_walk_with_partial_error_and_results_returns_success_true( + self, mock_walk, authenticated_client, test_credential_v2c + ): + mock_walk.return_value = { + 'results': [{'oid': '1.3.6.1.2.1.1.1.0', 'value': 'Linux'}], + 'error': 'End of MIB' + } + + response = authenticated_client.post( + '/SNMP/RunSNMPWalk/', + data=json.dumps({'host': '10.0.0.1', 'credential_id': test_credential_v2c.pk}), + content_type='application/json' + ) + assert response.status_code == 200 + data = json.loads(response.content) + assert data['success'] is True + assert data['partial_error'] == 'End of MIB' + + @patch('SNMP.snmp_test._perform_full_walk') + def test_custom_port_and_start_oid_forwarded( + self, mock_walk, authenticated_client, test_credential_v2c + ): + mock_walk.return_value = {'results': []} + + authenticated_client.post( + '/SNMP/RunSNMPWalk/', + data=json.dumps({ + 'host': '10.0.0.1', + 'port': 1161, + 'credential_id': test_credential_v2c.pk, + 'start_oid': '1.3.6.1.2.1.2' + }), + content_type='application/json' + ) + mock_walk.assert_called_once() + call_args = mock_walk.call_args + assert call_args[0][1] == 1161 # port + assert call_args[0][3] == '1.3.6.1.2.1.2' # start_oid + + @patch('SNMP.snmp_test._perform_full_walk') + def test_default_port_is_161(self, mock_walk, authenticated_client, test_credential_v2c): + mock_walk.return_value = {'results': []} + + authenticated_client.post( + '/SNMP/RunSNMPWalk/', + data=json.dumps({'host': '10.0.0.1', 'credential_id': test_credential_v2c.pk}), + content_type='application/json' + ) + call_args = mock_walk.call_args + assert call_args[0][1] == 161 + + @patch('SNMP.snmp_test._perform_full_walk') + def test_default_start_oid_is_1_3_6_1(self, mock_walk, authenticated_client, test_credential_v2c): + mock_walk.return_value = {'results': []} + + authenticated_client.post( + '/SNMP/RunSNMPWalk/', + data=json.dumps({'host': '10.0.0.1', 'credential_id': test_credential_v2c.pk}), + content_type='application/json' + ) + call_args = mock_walk.call_args + assert call_args[0][3] == '1.3.6.1' + + @patch('SNMP.snmp_test._perform_full_walk') + def test_response_includes_execution_time( + self, mock_walk, authenticated_client, test_credential_v2c + ): + mock_walk.return_value = {'results': []} + + response = authenticated_client.post( + '/SNMP/RunSNMPWalk/', + data=json.dumps({'host': '10.0.0.1', 'credential_id': test_credential_v2c.pk}), + content_type='application/json' + ) + data = json.loads(response.content) + assert 'execution_time' in data + + def test_get_method_not_allowed(self, authenticated_client): + response = authenticated_client.get('/SNMP/RunSNMPWalk/') + assert response.status_code == 405 diff --git a/tests/SNMP/unit/test_views.py b/tests/SNMP/unit/test_views.py new file mode 100644 index 0000000..d76d794 --- /dev/null +++ b/tests/SNMP/unit/test_views.py @@ -0,0 +1,1098 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +import pytest +from django.contrib.auth.models import User +from django.test import Client +from unittest.mock import patch, MagicMock +import json +import os + +from SNMP.models import Network, Device, Credential, Profile +from PipelineManager.models import Connection +from Management.models import UserProfile + + +@pytest.fixture +def admin_user(db): + """Create a user with admin profile""" + user = User.objects.create_user( + username='admin_user', + password='testpass123', + email='admin@example.com' + ) + profile, created = UserProfile.objects.get_or_create(user=user, defaults={'role': 'admin'}) + if not created: + profile.role = 'admin' + profile.save() + return user + + +@pytest.fixture +def readonly_user(db): + """Create a user with readonly profile""" + user = User.objects.create_user( + username='readonly_user', + password='testpass123', + email='readonly@example.com' + ) + profile = UserProfile.objects.get(user=user) + profile.role = 'readonly' + profile.save() + user.refresh_from_db() + return user + + +@pytest.fixture +def authenticated_client(admin_user): + """Create an authenticated client with admin user""" + client = Client() + client.force_login(admin_user) + return client + + +@pytest.fixture +def readonly_client(readonly_user): + """Create an authenticated client with readonly user""" + client = Client() + client.force_login(readonly_user) + return client + + +@pytest.fixture +def test_connection(db): + """Create a test Elasticsearch connection""" + return Connection.objects.create( + name='Test Connection', + connection_type='CENTRALIZED', + host='https://localhost:9200', + username='elastic', + password='changeme' + ) + + +@pytest.fixture +def test_credential(db): + """Create a test SNMP credential""" + return Credential.objects.create( + name='Test Credential', + version='2c', + community='public', + description='Test SNMP v2c credential' + ) + + +@pytest.fixture +def test_network(db, test_connection, test_credential): + """Create a test SNMP network""" + return Network.objects.create( + name='Test Network', + network_range='192.168.1.0/24', + connection=test_connection, + discovery_credential=test_credential, + discovery_enabled=True, + traps_enabled=False, + interval=30 + ) + + +@pytest.fixture +def test_device(db, test_network, test_credential): + """Create a test SNMP device""" + return Device.objects.create( + name='Test Device', + ip_address='192.168.1.100', + port=161, + retries=2, + timeout=1000, + credential=test_credential, + network=test_network + ) + + +@pytest.fixture +def test_profile(db): + """Create a test user profile""" + return Profile.objects.create( + name='custom_profile', + description='Custom test profile', + vendor='Generic', + profile_data={ + 'get': { + 'test.metric': '1.3.6.1.2.1.1.1.0' + }, + 'walk': {}, + 'table': {} + } + ) + + +# ============================================================================ +# View Tests - Read-Only Pages +# ============================================================================ + +@pytest.mark.django_db +class TestNetworksView: + """Test Networks page view""" + + def test_networks_view_requires_authentication(self, client): + """Test that Networks view requires authentication""" + response = client.get('/SNMP/Networks/') + assert response.status_code == 302 + assert '/Management/Login/' in response.url + + def test_networks_view_accessible_to_admin(self, authenticated_client): + """Test that admin users can access Networks view""" + response = authenticated_client.get('/SNMP/Networks/') + assert response.status_code == 200 + assert b'Networks' in response.content or b'networks' in response.content + + def test_networks_view_accessible_to_readonly(self, readonly_client): + """Test that readonly users can access Networks view""" + response = readonly_client.get('/SNMP/Networks/') + assert response.status_code == 200 + + def test_networks_view_displays_networks(self, authenticated_client, test_network): + """Test that Networks view displays existing networks""" + response = authenticated_client.get('/SNMP/Networks/') + assert response.status_code == 200 + # Networks are loaded via AJAX, so just verify the page loads and has the networks context + assert 'networks' in response.context + + def test_networks_view_with_connection_form(self, authenticated_client): + """Test that Networks view includes connection form""" + response = authenticated_client.get('/SNMP/Networks/') + assert response.status_code == 200 + # Should have form context + assert 'form' in response.context + + +@pytest.mark.django_db +class TestDevicesView: + """Test Devices page view""" + + def test_devices_view_requires_authentication(self, client): + """Test that Devices view requires authentication""" + response = client.get('/SNMP/Devices/') + assert response.status_code == 302 + assert '/Management/Login/' in response.url + + def test_devices_view_accessible_to_admin(self, authenticated_client): + """Test that admin users can access Devices view""" + response = authenticated_client.get('/SNMP/Devices/') + assert response.status_code == 200 + + def test_devices_view_accessible_to_readonly(self, readonly_client): + """Test that readonly users can access Devices view""" + response = readonly_client.get('/SNMP/Devices/') + assert response.status_code == 200 + + def test_devices_view_displays_devices(self, authenticated_client, test_device): + """Test that Devices view displays existing devices""" + response = authenticated_client.get('/SNMP/Devices/') + assert response.status_code == 200 + # Device data is loaded via AJAX, so just check page loads + assert b'devices' in response.content.lower() + + +@pytest.mark.django_db +class TestProfilesView: + """Test DeviceTemplates page (profiles now live there; /SNMP/Profiles/ no longer exists)""" + + def test_device_templates_accessible_to_admin(self, authenticated_client): + """Admin users can access DeviceTemplates (the new home for profiles)""" + response = authenticated_client.get('/SNMP/DeviceTemplates/') + assert response.status_code == 200 + + def test_device_templates_accessible_to_readonly(self, readonly_client): + """Readonly users can access DeviceTemplates""" + response = readonly_client.get('/SNMP/DeviceTemplates/') + assert response.status_code == 200 + + def test_device_templates_exposes_profiles_in_context(self, authenticated_client): + """DeviceTemplates view includes 'profiles' list in its context""" + response = authenticated_client.get('/SNMP/DeviceTemplates/') + assert response.status_code == 200 + assert 'profiles' in response.context + + def test_device_templates_displays_user_profiles(self, authenticated_client, test_profile): + """User-created profiles appear in the DeviceTemplates context""" + response = authenticated_client.get('/SNMP/DeviceTemplates/') + assert response.status_code == 200 + profiles = response.context['profiles'] + assert any(p['name'] == 'custom_profile' for p in profiles) + + def test_device_templates_excludes_placeholder_profiles(self, authenticated_client): + """Placeholder profiles are excluded from the profiles list on DeviceTemplates""" + Profile.objects.create( + name='placeholder.json', + vendor='Generic', + profile_data={'is_official_placeholder': True}, + description='Placeholder' + ) + response = authenticated_client.get('/SNMP/DeviceTemplates/') + assert response.status_code == 200 + profiles = response.context['profiles'] + user_profiles = [p for p in profiles if not p['is_official']] + assert not any(p['name'] == 'placeholder.json' for p in user_profiles) + + def test_device_templates_profiles_sorted_alphabetically(self, authenticated_client): + """Profiles are sorted alphabetically by display_name on DeviceTemplates""" + response = authenticated_client.get('/SNMP/DeviceTemplates/') + assert response.status_code == 200 + profiles = response.context['profiles'] + display_names = [p['display_name'] for p in profiles] + assert display_names == sorted(display_names) + + +@pytest.mark.django_db +class TestCredentialsView: + """Test Credentials page view""" + + def test_credentials_view_requires_authentication(self, client): + """Test that Credentials view requires authentication""" + response = client.get('/SNMP/Credentials/') + assert response.status_code == 302 + assert '/Management/Login/' in response.url + + def test_credentials_view_accessible_to_admin(self, authenticated_client): + """Test that admin users can access Credentials view""" + response = authenticated_client.get('/SNMP/Credentials/') + assert response.status_code == 200 + + def test_credentials_view_accessible_to_readonly(self, readonly_client): + """Test that readonly users can access Credentials view""" + response = readonly_client.get('/SNMP/Credentials/') + assert response.status_code == 200 + + def test_credentials_view_displays_credentials(self, authenticated_client, test_credential): + """Test that Credentials view displays existing credentials""" + response = authenticated_client.get('/SNMP/Credentials/') + assert response.status_code == 200 + # Credentials data is loaded via AJAX, so just check page loads + assert b'credentials' in response.content.lower() + + +# ============================================================================ +# Edge Cases and Error Handling +# ============================================================================ + +@pytest.mark.django_db +class TestViewsEdgeCases: + """Test edge cases and error handling in views""" + + def test_networks_view_with_no_networks(self, authenticated_client): + """Test Networks view when no networks exist""" + response = authenticated_client.get('/SNMP/Networks/') + assert response.status_code == 200 + assert 'networks' in response.context + assert len(response.context['networks']) == 0 + + def test_device_templates_with_invalid_json_file(self, authenticated_client, settings): + """DeviceTemplates handles malformed official profile JSON files gracefully""" + official_profiles_dir = os.path.join(settings.BASE_DIR, 'SNMP', 'data', 'official_profiles') + if os.path.exists(official_profiles_dir): + invalid_file = os.path.join(official_profiles_dir, 'test_invalid.json') + try: + with open(invalid_file, 'w') as f: + f.write('{ invalid json }') + response = authenticated_client.get('/SNMP/DeviceTemplates/') + assert response.status_code == 200 + finally: + if os.path.exists(invalid_file): + os.remove(invalid_file) + + def test_view_with_database_error(self, authenticated_client): + """Test views handle database errors gracefully""" + with patch('SNMP.models.Network.objects') as mock_objects: + mock_objects.select_related.side_effect = Exception("Database error") + + # Should return error response, not crash + try: + response = authenticated_client.get('/SNMP/Networks/') + # May return 500 or handle gracefully + assert response.status_code in [200, 500] + except Exception: + # If exception is raised, that's also acceptable for this test + pass + + +# ============================================================================ +# Additional context / content verification tests +# ============================================================================ + +@pytest.mark.django_db +class TestViewContextContent: + """Additional tests verifying the data passed to each template context""" + + def test_networks_context_contains_network_instance(self, authenticated_client, test_network): + """Networks context 'networks' queryset contains our test network""" + response = authenticated_client.get('/SNMP/Networks/') + assert response.status_code == 200 + network_names = [n.name for n in response.context['networks']] + assert 'Test Network' in network_names + + def test_devices_context_has_devices_key(self, authenticated_client, test_device): + """Devices view passes 'devices' queryset to template""" + response = authenticated_client.get('/SNMP/Devices/') + assert response.status_code == 200 + assert 'devices' in response.context + device_names = [d.name for d in response.context['devices']] + assert 'Test Device' in device_names + + def test_credentials_context_has_credentials_key(self, authenticated_client, test_credential): + """Credentials view passes 'credentials' queryset to template""" + response = authenticated_client.get('/SNMP/Credentials/') + assert response.status_code == 200 + assert 'credentials' in response.context + cred_names = [c.name for c in response.context['credentials']] + assert 'Test Credential' in cred_names + + def test_device_templates_user_profile_required_fields(self, authenticated_client, test_profile): + """User profile dicts in DeviceTemplates context contain all required fields""" + response = authenticated_client.get('/SNMP/DeviceTemplates/') + assert response.status_code == 200 + user_profiles = [p for p in response.context['profiles'] if not p['is_official']] + for p in user_profiles: + for key in ('name', 'display_name', 'description', 'vendor', 'is_official'): + assert key in p + + def test_device_templates_official_profile_fields(self, authenticated_client): + """Official profiles in DeviceTemplates context always have is_official=True""" + response = authenticated_client.get('/SNMP/DeviceTemplates/') + assert response.status_code == 200 + official_profiles = [p for p in response.context['profiles'] if p['is_official']] + for p in official_profiles: + assert p['is_official'] is True + for key in ('name', 'display_name', 'description', 'vendor'): + assert key in p + + def test_device_templates_invalid_json_handled_gracefully(self, authenticated_client, settings, tmp_path): + """DeviceTemplates gracefully skips official profile JSON files that cannot be parsed""" + official_dir = tmp_path / 'official_profiles' + official_dir.mkdir() + (official_dir / 'broken.json').write_text('{ not valid json }') + + with patch('SNMP.views.os.path.exists', return_value=True), \ + patch('SNMP.views.os.listdir', return_value=['broken.json']): + response = authenticated_client.get('/SNMP/DeviceTemplates/') + assert response.status_code == 200 + profiles = response.context['profiles'] + official_names = [p['name'] for p in profiles if p['is_official']] + assert 'broken' in official_names + broken = next(p for p in profiles if p['name'] == 'broken') + assert broken['description'] == '' + + def test_networks_view_form_is_connection_form(self, authenticated_client): + """Networks context form is a ConnectionForm instance""" + from PipelineManager.forms import ConnectionForm + response = authenticated_client.get('/SNMP/Networks/') + assert isinstance(response.context['form'], ConnectionForm) + + def test_profiles_alphabetical_sort_among_unpinned(self, authenticated_client): + """User profiles are sorted alphabetically by display_name on DeviceTemplates""" + Profile.objects.all().delete() # start clean for this test + Profile.objects.create(name='zebra_profile', vendor='Generic', description='', profile_data={'get': {}}) + Profile.objects.create(name='alpha_profile', vendor='Generic', description='', profile_data={'get': {}}) + + response = authenticated_client.get('/SNMP/DeviceTemplates/') + assert response.status_code == 200 + user_profiles = [p for p in response.context['profiles'] if not p['is_official']] + display_names = [p['display_name'] for p in user_profiles] + assert display_names == sorted(display_names) + + +# ============================================================================ +# Onboarding View Tests +# ============================================================================ + +@pytest.mark.django_db +class TestOnboardingView: + """Test the SNMP Onboarding page view""" + + def test_onboarding_requires_authentication(self, client): + response = client.get('/SNMP/Onboarding/') + assert response.status_code == 302 + assert '/Management/Login/' in response.url + + def test_onboarding_accessible_to_admin(self, authenticated_client): + response = authenticated_client.get('/SNMP/Onboarding/') + assert response.status_code == 200 + + def test_onboarding_accessible_to_readonly(self, readonly_client): + response = readonly_client.get('/SNMP/Onboarding/') + assert response.status_code == 200 + + def test_onboarding_has_all_required_context_keys(self, authenticated_client, test_network, test_credential, test_device): + response = authenticated_client.get('/SNMP/Onboarding/') + assert response.status_code == 200 + for key in ('connections', 'credentials', 'networks', 'templates', 'devices', 'device_count', 'form'): + assert key in response.context, f"Missing context key: {key}" + + def test_onboarding_connections_include_suggested_kibana_url(self, authenticated_client, test_connection): + response = authenticated_client.get('/SNMP/Onboarding/') + assert response.status_code == 200 + connections = list(response.context['connections']) + assert connections, 'expected at least the test_connection' + assert 'suggested_kibana_url' in connections[0] + + def test_onboarding_device_count_matches_db(self, authenticated_client, test_device): + from SNMP.models import Device as _Device + response = authenticated_client.get('/SNMP/Onboarding/') + assert response.status_code == 200 + assert response.context['device_count'] == _Device.objects.count() + + def test_onboarding_networks_excludes_nothing(self, authenticated_client, test_network): + response = authenticated_client.get('/SNMP/Onboarding/') + assert response.status_code == 200 + network_names = [n.name for n in response.context['networks']] + assert test_network.name in network_names + + def test_onboarding_templates_excludes_default(self, authenticated_client): + from SNMP.models import DeviceTemplate + DeviceTemplate.objects.create(name='default', vendor='Generic', description='default template') + DeviceTemplate.objects.create(name='custom_tpl', vendor='Generic', description='custom') + response = authenticated_client.get('/SNMP/Onboarding/') + assert response.status_code == 200 + template_names = [t.name for t in response.context['templates']] + assert 'default' not in template_names + assert 'custom_tpl' in template_names + + +# ============================================================================ +# CheckDeviceType View Tests +# ============================================================================ + +@pytest.mark.django_db +class TestCheckDeviceTypeView: + """Test the CheckDeviceType view (lightweight SNMP probe)""" + + def test_get_method_returns_405(self, authenticated_client): + response = authenticated_client.get('/SNMP/CheckDeviceType/') + assert response.status_code == 405 + + def test_missing_host_returns_400(self, authenticated_client, test_credential): + data = json.dumps({'credential_id': test_credential.id}) + response = authenticated_client.post('/SNMP/CheckDeviceType/', data, content_type='application/json') + assert response.status_code == 400 + assert 'host is required' in response.json()['error'] + + def test_missing_credential_id_returns_400(self, authenticated_client): + data = json.dumps({'host': '192.168.1.1'}) + response = authenticated_client.post('/SNMP/CheckDeviceType/', data, content_type='application/json') + assert response.status_code == 400 + assert 'credential_id is required' in response.json()['error'] + + def test_nonexistent_credential_returns_404(self, authenticated_client): + data = json.dumps({'host': '192.168.1.1', 'credential_id': 99999}) + response = authenticated_client.post('/SNMP/CheckDeviceType/', data, content_type='application/json') + assert response.status_code == 404 + assert 'not found' in response.json()['error'].lower() + + def test_invalid_json_body_returns_400(self, authenticated_client): + response = authenticated_client.post( + '/SNMP/CheckDeviceType/', 'not valid json', content_type='application/json' + ) + assert response.status_code == 400 + + def test_unreachable_host_returns_error_payload(self, authenticated_client, test_credential): + with patch('SNMP.views._snmp_get_sys_descr', return_value=None): + data = json.dumps({'host': '10.255.255.255', 'credential_id': test_credential.id}) + response = authenticated_client.post('/SNMP/CheckDeviceType/', data, content_type='application/json') + assert response.status_code == 200 + rdata = response.json() + assert rdata['success'] is False + assert 'error' in rdata + + def test_reachable_host_returns_sys_descr(self, authenticated_client, test_credential): + sys_descr = 'Linux router 5.10.0 #1 SMP x86_64' + with patch('SNMP.views._snmp_get_sys_descr', return_value=sys_descr): + data = json.dumps({'host': '192.168.1.1', 'credential_id': test_credential.id}) + response = authenticated_client.post('/SNMP/CheckDeviceType/', data, content_type='application/json') + assert response.status_code == 200 + rdata = response.json() + assert rdata['success'] is True + assert rdata['sys_descr'] == sys_descr + + def test_matched_template_returned_when_found(self, authenticated_client, test_credential): + from SNMP.models import DeviceTemplate, Profile as _Profile + profile = _Profile.objects.create( + name='linux_match_profile', vendor='Linux', + profile_data={'get': {}, 'walk': {}, 'table': {}} + ) + tpl = DeviceTemplate.objects.create( + name='linux_match', vendor='Linux', matching_rules=['Linux'] + ) + tpl.profiles.add(profile) + + with patch('SNMP.views._snmp_get_sys_descr', return_value='Linux router 5.10 SMP x86_64'), \ + patch('SNMP.snmp_crud.suggest_device_template', return_value=[tpl.id]): + data = json.dumps({'host': '192.168.1.1', 'credential_id': test_credential.id}) + response = authenticated_client.post('/SNMP/CheckDeviceType/', data, content_type='application/json') + assert response.status_code == 200 + rdata = response.json() + assert rdata['success'] is True + assert rdata['matched_template'] is not None + assert rdata['matched_template']['name'] == 'linux_match' + + def test_no_match_returns_null_template(self, authenticated_client, test_credential): + with patch('SNMP.views._snmp_get_sys_descr', return_value='Unknown Device XR-9000'), \ + patch('SNMP.snmp_crud.suggest_device_template', return_value=[]): + data = json.dumps({'host': '192.168.1.1', 'credential_id': test_credential.id}) + response = authenticated_client.post('/SNMP/CheckDeviceType/', data, content_type='application/json') + assert response.status_code == 200 + rdata = response.json() + assert rdata['success'] is True + assert rdata['matched_template'] is None + + def test_readonly_user_is_denied(self, readonly_client, test_credential): + data = json.dumps({'host': '192.168.1.1', 'credential_id': test_credential.id}) + response = readonly_client.post('/SNMP/CheckDeviceType/', data, content_type='application/json') + assert response.status_code == 403 + + def test_custom_port_accepted(self, authenticated_client, test_credential): + with patch('SNMP.views._snmp_get_sys_descr', return_value='Device Description') as mock_fn: + data = json.dumps({'host': '10.0.0.1', 'port': 1161, 'credential_id': test_credential.id}) + response = authenticated_client.post('/SNMP/CheckDeviceType/', data, content_type='application/json') + assert response.status_code == 200 + args = mock_fn.call_args[0] + assert args[1] == 1161 + + +# ============================================================================ +# ImportAIGeneratedDefinitions View Tests +# ============================================================================ + +@pytest.mark.django_db +class TestImportAIGeneratedDefinitions: + """Test the ImportAIGeneratedDefinitions view""" + + def test_get_method_returns_405(self, authenticated_client): + response = authenticated_client.get('/SNMP/ImportAIGeneratedDefinitions/') + assert response.status_code == 405 + + def test_invalid_json_body_returns_400(self, authenticated_client): + response = authenticated_client.post( + '/SNMP/ImportAIGeneratedDefinitions/', 'not json', content_type='application/json' + ) + assert response.status_code == 400 + + def test_profiles_not_list_returns_400(self, authenticated_client): + data = json.dumps({'profiles': 'not a list', 'device_template': {'name': 'tpl', 'profiles': []}}) + response = authenticated_client.post('/SNMP/ImportAIGeneratedDefinitions/', data, content_type='application/json') + assert response.status_code == 400 + assert 'profiles' in response.json()['error'] + + def test_template_not_dict_returns_400(self, authenticated_client): + data = json.dumps({'profiles': [], 'device_template': 'not a dict'}) + response = authenticated_client.post('/SNMP/ImportAIGeneratedDefinitions/', data, content_type='application/json') + assert response.status_code == 400 + + def test_template_missing_name_returns_400(self, authenticated_client): + data = json.dumps({'profiles': [], 'device_template': {'profiles': []}}) + response = authenticated_client.post('/SNMP/ImportAIGeneratedDefinitions/', data, content_type='application/json') + assert response.status_code == 400 + assert 'name' in response.json()['error'] + + def test_template_profiles_not_list_returns_400(self, authenticated_client): + data = json.dumps({'profiles': [], 'device_template': {'name': 'tpl', 'profiles': 'bad'}}) + response = authenticated_client.post('/SNMP/ImportAIGeneratedDefinitions/', data, content_type='application/json') + assert response.status_code == 400 + + def test_profile_missing_name_returns_422(self, authenticated_client): + data = json.dumps({ + 'profiles': [{'get': {}, 'walk': {}}], + 'device_template': {'name': 'My Template', 'profiles': []} + }) + response = authenticated_client.post('/SNMP/ImportAIGeneratedDefinitions/', data, content_type='application/json') + assert response.status_code == 422 + rdata = response.json() + assert rdata['success'] is False + assert len(rdata['errors']) > 0 + + def test_creates_new_profile_and_template(self, authenticated_client): + data = json.dumps({ + 'profiles': [ + { + 'name': 'import_test_profile', + 'vendor': 'Cisco', + 'description': 'Test profile', + 'get': {'cpu.0': '1.3.6.1.4.1.9.9.109.1.1.1.1.7.1'}, + 'walk': {}, + 'table': {} + } + ], + 'device_template': { + 'name': 'import_test_template', + 'vendor': 'Cisco', + 'description': 'Test template', + 'profiles': ['import_test_profile'] + } + }) + response = authenticated_client.post('/SNMP/ImportAIGeneratedDefinitions/', data, content_type='application/json') + assert response.status_code == 200 + rdata = response.json() + assert rdata['success'] is True + assert any(p['action'] == 'created' for p in rdata['profiles']) + assert rdata['template']['action'] == 'created' + assert Profile.objects.filter(name='import_test_profile').exists() + + def test_updates_existing_user_profile(self, authenticated_client, test_profile): + data = json.dumps({ + 'profiles': [ + {'name': 'custom_profile', 'vendor': 'Updated', 'get': {}, 'walk': {}, 'table': {}} + ], + 'device_template': { + 'name': 'update_import_tpl', + 'profiles': ['custom_profile'] + } + }) + response = authenticated_client.post('/SNMP/ImportAIGeneratedDefinitions/', data, content_type='application/json') + assert response.status_code == 200 + rdata = response.json() + assert any(p['action'] == 'updated' for p in rdata['profiles']) + + def test_skips_official_profile(self, authenticated_client): + Profile.objects.create( + name='official_cannot_overwrite', + vendor='Vendor', + official_key='vendor.official_cannot_overwrite', + profile_data={'get': {}, 'walk': {}, 'table': {}} + ) + data = json.dumps({ + 'profiles': [{'name': 'official_cannot_overwrite', 'get': {}, 'walk': {}}], + 'device_template': {'name': 'skip_official_tpl', 'profiles': ['official_cannot_overwrite']} + }) + response = authenticated_client.post('/SNMP/ImportAIGeneratedDefinitions/', data, content_type='application/json') + assert response.status_code == 200 + rdata = response.json() + assert any(p['action'] == 'skipped' for p in rdata['profiles']) + + def test_template_links_to_created_profiles(self, authenticated_client): + data = json.dumps({ + 'profiles': [ + {'name': 'linked_prof_a', 'vendor': 'Generic', 'get': {}, 'walk': {}, 'table': {}}, + {'name': 'linked_prof_b', 'vendor': 'Generic', 'get': {}, 'walk': {}, 'table': {}}, + ], + 'device_template': { + 'name': 'linked_template', + 'profiles': ['linked_prof_a', 'linked_prof_b'] + } + }) + response = authenticated_client.post('/SNMP/ImportAIGeneratedDefinitions/', data, content_type='application/json') + assert response.status_code == 200 + rdata = response.json() + assert rdata['success'] is True + from SNMP.models import DeviceTemplate + tpl = DeviceTemplate.objects.get(name='linked_template') + profile_names = set(tpl.profiles.values_list('name', flat=True)) + assert 'linked_prof_a' in profile_names + assert 'linked_prof_b' in profile_names + + def test_readonly_user_is_denied(self, readonly_client): + data = json.dumps({'profiles': [], 'device_template': {'name': 'tpl', 'profiles': []}}) + response = readonly_client.post('/SNMP/ImportAIGeneratedDefinitions/', data, content_type='application/json') + assert response.status_code == 403 + + def test_empty_profiles_list_creates_only_template(self, authenticated_client): + data = json.dumps({ + 'profiles': [], + 'device_template': {'name': 'empty_profiles_tpl', 'profiles': []} + }) + response = authenticated_client.post('/SNMP/ImportAIGeneratedDefinitions/', data, content_type='application/json') + assert response.status_code == 200 + rdata = response.json() + assert rdata['success'] is True + assert rdata['profiles'] == [] + assert rdata['template']['action'] == 'created' + + +# ============================================================================ +# Overview Page +# ============================================================================ + +@pytest.mark.django_db +class TestOverviewView: + """Test the SNMP Overview page view.""" + + def test_overview_requires_authentication(self, client): + response = client.get('/SNMP/Overview/') + assert response.status_code == 302 + assert '/Management/Login/' in response.url + + def test_overview_accessible_to_admin(self, authenticated_client): + response = authenticated_client.get('/SNMP/Overview/') + assert response.status_code == 200 + + def test_overview_accessible_to_readonly(self, readonly_client): + response = readonly_client.get('/SNMP/Overview/') + assert response.status_code == 200 + + +# ============================================================================ +# GetOverviewMetrics API +# ============================================================================ + +@pytest.mark.django_db +class TestGetOverviewMetricsView: + """Test the /SNMP/GetOverviewMetrics/ JSON endpoint.""" + + def test_get_overview_metrics_requires_auth(self, client): + response = client.get('/SNMP/GetOverviewMetrics/') + assert response.status_code == 302 + + def test_get_overview_metrics_success(self, authenticated_client): + """GetOverviewMetrics returns the expected JSON shape when ES helpers succeed.""" + with patch('SNMP.views.get_discovered_devices_count', return_value={'count': 5, 'errors': []}), \ + patch('SNMP.views.get_template_data_categories', return_value={'templates': [], 'errors': []}), \ + patch('SNMP.views.get_high_resource_usage', return_value={'high_cpu': [], 'high_memory': [], 'errors': []}): + response = authenticated_client.get('/SNMP/GetOverviewMetrics/') + + assert response.status_code == 200 + data = json.loads(response.content) + assert data['success'] is True + assert 'metrics' in data + assert data['metrics']['discovered_devices'] == 5 + assert 'high_usage' in data + assert 'data_quality' in data + + def test_get_overview_metrics_total_devices(self, authenticated_client, test_device): + """total_devices counts devices in the database.""" + with patch('SNMP.views.get_discovered_devices_count', return_value={'count': 0, 'errors': []}), \ + patch('SNMP.views.get_template_data_categories', return_value={'templates': [], 'errors': []}), \ + patch('SNMP.views.get_high_resource_usage', return_value={'high_cpu': [], 'high_memory': [], 'errors': []}): + response = authenticated_client.get('/SNMP/GetOverviewMetrics/') + + data = json.loads(response.content) + assert data['metrics']['total_devices'] >= 1 + + def test_get_overview_metrics_propagates_errors(self, authenticated_client): + """Errors from helpers are propagated to the response errors list.""" + with patch('SNMP.views.get_discovered_devices_count', return_value={'count': 0, 'errors': ['ES connection failed']}), \ + patch('SNMP.views.get_template_data_categories', return_value={'templates': [], 'errors': []}), \ + patch('SNMP.views.get_high_resource_usage', return_value={'high_cpu': [], 'high_memory': [], 'errors': []}): + response = authenticated_client.get('/SNMP/GetOverviewMetrics/') + + data = json.loads(response.content) + assert data['success'] is True + assert data['errors'] is not None + assert 'ES connection failed' in data['errors'] + + def test_get_overview_metrics_exception_returns_500(self, authenticated_client): + """An unexpected exception inside GetOverviewMetrics returns HTTP 500.""" + with patch('SNMP.views.get_discovered_devices_count', side_effect=Exception('Boom')): + response = authenticated_client.get('/SNMP/GetOverviewMetrics/') + assert response.status_code == 500 + data = json.loads(response.content) + assert data['success'] is False + + +# ============================================================================ +# CheckSNMPIndexTemplate +# ============================================================================ + +@pytest.mark.django_db +class TestCheckSNMPIndexTemplateView: + """Test the /SNMP/CheckSNMPIndexTemplate/ endpoint.""" + + def test_requires_post(self, authenticated_client): + response = authenticated_client.get('/SNMP/CheckSNMPIndexTemplate/') + assert response.status_code == 405 + + def test_requires_connection_ids(self, authenticated_client): + response = authenticated_client.post( + '/SNMP/CheckSNMPIndexTemplate/', + data=json.dumps({}), + content_type='application/json', + ) + assert response.status_code == 400 + + def test_invalid_json_body(self, authenticated_client): + response = authenticated_client.post( + '/SNMP/CheckSNMPIndexTemplate/', + data='not json', + content_type='application/json', + ) + assert response.status_code == 400 + + def test_connection_not_found(self, authenticated_client): + """A non-existent connection_id returns an error result (not a 500).""" + with patch('SNMP.views._load_snmp_template', return_value={'_meta': {'template_name': 'metrics-snmp.polling'}}), \ + patch('Common.elastic_utils.check_index_template', return_value={'status': 'installed', 'differences': []}): + response = authenticated_client.post( + '/SNMP/CheckSNMPIndexTemplate/', + data=json.dumps({'connection_ids': [99999]}), + content_type='application/json', + ) + assert response.status_code == 200 + data = json.loads(response.content) + result = data['results'][0] + assert result['status'] == 'error' + assert 'not found' in result['error'] + + def test_installed_status(self, authenticated_client, test_connection): + """Returns 'installed' status when template is present and up to date.""" + with patch('SNMP.views._load_snmp_template', return_value={'_meta': {'template_name': 'metrics-snmp.polling'}}), \ + patch('Common.elastic_utils.check_index_template', return_value={'status': 'installed', 'differences': []}): + response = authenticated_client.post( + '/SNMP/CheckSNMPIndexTemplate/', + data=json.dumps({'connection_ids': [test_connection.id]}), + content_type='application/json', + ) + assert response.status_code == 200 + data = json.loads(response.content) + assert data['results'][0]['status'] == 'installed' + assert data['results'][0]['connection_name'] == test_connection.name + + +# ============================================================================ +# InstallSNMPIndexTemplate +# ============================================================================ + +@pytest.mark.django_db +class TestInstallSNMPIndexTemplateView: + """Test the /SNMP/InstallSNMPIndexTemplate/ endpoint.""" + + def test_requires_admin(self, readonly_client, test_connection): + response = readonly_client.post( + '/SNMP/InstallSNMPIndexTemplate/', + data=json.dumps({'connection_ids': [test_connection.id]}), + content_type='application/json', + ) + assert response.status_code == 403 + + def test_requires_post(self, authenticated_client): + response = authenticated_client.get('/SNMP/InstallSNMPIndexTemplate/') + assert response.status_code == 405 + + def test_requires_connection_ids(self, authenticated_client): + response = authenticated_client.post( + '/SNMP/InstallSNMPIndexTemplate/', + data=json.dumps({}), + content_type='application/json', + ) + assert response.status_code == 400 + + def test_success(self, authenticated_client, test_connection): + """Successfully installing a template returns success=True.""" + with patch('SNMP.views._load_snmp_template', return_value={'_meta': {'template_name': 'metrics-snmp.polling'}}), \ + patch('Common.elastic_utils.create_index_template', return_value=None): + response = authenticated_client.post( + '/SNMP/InstallSNMPIndexTemplate/', + data=json.dumps({'connection_ids': [test_connection.id]}), + content_type='application/json', + ) + assert response.status_code == 200 + data = json.loads(response.content) + assert data['success'] is True + assert data['results'][0]['success'] is True + + def test_connection_not_found(self, authenticated_client): + """A non-existent connection_id records failure without crashing.""" + with patch('SNMP.views._load_snmp_template', return_value={'_meta': {'template_name': 'metrics-snmp.polling'}}): + response = authenticated_client.post( + '/SNMP/InstallSNMPIndexTemplate/', + data=json.dumps({'connection_ids': [99999]}), + content_type='application/json', + ) + assert response.status_code == 200 + data = json.loads(response.content) + assert data['success'] is False + assert data['results'][0]['success'] is False + + +# ============================================================================ +# CheckAgentBuilderResources +# ============================================================================ + +@pytest.mark.django_db +class TestCheckAgentBuilderResourcesView: + """Test the /SNMP/CheckAgentBuilderResources/ endpoint.""" + + def test_requires_post(self, authenticated_client): + response = authenticated_client.get('/SNMP/CheckAgentBuilderResources/') + assert response.status_code == 405 + + def test_requires_connection_id(self, authenticated_client): + response = authenticated_client.post( + '/SNMP/CheckAgentBuilderResources/', + data=json.dumps({}), + content_type='application/json', + ) + assert response.status_code == 400 + + def test_invalid_json_body(self, authenticated_client): + response = authenticated_client.post( + '/SNMP/CheckAgentBuilderResources/', + data='{ bad json', + content_type='application/json', + ) + assert response.status_code == 400 + + def test_success(self, authenticated_client): + """Returns result from AgentBuilder.check_resources when successful.""" + mock_result = {'tools': [], 'skills': [], 'agents': []} + with patch('Common.ai.agent_builder.AgentBuilder') as MockBuilder, \ + patch('Common.ai.agent_builder.load_resources_from_directory', return_value=([], [], [])): + MockBuilder.return_value.check_resources.return_value = mock_result + response = authenticated_client.post( + '/SNMP/CheckAgentBuilderResources/', + data=json.dumps({'connection_id': 1}), + content_type='application/json', + ) + assert response.status_code == 200 + + def test_agent_builder_exception_returns_500(self, authenticated_client): + """If AgentBuilder raises, the endpoint returns 500.""" + with patch('Common.ai.agent_builder.AgentBuilder', side_effect=Exception('KB down')), \ + patch('Common.ai.agent_builder.load_resources_from_directory', return_value=([], [], [])): + response = authenticated_client.post( + '/SNMP/CheckAgentBuilderResources/', + data=json.dumps({'connection_id': 1}), + content_type='application/json', + ) + assert response.status_code == 500 + + +# ============================================================================ +# InstallAgentBuilderPackage +# ============================================================================ + +@pytest.mark.django_db +class TestInstallAgentBuilderPackageView: + """Test the /SNMP/InstallAgentBuilderPackage/ endpoint.""" + + def test_requires_admin(self, readonly_client): + response = readonly_client.post( + '/SNMP/InstallAgentBuilderPackage/', + data=json.dumps({'connection_id': 1}), + content_type='application/json', + ) + assert response.status_code == 403 + + def test_requires_post(self, authenticated_client): + response = authenticated_client.get('/SNMP/InstallAgentBuilderPackage/') + assert response.status_code == 405 + + def test_requires_connection_id(self, authenticated_client): + response = authenticated_client.post( + '/SNMP/InstallAgentBuilderPackage/', + data=json.dumps({}), + content_type='application/json', + ) + assert response.status_code == 400 + + def test_success(self, authenticated_client): + """Returns result from AgentBuilder.apply_all_resources when successful.""" + mock_result = {'success': True, 'results': []} + with patch('Common.ai.agent_builder.AgentBuilder') as MockBuilder, \ + patch('Common.ai.agent_builder.load_resources_from_directory', return_value=([], [], [])): + MockBuilder.return_value.apply_all_resources.return_value = mock_result + response = authenticated_client.post( + '/SNMP/InstallAgentBuilderPackage/', + data=json.dumps({'connection_id': 1}), + content_type='application/json', + ) + assert response.status_code == 200 + + def test_exception_returns_500(self, authenticated_client): + """Unexpected exception returns 500 with success=False.""" + with patch('Common.ai.agent_builder.AgentBuilder', side_effect=Exception('Fail')), \ + patch('Common.ai.agent_builder.load_resources_from_directory', return_value=([], [], [])): + response = authenticated_client.post( + '/SNMP/InstallAgentBuilderPackage/', + data=json.dumps({'connection_id': 1}), + content_type='application/json', + ) + assert response.status_code == 500 + data = json.loads(response.content) + assert data['success'] is False + + +class TestGenerateTemplateGroundingInline: + """GenerateTemplateAndProfiles reduces the walk to MIB-grounded columns and passes + them INLINE to the agent. It must NOT stage the walk in a backend ES index — the + record of truth stays local to LogstashUI, which may connect to multiple backends, + so per-backend staging (residue + backend-dependent output) is disallowed.""" + + # Mixed walk: SNMPv2 + IF-MIB columns (grounded) plus an enterprise OID (ungrounded). + WALK = "\n".join([ + "1.3.6.1.2.1.1.1.0 = Cisco IOS Software, C2960X Software", + "1.3.6.1.2.1.1.3.0 = 44266130", + "1.3.6.1.2.1.1.5.0 = homelab-switch1", + "1.3.6.1.2.1.2.2.1.10.1 = 12345", # ifInOctets col (IF-MIB) -> grounded, instances=2 + "1.3.6.1.2.1.2.2.1.10.2 = 67890", + "1.3.6.1.4.1.9.9.999.1.0 = 1", # enterprise -> no compiled MIB -> ungrounded + ]) + + def _post(self, client, connection_id): + resp = client.post( + '/SNMP/GenerateTemplateAndProfiles/', + data=json.dumps({ + 'connection_id': connection_id, + 'walk_text': self.WALK, + 'inference_id': '.rainbow-sprinkles-elastic', + }), + content_type='application/json', + ) + # Drain the SSE stream. + return b''.join(resp.streaming_content).decode() + + @patch('Common.elastic_utils.bulk_index_documents') + @patch('Common.ai.agent_builder.AgentBuilder') + def test_grounded_columns_inline_no_backend_write( + self, MockAgentBuilder, mock_bulk, authenticated_client, test_connection + ): + instance = MockAgentBuilder.return_value + instance._kibana_url = 'https://kb.example' + captured = {} + + def _invoke(agent_id, message, **kwargs): + captured['agent_id'] = agent_id + captured['message'] = message + return iter(()) # empty agent stream is fine for this assertion + + instance.invoke_agent.side_effect = _invoke + + body = self._post(authenticated_client, test_connection.id) + + # 1. Nothing is written to any backend — no bulk index, no temp index name anywhere. + mock_bulk.assert_not_called() + assert 'snmp-template_generation' not in body + assert 'snmp-template_generation' not in captured['message'] + + # 2. The agent received the grounded columns INLINE (not an index to query). + assert 'grounded_columns' in captured['message'] + payload = json.loads(captured['message'][captured['message'].index('{'):]) + names = {c['name'] for c in payload['grounded_columns']} + assert 'sysDescr' in names # SNMPv2-MIB scalar grounded + assert 'ifInOctets' in names # IF-MIB table column grounded (multi-instance) + assert next(c for c in payload['grounded_columns'] if c['name'] == 'ifInOctets')['instances'] == 2 + # The enterprise OID had no compiled MIB -> reported for MIB-loading, not authored. + assert any(u['prefix'].startswith('1.3.6.1.4.1.9') for u in payload['ungrounded_subtrees']) + + # 3. SSE reports the grounding phase and never the old indexing phase. + assert '"phase": "grounding"' in body + assert 'indexing' not in body + + @patch('Common.elastic_utils.bulk_index_documents') + @patch('Common.ai.agent_builder.AgentBuilder') + def test_empty_grounding_errors_without_backend_write( + self, MockAgentBuilder, mock_bulk, authenticated_client, test_connection + ): + # A walk with only un-grounded enterprise OIDs -> no columns to author. + resp = authenticated_client.post( + '/SNMP/GenerateTemplateAndProfiles/', + data=json.dumps({ + 'connection_id': test_connection.id, + 'walk_text': '1.3.6.1.4.1.9999.1.2.3.0 = 5', + 'inference_id': '.rainbow-sprinkles-elastic', + }), + content_type='application/json', + ) + body = b''.join(resp.streaming_content).decode() + + assert '"phase": "error"' in body + mock_bulk.assert_not_called() + MockAgentBuilder.return_value.invoke_agent.assert_not_called() diff --git a/tests/Site/__init__.py b/tests/Site/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/Site/unit/__init__.py b/tests/Site/unit/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/Site/unit/test_views.py b/tests/Site/unit/test_views.py new file mode 100644 index 0000000..e5670b7 --- /dev/null +++ b/tests/Site/unit/test_views.py @@ -0,0 +1,381 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +import pytest +from django.urls import reverse +from unittest.mock import patch +from packaging import version +from Site import views + + +@pytest.mark.django_db +def test_health_check_returns_200(client): + url = reverse('health_check') + response = client.get(url) + assert response.status_code == 200 + assert response.json() == {'status': 'healthy', 'service': 'logstashui'} + + +@pytest.mark.django_db +def test_home_view_returns_200(client, django_user_model): + user = django_user_model.objects.create_user(username='testuser', password='testpass123') + client.force_login(user) + url = reverse('home') + response = client.get(url) + assert response.status_code == 200 + + +def test_parse_version_tag(): + assert views.parse_version_tag('v1.0.0') == version.parse('1.0.0') + assert views.parse_version_tag('2.1.3') == version.parse('2.1.3') + assert views.parse_version_tag('invalid_tag') is None + + +@patch('Site.views.requests.get') +def test_fetch_latest_version_from_docker_hub(mock_get): + class MockResponse: + def json(self): + return { + 'results': [ + {'name': 'v2.0.0'}, + {'name': '1.5.0'}, + {'name': 'latest'} + ] + } + + def raise_for_status(self): + pass + + mock_get.return_value = MockResponse() + + result = views.fetch_latest_version_from_docker_hub() + assert result == '2.0.0' + + +@patch('Site.views.get_latest_version') +@patch('Site.views.settings') +def test_check_for_update_newer_available(mock_settings, mock_get_latest): + mock_settings.__VERSION__ = '1.0.0' + mock_get_latest.return_value = '2.0.0' + + update_info = views.check_for_update() + + assert update_info is not None + assert update_info['update_available'] is True + assert update_info['latest_version'] == '2.0.0' + + +# ============================================================================ +# parse_version_tag — additional edge cases +# ============================================================================ + +def test_parse_version_tag_prerelease_is_parsed(): + """Pre-release tags parse to a version object (but is_prerelease=True)""" + result = views.parse_version_tag('v1.0.0a1') + assert result is not None + assert result.is_prerelease + + +def test_parse_version_tag_strips_whitespace(): + """Leading/trailing whitespace is stripped before parsing""" + result = views.parse_version_tag(' 1.2.3 ') + assert result is not None + from packaging import version as pkg_ver + assert result == pkg_ver.parse('1.2.3') + + +def test_parse_version_tag_empty_string_returns_none(): + """An empty string should return None, not raise""" + result = views.parse_version_tag('') + # An empty string isn't a valid semver — expect None + # (packaging may return a LegacyVersion or raise; either way we handle it) + # The function is supposed to return None on bad input via the except clause + # but packaging may allow it — just assert no exception is raised + # (returns None or the parsed value — both are acceptable) + assert result is None or result is not None # no exception + + +def test_parse_version_tag_v_prefix_stripped(): + """v-prefixed tags are parsed the same as the bare version""" + from packaging import version as pkg_ver + assert views.parse_version_tag('v3.0.1') == pkg_ver.parse('3.0.1') + + +# ============================================================================ +# fetch_latest_version_from_docker_hub — error and edge-case paths +# ============================================================================ + +@patch('Site.views.requests.get') +def test_fetch_latest_version_empty_results(mock_get): + """When results list is empty, returns None""" + mock_get.return_value.raise_for_status.return_value = None + mock_get.return_value.json.return_value = {'results': []} + + result = views.fetch_latest_version_from_docker_hub() + assert result is None + + +@patch('Site.views.requests.get') +def test_fetch_latest_version_only_non_semver_tags(mock_get): + """When no results have valid semver names, returns None""" + mock_get.return_value.raise_for_status.return_value = None + mock_get.return_value.json.return_value = { + 'results': [{'name': 'latest'}, {'name': 'edge'}, {'name': 'nightly'}] + } + + result = views.fetch_latest_version_from_docker_hub() + assert result is None + + +@patch('Site.views.requests.get') +def test_fetch_latest_version_only_prerelease_tags(mock_get): + """When all valid semver tags are pre-releases, returns None""" + mock_get.return_value.raise_for_status.return_value = None + mock_get.return_value.json.return_value = { + 'results': [{'name': 'v1.0.0a1'}, {'name': '2.0.0b3'}] + } + + result = views.fetch_latest_version_from_docker_hub() + assert result is None + + +@patch('Site.views.requests.get') +def test_fetch_latest_version_picks_highest(mock_get): + """Sorting picks the highest version, not just the first returned""" + mock_get.return_value.raise_for_status.return_value = None + mock_get.return_value.json.return_value = { + 'results': [ + {'name': '1.0.0'}, + {'name': '3.0.0'}, + {'name': '2.0.0'}, + ] + } + + result = views.fetch_latest_version_from_docker_hub() + assert result == '3.0.0' + + +@patch('Site.views.requests.get') +def test_fetch_latest_version_strips_v_prefix_from_result(mock_get): + """The returned version string has the leading 'v' stripped""" + mock_get.return_value.raise_for_status.return_value = None + mock_get.return_value.json.return_value = { + 'results': [{'name': 'v4.1.0'}] + } + + result = views.fetch_latest_version_from_docker_hub() + assert result == '4.1.0' + assert not result.startswith('v') + + +@patch('Site.views.requests.get', side_effect=__import__('requests').exceptions.Timeout) +def test_fetch_latest_version_timeout_returns_none(mock_get): + """A Timeout exception returns None gracefully""" + result = views.fetch_latest_version_from_docker_hub() + assert result is None + + +@patch('Site.views.requests.get', + side_effect=__import__('requests').exceptions.ConnectionError("refused")) +def test_fetch_latest_version_request_exception_returns_none(mock_get): + """A generic RequestException returns None gracefully""" + result = views.fetch_latest_version_from_docker_hub() + assert result is None + + +@patch('Site.views.requests.get', side_effect=ValueError("bad JSON")) +def test_fetch_latest_version_generic_exception_returns_none(mock_get): + """Any unexpected exception returns None gracefully""" + result = views.fetch_latest_version_from_docker_hub() + assert result is None + + +# ============================================================================ +# update_latest_version_cache +# ============================================================================ + +@patch('Site.views.cache') +@patch('Site.views.fetch_latest_version_from_docker_hub', return_value='1.2.3') +def test_update_latest_version_cache_sets_cache_on_success(mock_fetch, mock_cache): + """When fetch succeeds, the result is stored in the cache""" + views.update_latest_version_cache() + + mock_cache.set.assert_called_once_with(views.CACHE_KEY, '1.2.3', views.CACHE_TIMEOUT) + + +@patch('Site.views.cache') +@patch('Site.views.fetch_latest_version_from_docker_hub', return_value=None) +def test_update_latest_version_cache_does_not_set_on_failure(mock_fetch, mock_cache): + """When fetch returns None, cache.set is NOT called""" + views.update_latest_version_cache() + + mock_cache.set.assert_not_called() + + +@patch('Site.views.cache') +@patch('Site.views.fetch_latest_version_from_docker_hub', return_value='5.0.0') +def test_update_latest_version_cache_always_releases_lock(mock_fetch, mock_cache): + """Lock is always released via cache.delete in the finally block""" + views.update_latest_version_cache() + + mock_cache.delete.assert_called_once_with(views.CACHE_LOCK_KEY) + + +@patch('Site.views.cache') +@patch('Site.views.fetch_latest_version_from_docker_hub', side_effect=RuntimeError("explode")) +def test_update_latest_version_cache_releases_lock_on_exception(mock_fetch, mock_cache): + """Lock is released even when fetch_latest_version_from_docker_hub raises""" + # fetch raising inside update_latest_version_cache — that exception would propagate + # unless caught. The function doesn't catch it, so the finally still runs. + try: + views.update_latest_version_cache() + except RuntimeError: + pass # expected — the function doesn't swallow fetch exceptions + mock_cache.delete.assert_called_once_with(views.CACHE_LOCK_KEY) + + +# ============================================================================ +# get_latest_version — cache hit/miss and locking +# ============================================================================ + +@patch('Site.views.cache') +def test_get_latest_version_cache_hit_returns_immediately(mock_cache): + """On cache hit, the cached value is returned and no thread is spawned""" + mock_cache.get.return_value = '9.9.9' + + result = views.get_latest_version() + + assert result == '9.9.9' + # cache.add should NOT be called (no lock acquisition needed) + mock_cache.add.assert_not_called() + + +@patch('Site.views.threading.Thread') +@patch('Site.views.cache') +def test_get_latest_version_cache_miss_lock_acquired_spawns_thread(mock_cache, mock_thread): + """On cache miss, when lock is acquired, a background thread is started""" + mock_cache.get.return_value = None # cache miss + mock_cache.add.return_value = True # lock acquired + + mock_thread_instance = mock_thread.return_value + + result = views.get_latest_version() + + assert result is None # returns None synchronously while thread runs + mock_thread.assert_called_once() + mock_thread_instance.start.assert_called_once() + + +@patch('Site.views.threading.Thread') +@patch('Site.views.cache') +def test_get_latest_version_cache_miss_lock_not_acquired_no_thread(mock_cache, mock_thread): + """On cache miss, when lock is already held, no thread is spawned""" + mock_cache.get.return_value = None # cache miss + mock_cache.add.return_value = False # lock already held + + result = views.get_latest_version() + + assert result is None + mock_thread.assert_not_called() + + +# ============================================================================ +# check_for_update — additional edge cases +# ============================================================================ + +@patch('Site.views.settings') +def test_check_for_update_no_version_setting_returns_none(mock_settings): + """When __VERSION__ is not in settings, returns None""" + del mock_settings.__VERSION__ # simulate missing attribute + + result = views.check_for_update() + assert result is None + + +@patch('Site.views.get_latest_version', return_value=None) +@patch('Site.views.settings') +def test_check_for_update_no_latest_returns_none(mock_settings, mock_glv): + """When latest version is not cached yet, returns None""" + mock_settings.__VERSION__ = '1.0.0' + + result = views.check_for_update() + assert result is None + + +@patch('Site.views.get_latest_version', return_value='1.0.0') +@patch('Site.views.settings') +def test_check_for_update_same_version_returns_none(mock_settings, mock_glv): + """When running latest version already, returns None (no update)""" + mock_settings.__VERSION__ = '1.0.0' + + result = views.check_for_update() + assert result is None + + +@patch('Site.views.get_latest_version', return_value='0.9.0') +@patch('Site.views.settings') +def test_check_for_update_running_newer_returns_none(mock_settings, mock_glv): + """When current is newer than latest (dev build), returns None""" + mock_settings.__VERSION__ = '2.0.0' + + result = views.check_for_update() + assert result is None + + +@patch('Site.views.get_latest_version', return_value='not-a-version') +@patch('Site.views.settings') +def test_check_for_update_parse_error_returns_none(mock_settings, mock_glv): + """When version parsing raises, returns None gracefully""" + mock_settings.__VERSION__ = 'also-not-a-version' + + # packaging will raise on truly invalid strings + result = views.check_for_update() + # Either None (graceful error handling) or a valid dict — no exception + assert result is None or isinstance(result, dict) + + +@patch('Site.views.get_latest_version', return_value='5.0.0') +@patch('Site.views.settings') +def test_check_for_update_result_contains_expected_keys(mock_settings, mock_glv): + """The returned dict has all expected keys""" + mock_settings.__VERSION__ = '1.0.0' + + result = views.check_for_update() + + assert result is not None + assert set(result.keys()) == {'current_version', 'latest_version', 'update_available', 'release_url'} + assert result['current_version'] == '1.0.0' + assert result['latest_version'] == '5.0.0' + assert result['update_available'] is True + assert 'v5.0.0' in result['release_url'] + + +# ============================================================================ +# View authentication behaviour +# ============================================================================ + +@pytest.mark.django_db +def test_health_check_unauthenticated_returns_200(client): + """health_check has no login requirement — anonymous requests return 200""" + response = client.get(reverse('health_check')) + assert response.status_code == 200 + + +@pytest.mark.django_db +def test_home_unauthenticated_redirects(client): + """Home requires login — unauthenticated requests are redirected""" + response = client.get(reverse('home')) + # Should redirect to login, not return 200 + assert response.status_code == 302 + assert '/Management/Login/' in response.url + + +@pytest.mark.django_db +def test_home_uses_home_template(client, django_user_model): + """Home view renders the home.html template""" + user = django_user_model.objects.create_user(username='tmpluser', password='pass123') + client.force_login(user) + response = client.get(reverse('home')) + assert response.status_code == 200 + assert any('home.html' in t.name for t in response.templates) diff --git a/tests/Utilities/__init__.py b/tests/Utilities/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/Utilities/data b/tests/Utilities/data new file mode 120000 index 0000000..48c5ad3 --- /dev/null +++ b/tests/Utilities/data @@ -0,0 +1 @@ +../../src/logstashui/Utilities/data \ No newline at end of file diff --git a/tests/Utilities/unit/__init__.py b/tests/Utilities/unit/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/Utilities/unit/test_grok_patterns.py b/tests/Utilities/unit/test_grok_patterns.py new file mode 100644 index 0000000..83153f4 --- /dev/null +++ b/tests/Utilities/unit/test_grok_patterns.py @@ -0,0 +1,138 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +import pytest +import os +import re +import logging +from django.conf import settings + +logging.basicConfig(level=logging.INFO) +logger = logging.getLogger(__name__) + + +@pytest.fixture +def grok_patterns_file_path(): + """Path to the grok patterns file""" + utilities_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) + patterns_file = os.path.join(utilities_dir, 'data', 'grok-patterns.txt') + + if not os.path.exists(patterns_file): + patterns_file = os.path.join(utilities_dir, 'grok-patterns') + + if not os.path.exists(patterns_file): + patterns_file = os.path.join(utilities_dir, 'static', 'grok-patterns') + + return patterns_file + + +@pytest.mark.django_db +class TestGrokPatternsFile: + """Tests for the grok-patterns.txt file""" + + def test_grok_patterns_file_exists(self, grok_patterns_file_path): + """Verify grok-patterns.txt file exists""" + assert os.path.exists(grok_patterns_file_path), \ + f"Grok patterns file should exist at {grok_patterns_file_path}" + + def test_grok_patterns_file_readable(self, grok_patterns_file_path): + """Verify file is readable""" + try: + with open(grok_patterns_file_path, 'r', encoding='utf-8') as f: + content = f.read() + assert len(content) > 0, "Grok patterns file should not be empty" + except Exception as e: + pytest.fail(f"Failed to read grok patterns file: {e}") + + def test_grok_patterns_file_format(self, grok_patterns_file_path): + """Verify all patterns follow correct format""" + with open(grok_patterns_file_path, 'r', encoding='utf-8') as f: + for line_num, line in enumerate(f, 1): + line = line.strip() + + # Skip empty lines and comments + if not line or line.startswith('#'): + continue + + # Pattern should be: NAME definition + if ' ' not in line: + pytest.fail( + f"Line {line_num} is malformed (no space separator): {line}" + ) + + parts = line.split(None, 1) + if len(parts) != 2: + pytest.fail( + f"Line {line_num} is malformed (expected 2 parts): {line}" + ) + + pattern_name, pattern_def = parts + + # Pattern name should be uppercase alphanumeric with underscores + if not re.match(r'^[A-Z0-9_]+$', pattern_name): + pytest.fail( + f"Line {line_num} has invalid pattern name '{pattern_name}': " + f"should be uppercase alphanumeric with underscores" + ) + + def test_grok_patterns_no_duplicates(self, grok_patterns_file_path): + """Verify no duplicate pattern names""" + pattern_names = [] + + with open(grok_patterns_file_path, 'r', encoding='utf-8') as f: + for line in f: + line = line.strip() + if not line or line.startswith('#'): + continue + + if ' ' in line: + pattern_name = line.split(None, 1)[0] + pattern_names.append(pattern_name) + + duplicates = [name for name in pattern_names if pattern_names.count(name) > 1] + duplicates = list(set(duplicates)) + + assert len(duplicates) == 0, \ + f"Found duplicate pattern names: {duplicates}" + + def test_grok_patterns_contains_essential_patterns(self, grok_patterns_file_path): + """Verify essential patterns are present""" + essential_patterns = [ + 'USERNAME', 'USER', 'INT', 'NUMBER', 'WORD', + 'NOTSPACE', 'SPACE', 'DATA', 'GREEDYDATA', 'IP', 'IPV4' + ] + + with open(grok_patterns_file_path, 'r', encoding='utf-8') as f: + content = f.read() + + missing_patterns = [] + for pattern in essential_patterns: + # Look for pattern at start of line (with word boundary) + if not re.search(rf'^{pattern}\s', content, re.MULTILINE): + missing_patterns.append(pattern) + + assert len(missing_patterns) == 0, \ + f"Missing essential patterns: {missing_patterns}" + + def test_grok_patterns_encoding(self, grok_patterns_file_path): + """Verify file uses UTF-8 encoding""" + try: + with open(grok_patterns_file_path, 'r', encoding='utf-8') as f: + f.read() + except UnicodeDecodeError: + pytest.fail("Grok patterns file should be UTF-8 encoded") + + def test_grok_patterns_no_trailing_whitespace(self, grok_patterns_file_path): + """Verify no lines have trailing whitespace (code quality check)""" + lines_with_trailing_ws = [] + + with open(grok_patterns_file_path, 'r', encoding='utf-8') as f: + for line_num, line in enumerate(f, 1): + # Check for trailing whitespace (but not newlines) + if line.rstrip('\r\n') != line.rstrip(): + lines_with_trailing_ws.append(line_num) + + # This is a soft check - we'll warn but not fail + if lines_with_trailing_ws: + logger.warning(f"Lines with trailing whitespace: {lines_with_trailing_ws[:10]}") diff --git a/tests/Utilities/unit/test_views.py b/tests/Utilities/unit/test_views.py new file mode 100644 index 0000000..471b0e8 --- /dev/null +++ b/tests/Utilities/unit/test_views.py @@ -0,0 +1,671 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +import pytest +from django.test import RequestFactory, Client +from django.http import JsonResponse, HttpResponse +from Utilities.views import ( + GrokDebugger, + get_grok_patterns, + simulate_grok, + generate_results_html +) +import json +import os +from django.conf import settings + + +@pytest.fixture +def sample_log_data(): + """Sample log data for testing grok patterns""" + return { + 'simple': '192.168.1.1 - - [01/Jan/2024:12:00:00 +0000] "GET /index.html HTTP/1.1" 200 1234', + 'multiline': 'Line 1\nLine 2\nLine 3', + 'special_chars': '', + 'unicode': 'User José logged in from München' + } + + +@pytest.fixture +def custom_patterns(): + """Custom grok pattern definitions for testing""" + return r"""CUSTOM_EMAIL [A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,} +CUSTOM_DATE \d{4}-\d{2}-\d{2}""" + + +@pytest.fixture +def grok_patterns_file_path(): + """Path to the grok patterns file""" + utilities_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) + patterns_file = os.path.join(utilities_dir, 'data', 'grok-patterns.txt') + + if not os.path.exists(patterns_file): + patterns_file = os.path.join(utilities_dir, 'grok-patterns') + + if not os.path.exists(patterns_file): + patterns_file = os.path.join(utilities_dir, 'static', 'grok-patterns') + + return patterns_file + + +@pytest.mark.django_db +class TestGrokDebuggerView: + """Tests for the main Grok Debugger view""" + + def test_grok_debugger_renders_template(self, request_factory): + """Test that GrokDebugger view renders the correct template""" + request = request_factory.get('/Utilities/GrokDebugger/') + response = GrokDebugger(request) + + assert response.status_code == 200 + + def test_grok_debugger_get_request(self, authenticated_client): + """Test GET request to Grok Debugger""" + response = authenticated_client.get('/Utilities/GrokDebugger/') + assert response.status_code == 200 + + +@pytest.mark.django_db +class TestGetGrokPatternsView: + """Tests for get_grok_patterns view""" + + def test_get_grok_patterns_success(self, request_factory, grok_patterns_file_path): + """Test successful loading of grok patterns""" + request = request_factory.get('/Utilities/GrokDebugger/patterns/') + response = get_grok_patterns(request) + + assert response.status_code == 200 + assert isinstance(response, JsonResponse) + + data = json.loads(response.content) + assert 'patterns' in data + assert isinstance(data['patterns'], dict) + assert len(data['patterns']) > 0 + + def test_get_grok_patterns_contains_common_patterns(self, request_factory): + """Test that common patterns are present""" + request = request_factory.get('/Utilities/GrokDebugger/patterns/') + response = get_grok_patterns(request) + + data = json.loads(response.content) + patterns = data['patterns'] + + # Check for some common patterns + common_patterns = ['USERNAME', 'IP', 'WORD', 'NUMBER', 'DATA'] + for pattern in common_patterns: + assert pattern in patterns, f"Pattern {pattern} should be in grok patterns" + + def test_get_grok_patterns_file_exists(self, grok_patterns_file_path): + """Test that the grok patterns file exists""" + assert os.path.exists(grok_patterns_file_path), "Grok patterns file should exist" + + +@pytest.mark.django_db +class TestSimulateGrokView: + """Tests for simulate_grok view""" + + def test_simulate_grok_single_line_match(self, request_factory, sample_log_data): + """Test successful pattern matching on single line""" + request = request_factory.post('/Utilities/GrokDebugger/simulate/', { + 'sample_data': sample_log_data['simple'], + 'grok_pattern': '%{IP:client_ip}', + 'custom_patterns': '', + 'multiline_mode': 'false' + }) + + response = simulate_grok(request) + assert response.status_code == 200 + assert isinstance(response, HttpResponse) + + content = response.content.decode('utf-8') + assert '192.168.1.1' in content + assert 'Match Found' in content or 'success' in content.lower() + + def test_simulate_grok_single_line_no_match(self, request_factory): + """Test pattern that doesn't match input""" + request = request_factory.post('/Utilities/GrokDebugger/simulate/', { + 'sample_data': 'This is plain text', + 'grok_pattern': '%{IP:client_ip}', + 'custom_patterns': '', + 'multiline_mode': 'false' + }) + + response = simulate_grok(request) + assert response.status_code == 200 + + content = response.content.decode('utf-8') + assert 'No Match' in content or 'did not match' in content.lower() + + def test_simulate_grok_multiline_mode(self, request_factory, sample_log_data): + """Test multiline mode treats entire input as single string""" + request = request_factory.post('/Utilities/GrokDebugger/simulate/', { + 'sample_data': sample_log_data['multiline'], + 'grok_pattern': '%{GREEDYDATA:message}', + 'custom_patterns': '', + 'multiline_mode': 'true' + }) + + response = simulate_grok(request) + assert response.status_code == 200 + + content = response.content.decode('utf-8') + # In multiline mode, should treat as one input + assert 'Line 1' in content + + def test_simulate_grok_multiple_patterns(self, request_factory): + """Test multiple patterns against single input""" + request = request_factory.post('/Utilities/GrokDebugger/simulate/', { + 'sample_data': '192.168.1.1', + 'grok_pattern': '%{IP:ip}\n%{WORD:word}', + 'custom_patterns': '', + 'multiline_mode': 'false' + }) + + response = simulate_grok(request) + assert response.status_code == 200 + + content = response.content.decode('utf-8') + # Should show results for Pattern 1 and Pattern 2 + assert 'Pattern 1' in content + assert 'Pattern 2' in content + + def test_simulate_grok_custom_patterns(self, request_factory, custom_patterns): + """Test with custom pattern definitions""" + request = request_factory.post('/Utilities/GrokDebugger/simulate/', { + 'sample_data': 'test@example.com', + 'grok_pattern': '%{CUSTOM_EMAIL:email}', + 'custom_patterns': custom_patterns, + 'multiline_mode': 'false' + }) + + response = simulate_grok(request) + assert response.status_code == 200 + + content = response.content.decode('utf-8') + assert 'test@example.com' in content + + def test_simulate_grok_dot_notation_fields(self, request_factory): + """Test field names with dots create nested dictionaries""" + request = request_factory.post('/Utilities/GrokDebugger/simulate/', { + 'sample_data': '192.168.1.1', + 'grok_pattern': '%{IP:client.ip.address}', + 'custom_patterns': '', + 'multiline_mode': 'false' + }) + + response = simulate_grok(request) + assert response.status_code == 200 + + content = response.content.decode('utf-8') + # Should show nested structure + assert 'client' in content + assert '192.168.1.1' in content + + def test_simulate_grok_pattern_compilation_error(self, request_factory): + """Test invalid grok pattern syntax""" + request = request_factory.post('/Utilities/GrokDebugger/simulate/', { + 'sample_data': 'test data', + 'grok_pattern': '%{INVALID_PATTERN_THAT_DOES_NOT_EXIST:field}', + 'custom_patterns': '', + 'multiline_mode': 'false' + }) + + response = simulate_grok(request) + assert response.status_code == 200 + + content = response.content.decode('utf-8') + assert 'error' in content.lower() or 'compilation' in content.lower() + + def test_simulate_grok_empty_sample_data(self, request_factory): + """Test with empty sample data""" + request = request_factory.post('/Utilities/GrokDebugger/simulate/', { + 'sample_data': '', + 'grok_pattern': '%{IP:ip}', + 'custom_patterns': '', + 'multiline_mode': 'false' + }) + + response = simulate_grok(request) + assert response.status_code == 200 + + def test_simulate_grok_empty_pattern(self, request_factory): + """Test with empty pattern""" + request = request_factory.post('/Utilities/GrokDebugger/simulate/', { + 'sample_data': 'test data', + 'grok_pattern': '', + 'custom_patterns': '', + 'multiline_mode': 'false' + }) + + response = simulate_grok(request) + assert response.status_code == 200 + + def test_simulate_grok_invalid_request_method(self, request_factory): + """Test GET request to simulate endpoint (should only accept POST)""" + request = request_factory.get('/Utilities/GrokDebugger/simulate/') + response = simulate_grok(request) + + assert response.status_code == 200 + content = response.content.decode('utf-8') + assert 'Invalid request method' in content + + def test_simulate_grok_special_characters(self, request_factory, sample_log_data): + """Test handling of special characters and potential XSS""" + request = request_factory.post('/Utilities/GrokDebugger/simulate/', { + 'sample_data': sample_log_data['special_chars'], + 'grok_pattern': '%{GREEDYDATA:data}', + 'custom_patterns': '', + 'multiline_mode': 'false' + }) + + response = simulate_grok(request) + assert response.status_code == 200 + + content = response.content.decode('utf-8') + # Check that HTML is escaped + assert '<script>' in content or '', + 'pattern_number': 1, + 'matches': [{ + 'line_number': 1, + 'sample': '', + 'success': False, + 'error': '' + }] + }] + + html = generate_results_html(results) + + # All user input should be escaped + assert '<script>' in html + assert '<img' in html + + def test_generate_results_html_nested_data(self): + """Test HTML generation with nested parsed data""" + results = [{ + 'pattern': '%{IP:client.ip}', + 'pattern_number': 1, + 'matches': [{ + 'line_number': 1, + 'sample': '192.168.1.1', + 'success': True, + 'parsed_data': {'client': {'ip': '192.168.1.1'}} + }] + }] + + html = generate_results_html(results) + + assert 'client' in html + assert '192.168.1.1' in html + + +@pytest.mark.django_db +class TestGrokDebuggerIntegration: + """Integration tests for the full Grok Debugger workflow""" + + def test_full_workflow_simple_pattern(self, authenticated_client): + """Test complete workflow from page load to simulation""" + # Load the page + response = authenticated_client.get('/Utilities/GrokDebugger/') + assert response.status_code == 200 + + # Simulate a grok pattern + response = authenticated_client.post('/Utilities/GrokDebugger/simulate/', { + 'sample_data': '192.168.1.1', + 'grok_pattern': '%{IP:ip}', + 'custom_patterns': '', + 'multiline_mode': 'false' + }) + assert response.status_code == 200 + assert b'192.168.1.1' in response.content + + def test_full_workflow_with_custom_patterns(self, authenticated_client, custom_patterns): + """Test workflow with custom patterns""" + response = authenticated_client.post('/Utilities/GrokDebugger/simulate/', { + 'sample_data': '2024-01-15', + 'grok_pattern': '%{CUSTOM_DATE:date}', + 'custom_patterns': custom_patterns, + 'multiline_mode': 'false' + }) + assert response.status_code == 200 + assert b'2024-01-15' in response.content + + +# ============================================================================ +# Additional gap-filling tests +# ============================================================================ + +@pytest.mark.django_db +class TestGetGrokPatternsErrors: + """Test error-handling branches in get_grok_patterns""" + + def test_get_grok_patterns_file_missing_returns_500(self, request_factory): + """When the grok-patterns file does not exist, the view returns 500 with an error key""" + from unittest.mock import patch + request = request_factory.get('/Utilities/GrokDebugger/patterns/') + + with patch('Utilities.views.open', side_effect=FileNotFoundError("no such file")): + response = get_grok_patterns(request) + + assert response.status_code == 500 + data = json.loads(response.content) + assert 'error' in data + + def test_get_grok_patterns_skips_comment_and_blank_lines(self, request_factory): + """Lines starting with # or blank lines are not included as patterns""" + from unittest.mock import patch, mock_open + fake_content = "# This is a comment\n\nWORD \\b\\w+\\b\n" + request = request_factory.get('/Utilities/GrokDebugger/patterns/') + + with patch('builtins.open', mock_open(read_data=fake_content)): + response = get_grok_patterns(request) + + data = json.loads(response.content) + patterns = data['patterns'] + # Only WORD should be loaded; the comment and blank line must be absent + assert 'WORD' in patterns + for key in patterns: + assert not key.startswith('#') + + def test_get_grok_patterns_skips_lines_without_space(self, request_factory): + """Lines with no whitespace (can't be split into name + definition) are silently skipped""" + from unittest.mock import patch, mock_open + fake_content = "BADLINE\nGOOD pattern_def\n" + request = request_factory.get('/Utilities/GrokDebugger/patterns/') + + with patch('builtins.open', mock_open(read_data=fake_content)): + response = get_grok_patterns(request) + + data = json.loads(response.content) + patterns = data['patterns'] + assert 'GOOD' in patterns + assert 'BADLINE' not in patterns + + +@pytest.mark.django_db +class TestSimulateGrokAdditional: + """Additional simulate_grok edge-case tests""" + + def test_whitespace_only_lines_filtered_from_sample(self, request_factory): + """Whitespace-only sample lines are filtered out before matching""" + request = request_factory.post('/Utilities/GrokDebugger/simulate/', { + 'sample_data': ' \n \t ', + 'grok_pattern': '%{IP:ip}', + 'custom_patterns': '', + 'multiline_mode': 'false' + }) + response = simulate_grok(request) + assert response.status_code == 200 + # No sample lines to process → HTML has no match results + content = response.content.decode('utf-8') + assert 'Match Found' not in content + assert 'No Match' not in content + + def test_whitespace_only_pattern_lines_filtered(self, request_factory): + """Whitespace-only pattern lines are filtered; result is empty HTML""" + request = request_factory.post('/Utilities/GrokDebugger/simulate/', { + 'sample_data': '192.168.1.1', + 'grok_pattern': ' \n\t', + 'custom_patterns': '', + 'multiline_mode': 'false' + }) + response = simulate_grok(request) + assert response.status_code == 200 + # No patterns → empty body (no Pattern N headings) + content = response.content.decode('utf-8') + assert 'Pattern 1' not in content + + def test_custom_patterns_blank_and_comment_lines_ignored(self, request_factory): + """Blank lines and lines without a space in custom_patterns are silently skipped""" + custom = "# comment\n\nMY_IP (?:\\d{1,3}\\.){3}\\d{1,3}\n" + request = request_factory.post('/Utilities/GrokDebugger/simulate/', { + 'sample_data': '10.0.0.1', + 'grok_pattern': '%{MY_IP:ip}', + 'custom_patterns': custom, + 'multiline_mode': 'false' + }) + response = simulate_grok(request) + assert response.status_code == 200 + content = response.content.decode('utf-8') + # MY_IP should have been parsed; match should succeed + assert 'Match Found' in content + + def test_multiline_mode_false_splits_on_newlines(self, request_factory): + """When multiline_mode is false, each non-blank line is treated independently""" + request = request_factory.post('/Utilities/GrokDebugger/simulate/', { + 'sample_data': '192.168.1.1\n10.0.0.1', + 'grok_pattern': '%{IP:ip}', + 'custom_patterns': '', + 'multiline_mode': 'false' + }) + response = simulate_grok(request) + assert response.status_code == 200 + content = response.content.decode('utf-8') + # Both IPs should appear in Line 1 and Line 2 results + assert 'Line 1' in content + assert 'Line 2' in content + assert '192.168.1.1' in content + assert '10.0.0.1' in content + + def test_multiline_mode_true_no_split(self, request_factory): + """When multiline_mode is true, the two-line input is treated as a single chunk""" + request = request_factory.post('/Utilities/GrokDebugger/simulate/', { + 'sample_data': '192.168.1.1\n10.0.0.1', + 'grok_pattern': '%{GREEDYDATA:msg}', + 'custom_patterns': '', + 'multiline_mode': 'true' + }) + response = simulate_grok(request) + assert response.status_code == 200 + content = response.content.decode('utf-8') + # Only one entry (Line 1); Line 2 label must not appear + assert 'Line 2' not in content + + +@pytest.mark.django_db +class TestGenerateResultsHtmlAdditional: + """Additional generate_results_html tests""" + + def test_empty_results_list_returns_empty_string(self): + """No results → empty string (no crash, no stray HTML)""" + output = generate_results_html([]) + assert output == '' + + def test_pattern_error_field_shown_in_html(self): + """When a result has pattern_error set, the error information is included""" + results = [{ + 'pattern': '%{BAD_PATTERN:x}', + 'pattern_number': 1, + 'pattern_error': 'Undefined pattern: BAD_PATTERN', + 'matches': [{ + 'line_number': 1, + 'sample': 'anything', + 'success': False, + 'error': 'Pattern compilation error: Undefined pattern: BAD_PATTERN', + 'error_type': 'compilation' + }] + }] + output = generate_results_html(results) + # The pattern header and the failed match entry should both be present + assert 'Pattern 1' in output + assert 'No Match' in output + assert 'compilation' in output.lower() or 'Pattern compilation' in output + + def test_zero_matched_badge_correct(self): + """Badge shows 0 matched when all lines fail""" + results = [{ + 'pattern': '%{IP:ip}', + 'pattern_number': 1, + 'matches': [ + {'line_number': 1, 'sample': 'hello', 'success': False, 'error': 'no match'}, + {'line_number': 2, 'sample': 'world', 'success': False, 'error': 'no match'}, + ] + }] + output = generate_results_html(results) + assert '0 matched' in output + assert '2 failed' in output + + def test_all_matched_badge_correct(self): + """Badge shows 0 failed when all lines succeed""" + results = [{ + 'pattern': '%{IP:ip}', + 'pattern_number': 1, + 'matches': [ + {'line_number': 1, 'sample': '1.1.1.1', 'success': True, 'parsed_data': {'ip': '1.1.1.1'}}, + {'line_number': 2, 'sample': '2.2.2.2', 'success': True, 'parsed_data': {'ip': '2.2.2.2'}}, + ] + }] + output = generate_results_html(results) + assert '2 matched' in output + assert '0 failed' in output + + def test_html_escape_in_error_message(self): + """Error messages containing HTML special chars are escaped""" + results = [{ + 'pattern': 'p', + 'pattern_number': 1, + 'matches': [{ + 'line_number': 1, + 'sample': 'x', + 'success': False, + 'error': 'bad & "error"' + }] + }] + output = generate_results_html(results) + assert '' not in output # raw tag must not appear + assert '<b>' in output # escaped version must appear + + +@pytest.mark.django_db +class TestAuthenticationAndRouting: + """URL-level authentication and routing tests""" + + def test_grok_debugger_requires_authentication(self, client): + """Unauthenticated request to GrokDebugger redirects to login""" + response = client.get('/Utilities/GrokDebugger/') + assert response.status_code == 302 + assert '/Management/Login/' in response.url + + def test_simulate_grok_requires_authentication(self, client): + """Unauthenticated POST to simulate endpoint redirects to login""" + response = client.post('/Utilities/GrokDebugger/simulate/', { + 'sample_data': '192.168.1.1', + 'grok_pattern': '%{IP:ip}', + }) + assert response.status_code == 302 + assert '/Management/Login/' in response.url + + def test_get_grok_patterns_requires_authentication(self, client): + """Unauthenticated GET to patterns endpoint redirects to login""" + response = client.get('/Utilities/GrokDebugger/patterns/') + assert response.status_code == 302 + assert '/Management/Login/' in response.url diff --git a/tests/conftest.py b/tests/conftest.py new file mode 100644 index 0000000..1c7ab99 --- /dev/null +++ b/tests/conftest.py @@ -0,0 +1,58 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +from django.contrib.auth.models import User +from django.test import Client, RequestFactory +from PipelineManager.models import Connection + +import pytest + + +@pytest.fixture +def request_factory(): + return RequestFactory() + + +@pytest.fixture +def authenticated_client(client, test_user): + client.login(username='testuser', password='testpass123') + return client + + +@pytest.fixture +def client(): + return Client() + + +@pytest.fixture +def test_user(db): + from Management.models import UserProfile + + user = User.objects.create_user( + username='testuser', + password='testpass123', + email='test@example.com' + ) + user.is_superuser = True + user.is_staff = True + user.save() + + UserProfile.objects.get_or_create( + user=user, + defaults={'role': 'admin'} + ) + + return user + + +@pytest.fixture +def test_connection(db): + connection = Connection.objects.create( + name='Test Connection', + connection_type='CENTRALIZED', + host='https://localhost:9200', + username='elastic', + password='changeme' + ) + return connection From 5016babdf1a6c87df04891be0b2b973d2e119054 Mon Sep 17 00:00:00 2001 From: Aaron Mildenstein Date: Thu, 3 Sep 2026 14:16:14 -0600 Subject: [PATCH 31/62] Fixed bugs that came up during testing Also set testcontainers version higher to eliminate a warning, and use the newer testcontainers.community.* modules --- CHANGELOG.md | 3 ++ pyproject.toml | 4 +- src/logstashui/PipelineManager/agent_modes.py | 27 +++++++++++-- tests/Database/integration/conftest.py | 6 +-- tests/Database/integration/test_db_config.py | 34 ++++++++++++----- .../integration/test_migrate_engine.py | 16 ++++---- tests/Database/integration/test_orm.py | 38 ++++++++++++++----- .../PipelineManager/unit/test_agent_modes.py | 13 ++++++- uv.lock | 4 +- 9 files changed, 107 insertions(+), 38 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 819badc..08989d1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -36,6 +36,8 @@ SQLite does not scale under gunicorn/gevent. Operators can now run LogstashUI on - `bin/test_databases.sh` / `bin/test_databases.bat` start local Docker Postgres 16, MariaDB 11, and MySQL 8.0 and run the full suite on each engine, then run `tests/Database/` via testcontainers. CI workflow `.github/workflows/test-databases.yml` calls the same script. - Self-contained database test suite at `tests/Database/integration/` uses `testcontainers` (postgres:16, mysql:8.0, mariadb:11 — no external Docker Compose). Parametrized over PostgreSQL and MySQL; MariaDB covered for `check_server_version` version-detection. Skips gracefully when Docker is unavailable. Covers DB config, migrations (clean, idempotent, no unapplied), ORM CRUD/JSON/uniqueness, and full SQLite → PG/MySQL/MariaDB `migrate-engine` round-trips. Run with `uv run pytest tests/Database/ -v --no-cov` after `uv sync --group dev --extra databases`. - All tests migrated from `src/logstashui//tests/` to a dedicated `tests/` tree at the project root. Layout: `tests//unit/` per Django app, `tests/Database/unit|integration/` for database and migration tests. Shared fixtures moved from `Common/test_resources.py` to `tests/conftest.py` (auto-discovered by pytest). `testpaths` trimmed to `["tests"]`; `pythonpath = ["src/logstashui"]` unchanged so app imports still resolve. Run any app in isolation: `uv run pytest tests/SNMP -v`. +- `test_case_sensitive_unique` now asserts case-sensitive uniqueness at both layers. `Network.save()` calls `full_clean()`, so a duplicate name raises `ValidationError` from `validate_unique()` and the `INSERT` is never issued — the test previously expected an `IntegrityError` that the database could never raise, and failed identically on both engines. The collation check (`utf8mb4_bin` on MySQL, default on PostgreSQL) rides on the `validate_unique()` query; the database unique index is verified separately via `bulk_create()`, which bypasses `save()`, inside `transaction.atomic()` so PostgreSQL can roll back the aborted statement before cleanup. +- Container fixtures moved from the deprecated `testcontainers.postgres` / `testcontainers.mysql` shims to `testcontainers.community.*`. The dev-dependency floor is raised to `testcontainers>=4.15.0`, the first release containing that package. - Smoke compose is still SQLite (product CA / PUID unchanged). ### Kubernetes and database docs @@ -48,6 +50,7 @@ SQLite does not scale under gunicorn/gevent. Operators can now run LogstashUI on ### Fixes - `LOGSTASHUI_TLS=false` now suppresses the Django-level HTTP→HTTPS redirect (`SECURE_SSL_REDIRECT`) in addition to disabling the Gunicorn TLS certificate. Previously, running the container with `-e LOGSTASHUI_TLS=false` still returned a `301` because `SECURE_SSL_REDIRECT` was gated on `DEBUG` only. Both knobs are now independent. +- The pipeline editor's simulation **Target** picker no longer hides the embedded agent on first page load. Making the probe non-blocking left `list_simulation_targets()` requiring a successful probe it never performed, so the sticky embedded row lost a race against the background thread and the dropdown rendered empty until a later refresh. The row is now dropped only when a probe has *explicitly* reported the agent offline — never-probed is treated as unknown, not offline. Target rows also carry a `discovered` flag so callers can distinguish a confirmed agent from an unconfirmed one. ### Agent version display diff --git a/pyproject.toml b/pyproject.toml index 26f98ed..4b15db3 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -99,8 +99,8 @@ dev = [ "pytest>=9.0.2", "pytest-cov>=7.1.0", "pytest-django>=4.10.0", - "testcontainers[postgres]>=4.8.0", - "testcontainers[mysql]>=4.8.0", + "testcontainers[postgres]>=4.15.0", + "testcontainers[mysql]>=4.15.0", ] [tool.pytest.ini_options] diff --git a/src/logstashui/PipelineManager/agent_modes.py b/src/logstashui/PipelineManager/agent_modes.py index 8851310..a47fe9b 100644 --- a/src/logstashui/PipelineManager/agent_modes.py +++ b/src/logstashui/PipelineManager/agent_modes.py @@ -528,7 +528,8 @@ def ensure_embedded_connection(*, probe: bool = True) -> Connection | None: agent_api_port=port, logstash_api_port=EMBEDDED_LOGSTASH_API_PORT, last_check_in=now if online else None, - status_blob=update_fields.get("status_blob") or {"embedded": True, "online": online}, + # probe=False leaves online *unknown* rather than asserting offline + status_blob=update_fields.get("status_blob") or {"embedded": True}, ) conn.save() return conn @@ -574,6 +575,21 @@ def is_embedded_discovered(conn) -> bool: return (now - ts).total_seconds() < 600 +def embedded_probe_failed(conn) -> bool: + """True only when a probe explicitly reported the embedded agent offline. + + A row that has never been probed carries no ``online`` key — that is + unknown, not failed, and the sticky picker row stays visible. + """ + if conn is None: + return False + if isinstance(conn, dict): + blob = conn.get("status_blob") or {} + else: + blob = getattr(conn, "status_blob", None) or {} + return blob.get("online") is False + + def is_embedded_connection(conn) -> bool: """True for the docker/local pseudo agent (dict or model).""" if conn is None: @@ -620,9 +636,12 @@ def list_simulation_targets(active_only: bool = True, *, ensure_embedded: bool = or ("system" if policy.policy_type == Policy.PolicyType.SIMULATE else "") ) if policy.policy_type == Policy.PolicyType.EMBEDDED: - # Picker only — and only after a successful live probe - if not is_embedded_discovered(conn): + # Picker only. This path never probes (see + # refresh_embedded_connection_async), so keep the sticky row until + # a probe has actually failed — unprobed is unknown, not offline. + if embedded_probe_failed(conn): continue + discovered = is_embedded_discovered(conn) # Closed select: terse; detail on hover / open option list label = "embedded" ver_label = version or "docker" @@ -640,6 +659,7 @@ def list_simulation_targets(active_only: bool = True, *, ensure_embedded: bool = host = conn.host or "127.0.0.1" detail = f"embedded · {host} · Logstash {ver_label}" else: + discovered = True n = conn.instance_id or "?" ver_label = version or "system" host = conn.host or "127.0.0.1" @@ -664,6 +684,7 @@ def list_simulation_targets(active_only: bool = True, *, ensure_embedded: bool = "logstash_source": policy.logstash_source, "host": host, "base_url": base_url, + "discovered": discovered, "last_selected_at": conn.last_selected_at.isoformat() if conn.last_selected_at else None, diff --git a/tests/Database/integration/conftest.py b/tests/Database/integration/conftest.py index 1ae0172..73d75d7 100644 --- a/tests/Database/integration/conftest.py +++ b/tests/Database/integration/conftest.py @@ -50,7 +50,7 @@ def skip_if_no_docker(): @pytest.fixture(scope="session") def postgres_container(): - from testcontainers.postgres import PostgresContainer + from testcontainers.community.postgres import PostgresContainer with PostgresContainer( image="postgres:16", @@ -63,7 +63,7 @@ def postgres_container(): @pytest.fixture(scope="session") def mysql_container(): - from testcontainers.mysql import MySqlContainer + from testcontainers.community.mysql import MySqlContainer c = MySqlContainer( image="mysql:8.0", @@ -79,7 +79,7 @@ def mysql_container(): @pytest.fixture(scope="session") def mariadb_container(): - from testcontainers.mysql import MySqlContainer + from testcontainers.community.mysql import MySqlContainer c = MySqlContainer( image="mariadb:11", diff --git a/tests/Database/integration/test_db_config.py b/tests/Database/integration/test_db_config.py index 0aed525..dcf731b 100644 --- a/tests/Database/integration/test_db_config.py +++ b/tests/Database/integration/test_db_config.py @@ -1,6 +1,6 @@ -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. +# Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +# or more contributor license agreements. Licensed under the Elastic License; +# you may not use this file except in compliance with the Elastic License. """ Integration tests for database configuration and server version checking. @@ -16,11 +16,11 @@ from LogstashUI.database import build_databases - # --------------------------------------------------------------------------- # Subprocess helper # --------------------------------------------------------------------------- + def _run_python(code: str, extra_env: dict[str, str]) -> str: from LogstashUI import migrate_engine as me @@ -75,18 +75,25 @@ def _run_python(code: str, extra_env: dict[str, str]) -> str: # Tests — build_databases() dict structure (no Docker needed) # --------------------------------------------------------------------------- + def test_mysql_options_include_utf8mb4(monkeypatch, tmp_path): """build_databases() for MySQL must include utf8mb4 charset and utf8mb4_bin collation.""" for key in ( - "LOGSTASHUI_DB_ENGINE", "LOGSTASHUI_DB_HOST", "LOGSTASHUI_DB_PORT", - "LOGSTASHUI_DB_USER", "LOGSTASHUI_DB_PASSWORD", "LOGSTASHUI_DB_NAME", + "LOGSTASHUI_DB_ENGINE", + "LOGSTASHUI_DB_HOST", + "LOGSTASHUI_DB_PORT", + "LOGSTASHUI_DB_USER", + "LOGSTASHUI_DB_PASSWORD", + "LOGSTASHUI_DB_NAME", "LOGSTASHUI_DB_CONN_MAX_AGE", ): monkeypatch.delenv(key, raising=False) monkeypatch.setenv("LOGSTASHUI_DB_ENGINE", "mysql") monkeypatch.setenv("LOGSTASHUI_DB_HOST", "127.0.0.1") monkeypatch.setenv("LOGSTASHUI_DB_USER", "root") - monkeypatch.setattr("LogstashUI.database._import_or_raise", lambda *a, **k: _fake_pymysql()) + monkeypatch.setattr( + "LogstashUI.database._import_or_raise", lambda *a, **k: _fake_pymysql() + ) db = build_databases(tmp_path)["default"] assert db["OPTIONS"]["charset"] == "utf8mb4" assert "utf8mb4_bin" in db["OPTIONS"]["init_command"] @@ -96,6 +103,7 @@ def test_mysql_options_include_utf8mb4(monkeypatch, tmp_path): def _fake_pymysql(): from types import SimpleNamespace + fake = SimpleNamespace( version_info=(1, 1, 1, "final", 0), install_as_MySQLdb=lambda: None, @@ -106,8 +114,12 @@ def _fake_pymysql(): def test_conn_max_age_applied(monkeypatch, tmp_path): """LOGSTASHUI_DB_CONN_MAX_AGE overrides the default 60s for postgres.""" for key in ( - "LOGSTASHUI_DB_ENGINE", "LOGSTASHUI_DB_HOST", "LOGSTASHUI_DB_PORT", - "LOGSTASHUI_DB_USER", "LOGSTASHUI_DB_PASSWORD", "LOGSTASHUI_DB_NAME", + "LOGSTASHUI_DB_ENGINE", + "LOGSTASHUI_DB_HOST", + "LOGSTASHUI_DB_PORT", + "LOGSTASHUI_DB_USER", + "LOGSTASHUI_DB_PASSWORD", + "LOGSTASHUI_DB_NAME", "LOGSTASHUI_DB_CONN_MAX_AGE", ): monkeypatch.delenv(key, raising=False) @@ -144,6 +156,7 @@ def test_build_databases_returns_valid_dict(engine_env, tmp_path, monkeypatch): # Tests — real container connections (subprocess) # --------------------------------------------------------------------------- + def test_real_connection_opens(engine_env, tmp_path): """Django can open a connection to the container database.""" engine, env = engine_env @@ -160,7 +173,8 @@ def test_check_server_version_passes_on_real_connection(engine_env, tmp_path): def test_check_server_version_mariadb_branch(mariadb_container, tmp_path): """check_server_version() MariaDB detection branch passes on a real MariaDB server.""" - from tests.integration.conftest import mysql_env + from tests.Database.integration.conftest import mysql_env + env = mysql_env(mariadb_container) full_env = {**env, "LOGSTASHUI_DATA_DIR": str(tmp_path)} _run_python(_CHECK_SERVER_VERSION, full_env) diff --git a/tests/Database/integration/test_migrate_engine.py b/tests/Database/integration/test_migrate_engine.py index fab32f8..d9d7385 100644 --- a/tests/Database/integration/test_migrate_engine.py +++ b/tests/Database/integration/test_migrate_engine.py @@ -1,6 +1,6 @@ -#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one -#or more contributor license agreements. Licensed under the Elastic License; -#you may not use this file except in compliance with the Elastic License. +# Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +# or more contributor license agreements. Licensed under the Elastic License; +# you may not use this file except in compliance with the Elastic License. """ Integration tests for cmd_migrate_engine (SQLite → PostgreSQL / MySQL / MariaDB). @@ -18,7 +18,7 @@ import pytest from LogstashUI import migrate_engine as me -from tests.integration.conftest import ( +from tests.Database.integration.conftest import ( create_mysql_db, create_pg_db, drop_mysql_db, @@ -28,11 +28,11 @@ pg_env, ) - # --------------------------------------------------------------------------- # Subprocess helper # --------------------------------------------------------------------------- + def _run_python(code: str, extra_env: dict[str, str]) -> str: env = os.environ.copy() env.update(extra_env) @@ -128,6 +128,7 @@ def _run_python(code: str, extra_env: dict[str, str]) -> str: # Core migration helper # --------------------------------------------------------------------------- + def _run_to(tmp_path, target_env: dict[str, str]) -> dict: """ Seed a fresh SQLite database, run cmd_migrate_engine to the target, @@ -156,9 +157,7 @@ def _run_to(tmp_path, target_env: dict[str, str]) -> dict: try: rc = me.cmd_migrate_engine(ns) except SystemExit as exc: - raise AssertionError( - f"cmd_migrate_engine SystemExit {exc.code}" - ) from exc + raise AssertionError(f"cmd_migrate_engine SystemExit {exc.code}") from exc assert rc == 0 raw = _run_python(_COUNT, full_target) finally: @@ -181,6 +180,7 @@ def _run_to(tmp_path, target_env: dict[str, str]) -> dict: # Tests # --------------------------------------------------------------------------- + def test_migrate_engine_to_postgres(postgres_container, tmp_path): dbname = new_dbname() base = pg_env(postgres_container) diff --git a/tests/Database/integration/test_orm.py b/tests/Database/integration/test_orm.py index 5aef6ab..c7ddff0 100644 --- a/tests/Database/integration/test_orm.py +++ b/tests/Database/integration/test_orm.py @@ -294,18 +294,34 @@ def _migrate(env: dict[str, str]) -> None: import json, os, django os.environ.setdefault("DJANGO_SETTINGS_MODULE", "LogstashUI.settings") django.setup() -from django.db import IntegrityError +from django.core.exceptions import ValidationError +from django.db import IntegrityError, transaction from SNMP.models import Network Network.objects.filter(name__in=["CaseNet", "casenet"]).delete() -Network.objects.create(name="CaseNet", network_range="10.1.0.0/24") -# Different case must succeed -Network.objects.create(name="casenet", network_range="10.2.0.0/24") -# Exact duplicate must fail try: - Network.objects.create(name="CaseNet", network_range="10.3.0.0/24") - raise AssertionError("Expected IntegrityError for duplicate Network.name") -except IntegrityError: - pass + Network.objects.create(name="CaseNet", network_range="10.1.0.0/24") + # Different case must succeed. This is the actual collation assertion: + # Network.save() -> full_clean() -> validate_unique() runs + # SELECT ... WHERE name = 'casenet', which matches 'CaseNet' under a + # case-insensitive collation and raises ValidationError. + Network.objects.create(name="casenet", network_range="10.2.0.0/24") + # Exact duplicate via the ORM: full_clean() rejects it before the INSERT. + try: + Network.objects.create(name="CaseNet", network_range="10.3.0.0/24") + raise AssertionError("Expected ValidationError for duplicate Network.name") + except ValidationError: + pass + # ...and the DB unique index still holds when full_clean() is bypassed. + # bulk_create() does not call save(). atomic() so the aborted statement is + # rolled back and the cleanup below can still run on PostgreSQL. + try: + with transaction.atomic(): + Network.objects.bulk_create( + [Network(name="CaseNet", network_range="10.3.0.0/24")] + ) + raise AssertionError("Expected IntegrityError for duplicate Network.name") + except IntegrityError: + pass finally: Network.objects.filter(name__in=["CaseNet", "casenet"]).delete() print(json.dumps({"ok": True})) @@ -381,7 +397,11 @@ def test_unique_pipeline_per_policy(engine_env, tmp_path): def test_case_sensitive_unique(engine_env, tmp_path): """Both engines treat unique names as case-sensitive. + On MySQL this validates utf8mb4_bin is active; on PostgreSQL it's the default. + The collation check rides on validate_unique(), since Network.save() calls + full_clean() and so never reaches the INSERT. The DB-level unique index is + verified separately via bulk_create(), which bypasses save(). """ engine, env = engine_env full_env = {**env, "LOGSTASHUI_DATA_DIR": str(tmp_path)} diff --git a/tests/PipelineManager/unit/test_agent_modes.py b/tests/PipelineManager/unit/test_agent_modes.py index 1d44c8c..7f40e55 100644 --- a/tests/PipelineManager/unit/test_agent_modes.py +++ b/tests/PipelineManager/unit/test_agent_modes.py @@ -624,7 +624,7 @@ def test_list_targets_includes_embedded(system_policies, monkeypatch): assert targets[-1]['label'] == 'embedded' -def test_list_targets_omits_undiscovered_embedded(system_policies, monkeypatch): +def test_list_targets_keeps_unprobed_embedded(system_policies, monkeypatch): """Listing does not probe; the sticky embedded row is still included.""" monkeypatch.setattr( 'PipelineManager.agent_modes.probe_embedded_agent_online', @@ -635,6 +635,17 @@ def test_list_targets_omits_undiscovered_embedded(system_policies, monkeypatch): assert targets[-1]['label'] == 'embedded' +def test_list_targets_hides_embedded_after_failed_probe(system_policies, monkeypatch): + """An explicit probe failure removes the sticky row; never-probed does not.""" + monkeypatch.setattr( + 'PipelineManager.agent_modes.probe_embedded_agent_online', + lambda timeout=2.0: False, + ) + ensure_embedded_connection() # probe=True -> records online: False + targets = list_simulation_targets(ensure_embedded=True) + assert not any(t['policy_type'] == 'EMBEDDED' for t in targets) + + def test_list_targets_embedded_after_simulate(system_policies, monkeypatch): _, _, simulate, _ = system_policies monkeypatch.setattr( diff --git a/uv.lock b/uv.lock index 9feeb07..3816186 100644 --- a/uv.lock +++ b/uv.lock @@ -646,8 +646,8 @@ dev = [ { name = "pytest", specifier = ">=9.0.2" }, { name = "pytest-cov", specifier = ">=7.1.0" }, { name = "pytest-django", specifier = ">=4.10.0" }, - { name = "testcontainers", extras = ["mysql"], specifier = ">=4.8.0" }, - { name = "testcontainers", extras = ["postgres"], specifier = ">=4.8.0" }, + { name = "testcontainers", extras = ["mysql"], specifier = ">=4.15.0" }, + { name = "testcontainers", extras = ["postgres"], specifier = ">=4.15.0" }, ] [[package]] From 8eaf0140bcc743fcfe526779bd64986f288c38b5 Mon Sep 17 00:00:00 2001 From: Aaron Mildenstein Date: Thu, 3 Sep 2026 15:48:40 -0600 Subject: [PATCH 32/62] Add basic API access to the apps The primary use case for this right now is to allow adding CPM endpoints by API call. Creating/Revoking/Deleting API keys is done in the UI in the Management pane. More features and a real `/api` endpoint will eventually follow. --- CHANGELOG.md | 11 + docs/docs/logstashui/api_access.md | 106 +++++++ docs/docs/logstashui/index.md | 1 + pyproject.toml | 2 +- src/logstashui/Common/decorators.py | 25 +- src/logstashui/Common/middleware.py | 93 +++++++ src/logstashui/Documentation/views.py | 1 + src/logstashui/LogstashUI/settings.py | 6 + .../Management/templates/api_tokens.html | 153 ++++++++++ .../components/api_token_created.html | 51 ++++ .../templates/components/api_token_row.html | 84 ++++++ .../Management/templates/management.html | 13 + src/logstashui/Management/urls.py | 1 + src/logstashui/Management/views.py | 107 ++++++- .../migrations/0028_apikey_admin_tokens.py | 80 ++++++ src/logstashui/PipelineManager/models.py | 140 +++++++++- .../Common/unit/test_api_token_middleware.py | 263 ++++++++++++++++++ tests/Management/unit/test_api_tokens.py | 166 +++++++++++ 18 files changed, 1283 insertions(+), 20 deletions(-) create mode 100644 docs/docs/logstashui/api_access.md create mode 100644 src/logstashui/Management/templates/api_tokens.html create mode 100644 src/logstashui/Management/templates/components/api_token_created.html create mode 100644 src/logstashui/Management/templates/components/api_token_row.html create mode 100644 src/logstashui/PipelineManager/migrations/0028_apikey_admin_tokens.py create mode 100644 tests/Common/unit/test_api_token_middleware.py create mode 100644 tests/Management/unit/test_api_tokens.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 08989d1..e3510aa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -38,6 +38,7 @@ SQLite does not scale under gunicorn/gevent. Operators can now run LogstashUI on - All tests migrated from `src/logstashui//tests/` to a dedicated `tests/` tree at the project root. Layout: `tests//unit/` per Django app, `tests/Database/unit|integration/` for database and migration tests. Shared fixtures moved from `Common/test_resources.py` to `tests/conftest.py` (auto-discovered by pytest). `testpaths` trimmed to `["tests"]`; `pythonpath = ["src/logstashui"]` unchanged so app imports still resolve. Run any app in isolation: `uv run pytest tests/SNMP -v`. - `test_case_sensitive_unique` now asserts case-sensitive uniqueness at both layers. `Network.save()` calls `full_clean()`, so a duplicate name raises `ValidationError` from `validate_unique()` and the `INSERT` is never issued — the test previously expected an `IntegrityError` that the database could never raise, and failed identically on both engines. The collation check (`utf8mb4_bin` on MySQL, default on PostgreSQL) rides on the `validate_unique()` query; the database unique index is verified separately via `bulk_create()`, which bypasses `save()`, inside `transaction.atomic()` so PostgreSQL can roll back the aborted statement before cleanup. - Container fixtures moved from the deprecated `testcontainers.postgres` / `testcontainers.mysql` shims to `testcontainers.community.*`. The dev-dependency floor is raised to `testcontainers>=4.15.0`, the first release containing that package. +- API token coverage in `tests/Common/unit/test_api_token_middleware.py` and `tests/Management/unit/test_api_tokens.py`. The load-bearing case is the inverse one: a logged-in session POSTing without a CSRF token must still be rejected, so the exemption cannot regress into a site-wide CSRF bypass. Also covers revoked/expired/inactive-owner tokens, readonly-owner denial, agent keys passing through untouched, and that re-saving a token does not double-hash it. - Smoke compose is still SQLite (product CA / PUID unchanged). ### Kubernetes and database docs @@ -50,8 +51,18 @@ SQLite does not scale under gunicorn/gevent. Operators can now run LogstashUI on ### Fixes - `LOGSTASHUI_TLS=false` now suppresses the Django-level HTTP→HTTPS redirect (`SECURE_SSL_REDIRECT`) in addition to disabling the Gunicorn TLS certificate. Previously, running the container with `-e LOGSTASHUI_TLS=false` still returned a `301` because `SECURE_SSL_REDIRECT` was gated on `DEBUG` only. Both knobs are now independent. +- `ApiKey.save()` no longer re-hashes an already-hashed key. `make_password()` ran unconditionally on every save, so any update to an existing row silently rewrote the hash and invalidated the credential. Latent until now — nothing re-saved an `ApiKey` — but renaming or revoking a token does. - The pipeline editor's simulation **Target** picker no longer hides the embedded agent on first page load. Making the probe non-blocking left `list_simulation_targets()` requiring a successful probe it never performed, so the sticky embedded row lost a race against the background thread and the dropdown rendered empty until a later refresh. The row is now dropped only when a probe has *explicitly* reported the agent offline — never-probed is treated as unknown, not offline. Target rows also carry a `discovered` flag so callers can distinguish a confirmed agent from an unconfirmed one. +### API access + +- **Admin API tokens** (`Management → API Tokens`) let scripts, CI, and provisioning tools call LogstashUI's JSON endpoints without a browser session. Send `Authorization: ApiKey lsui__`. The original motivation was registering a remote Elasticsearch cluster for Centralized Pipeline Management from `curl`, which previously failed CSRF verification; it now works against the existing `/ConnectionManager/AddConnection` URL. Every other JSON endpoint accepts a token too — no per-endpoint opt-in. +- A token acts as the user who created it and inherits that account's role, so audit lines stay attributable and a `readonly` user's token stays readonly. Tokens can carry an optional expiry and be revoked or deleted; revocation takes effect on the next request. Only a hash is stored, so the secret is displayed exactly once at creation. +- Implemented as a pair of middlewares rather than per-view decorators. `ApiTokenCsrfMiddleware` runs immediately before `CsrfViewMiddleware` and sets `_dont_enforce_csrf_checks` **only after a token verifies** — a forged or absent header cannot switch CSRF off, and cookie-authenticated browser requests are unaffected. `ApiTokenUserMiddleware` runs just after `AuthenticationMiddleware`, which would otherwise overwrite `request.user`; the split is forced by that ordering. Because `request.user` becomes a real user, `LoginRequiredMiddleware` and `require_admin_role` pass on their own — no `@csrf_exempt` and no `LOGIN_REQUIRED_IGNORE_PATHS` entries were added. +- `require_admin_role` now answers API-token callers with JSON instead of an `HX-Trigger` toast, which is unreadable to a script. +- Admin tokens reuse the agent `ApiKey` table and its `make_password`/`check_password` machinery. The one addition is an unhashed, indexed `prefix` column: agent keys are found via the `connection_id` in the request body before their hash is checked, but a token presents only a header, and without a lookup key resolving it would mean a PBKDF2 comparison against every row. Existing agent keys get `prefix=NULL` and are never matched by the middleware. +- See [API Access](docs/docs/logstashui/api_access.md). + ### Agent version display - ConnectionManager and Policy editor Agents tab show a cyan **LS X.Y.Z** pill for the Logstash version the agent reported (`logstash_version_resolved`, else Logstash API version). Hidden until known. diff --git a/docs/docs/logstashui/api_access.md b/docs/docs/logstashui/api_access.md new file mode 100644 index 0000000..5566668 --- /dev/null +++ b/docs/docs/logstashui/api_access.md @@ -0,0 +1,106 @@ +# API Access + +LogstashUI's JSON endpoints are normally reached from a logged-in browser session. An **API +token** lets a script, CI job, or provisioning tool call the same endpoints with `curl`. + +## Creating a token + +**Management → API Tokens → Create Token.** + +Give the token a name, optionally set an expiry in days, and copy the value. It looks like: + +```plain +lsui_bfed5132382f_kmrqP9ETXxF0yKz1s--1SPrHl93yllxhq_kj2Hpz3i8 +``` + +Only a hash is stored, so **the token is shown exactly once**. If you lose it, revoke it and mint +a new one. + +A token acts as the user who created it and inherits that account's role. A token created by a +`readonly` user can read but not write. Revoking takes effect on the next request. + +## Making requests + +Send the token in an `Authorization` header: + +```bash +curl -k -X POST \ + -H "Authorization: ApiKey lsui__" \ + -d "connection_type=CENTRALIZED&name=my-cluster&host=https://es.example.com&port=443&api_key=" \ + https://localhost:8443/ConnectionManager/AddConnection +``` + +Token requests are exempt from CSRF, so no cookie jar or `X-CSRFToken` header is needed. CSRF +remains fully enforced for ordinary browser sessions — the exemption applies only after a token +has been verified. + +Every existing JSON endpoint accepts a token, not just the one below. + +## Adding a remote Elasticsearch cluster for Centralized Pipeline Management + +`POST /ConnectionManager/AddConnection` (note: **no trailing slash**), form-encoded. + +| Field | Required | Notes | +| --- | --- | --- | +| `connection_type` | yes | `CENTRALIZED` | +| `name` | yes | Display name, must be unique | +| `host` | one of | Full URL, e.g. `https://es.example.com` | +| `cloud_id` | one of | Elastic Cloud ID, as an alternative to `host` | +| `port` | no | Usually `9200`, or `443` for Cloud | +| `api_key` | no | Elasticsearch API key | +| `username` / `password` | no | Basic auth, as an alternative to `api_key` | +| `cloud_url` | no | Full cluster URL | + +Either `host` or `cloud_id` must be present. Supply either `api_key` or `username`+`password`; +missing credentials are not rejected by validation but will fail the connectivity test. + +On success: + +```json +{"success": true, "connection_id": 7, "message": "Connection created and tested successfully!"} +``` + +LogstashUI pings the cluster before saving. If the ping fails, the connection is rolled back and +the error is returned. + +## Two things that will bite you + +**Check the body, not the status code.** `AddConnection` returns **HTTP 200** even when the +request fails, with the reason in the body: + +```json +{"success": false, "error": "Connection error caused by: ..."} +``` + +So test `.success`: + +```bash +resp=$(curl -sk -X POST -H "Authorization: ApiKey $TOKEN" -d "$BODY" \ + https://localhost:8443/ConnectionManager/AddConnection) +echo "$resp" | grep -q '"success": true' || { echo "failed: $resp" >&2; exit 1; } +``` + +**`-k` is required against the built-in certificate.** LogstashUI serves TLS from its own product +CA, which curl does not trust by default. Either pass `-k`, or fetch the CA and trust it properly: + +```bash +curl -sk https://localhost:8443/.well-known/logstashui/ca.crt -o logstashui-ca.crt +curl --cacert logstashui-ca.crt -X POST -H "Authorization: ApiKey $TOKEN" ... +``` + +## Error responses + +| Status | Meaning | +| --- | --- | +| `401` | Token missing, malformed, unknown, revoked, expired, or owned by a disabled user | +| `403` | Token is valid but its owner lacks the required role | +| `405` | Wrong HTTP method | +| `200` with `"success": false` | Request reached the view and was rejected there | + +## Notes + +- Agent enrollment keys are a separate credential and are unaffected. They carry no `lsui_` + marker and continue to authenticate as before. +- Verifying a token costs one PBKDF2 comparison. That is deliberate, and fine at scripting rates, + but do not put a token request in a tight loop. +- Never commit a token. Treat it as equivalent to the password of the account that created it. diff --git a/docs/docs/logstashui/index.md b/docs/docs/logstashui/index.md index 2f15ebb..d550264 100644 --- a/docs/docs/logstashui/index.md +++ b/docs/docs/logstashui/index.md @@ -39,6 +39,7 @@ Configure polling, traps, and discovery through a web interface. ## Documentation +- **[API Access](/docs/docs/logstashui/api_access.md)** - Scripting LogstashUI with API tokens and curl - **[Architecture](/docs/docs/logstashui/architecture.md)** - System architecture - **[Compatibility](/docs/docs/logstashui/compatibility.md)** - Logstash version compatibility and requirements - **[Configuration](/docs/docs/logstashui/configuration/index.md)** - Configuration options and settings for LogstashUI diff --git a/pyproject.toml b/pyproject.toml index 4b15db3..9ef6de6 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "LogstashUI" -version = "0.5.1" +version = "0.5.2" description = "A control plane and UI for Logstash!" readme = "README.md" license = { file = "LICENSE.txt" } diff --git a/src/logstashui/Common/decorators.py b/src/logstashui/Common/decorators.py index 243e7cc..dc705cc 100644 --- a/src/logstashui/Common/decorators.py +++ b/src/logstashui/Common/decorators.py @@ -3,11 +3,26 @@ #you may not use this file except in compliance with the Elastic License. from functools import wraps -from django.http import HttpResponse +from django.http import HttpResponse, JsonResponse import logging logger = logging.getLogger(__name__) + +def _denied(request, message): + """Build a denial a browser *or* a script can read. + + The HX-Trigger toast is meaningless to curl, so API-token callers get JSON. + """ + if getattr(request, '_api_token', None) is not None: + return JsonResponse({'success': False, 'error': message}, status=403) + response = HttpResponse(message, status=403) + response['HX-Trigger'] = ( + '{"showToastEvent": {"message": "%s", "type": "error"}}' % message + ) + return response + + def require_admin_role(view_func): """ Decorator to check if user has admin role before allowing access to view. @@ -17,17 +32,13 @@ def require_admin_role(view_func): def wrapper(request, *args, **kwargs): # Check if user is authenticated if not request.user.is_authenticated: - response = HttpResponse('You must be logged in to perform this action', status=403) - response['HX-Trigger'] = '{"showToastEvent": {"message": "You must be logged in to perform this action", "type": "error"}}' - return response + return _denied(request, 'You must be logged in to perform this action') # Check if user has admin role if not hasattr(request.user, 'profile') or request.user.profile.role != 'admin': role_info = f"'{request.user.profile.role}'" if hasattr(request.user, 'profile') else 'no profile' logger.warning(f"User '{request.user.username}' with {role_info} attempted to access admin-only function: {view_func.__name__}") - response = HttpResponse('Access denied: Admin role required', status=403) - response['HX-Trigger'] = '{"showToastEvent": {"message": "Access denied: Admin role required", "type": "error"}}' - return response + return _denied(request, 'Access denied: Admin role required') # User is admin, proceed with the view return view_func(request, *args, **kwargs) diff --git a/src/logstashui/Common/middleware.py b/src/logstashui/Common/middleware.py index df1d7c9..9b65a4e 100644 --- a/src/logstashui/Common/middleware.py +++ b/src/logstashui/Common/middleware.py @@ -3,6 +3,99 @@ #you may not use this file except in compliance with the Elastic License. +import logging + +from django.http import JsonResponse +from django.utils import timezone + +logger = logging.getLogger(__name__) + +_AUTH_SCHEME = 'ApiKey ' + + +def _resolve_api_token(request): + """Resolve an admin API token from the Authorization header. + + Returns the ``ApiKey`` row on success, ``None`` when the request carries no + admin token at all (browser traffic, or an agent key — those have no + ``lsui_`` marker and are authenticated by the agent views themselves), or + the string ``'invalid'`` when a token was offered but is not usable. + """ + header = request.headers.get('Authorization', '') + if not header.startswith(_AUTH_SCHEME): + return None + + from PipelineManager.models import ApiKey + + prefix, secret = ApiKey.parse_token(header[len(_AUTH_SCHEME):].strip()) + if prefix is None: + # Not an admin token. Leave agent keys entirely alone. + return None + + token = ApiKey.objects.filter(prefix=prefix).select_related('user').first() + if token is None or token.user is None: + return 'invalid' + if not token.is_active or not token.user.is_active: + return 'invalid' + if not token.verify_api_key(secret): + return 'invalid' + return token + + +class ApiTokenCsrfMiddleware: + """Authenticate admin API tokens and exempt only those requests from CSRF. + + Must run immediately *before* ``CsrfViewMiddleware``. The companion + ``ApiTokenUserMiddleware`` assigns ``request.user`` afterwards, because + ``AuthenticationMiddleware`` runs after CSRF and would overwrite anything + set here. + + ``_dont_enforce_csrf_checks`` is set only once the token has verified, so a + forged or absent token cannot be used to switch CSRF off. + """ + + def __init__(self, get_response): + self.get_response = get_response + + def __call__(self, request): + try: + token = _resolve_api_token(request) + except Exception: + # DB not ready (pre-migration) — behave as if no token was offered. + logger.exception("API token resolution failed") + token = None + + if token == 'invalid': + return JsonResponse( + {'success': False, 'error': 'Invalid or expired API token'}, + status=401, + ) + + if token is not None: + request._api_token = token + request._dont_enforce_csrf_checks = True + + return self.get_response(request) + + +class ApiTokenUserMiddleware: + """Act as the token's owner. Runs just after ``AuthenticationMiddleware``.""" + + def __init__(self, get_response): + self.get_response = get_response + + def __call__(self, request): + token = getattr(request, '_api_token', None) + if token is not None: + request.user = token.user + # .update() rather than .save() so the write never round-trips + # through the hashing path in ApiKey.save(). + type(token).objects.filter(pk=token.pk).update( + last_used_at=timezone.now() + ) + return self.get_response(request) + + class NoAuthMiddleware: """ Sandbox middleware: auto-authenticates every request as the first active user. diff --git a/src/logstashui/Documentation/views.py b/src/logstashui/Documentation/views.py index f8f4d99..eceadaf 100644 --- a/src/logstashui/Documentation/views.py +++ b/src/logstashui/Documentation/views.py @@ -24,6 +24,7 @@ 'logstashagent.yml': 'logstashagent.yml', 'logstashui.yml': 'logstashui.yml', 'SNMP': 'SNMP', + 'api_access': 'API Access', 'tsds_implementation': 'TSDS Implementation', 'data_overview': 'Data Overview', 'pipeline_generation': 'Pipeline Generation', diff --git a/src/logstashui/LogstashUI/settings.py b/src/logstashui/LogstashUI/settings.py index e4a850a..8dac2b2 100644 --- a/src/logstashui/LogstashUI/settings.py +++ b/src/logstashui/LogstashUI/settings.py @@ -110,8 +110,14 @@ def _get_version(): 'whitenoise.middleware.WhiteNoiseMiddleware', 'django.contrib.sessions.middleware.SessionMiddleware', 'django.middleware.common.CommonMiddleware', + # Must precede CsrfViewMiddleware: it exempts verified API-token requests + # from CSRF. Browser and agent traffic pass through untouched. + 'Common.middleware.ApiTokenCsrfMiddleware', 'django.middleware.csrf.CsrfViewMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', + # Must follow AuthenticationMiddleware, which would otherwise overwrite + # request.user with the lazy session user. + 'Common.middleware.ApiTokenUserMiddleware', 'django.contrib.messages.middleware.MessageMiddleware', 'django.middleware.clickjacking.XFrameOptionsMiddleware', 'django_htmx.middleware.HtmxMiddleware', diff --git a/src/logstashui/Management/templates/api_tokens.html b/src/logstashui/Management/templates/api_tokens.html new file mode 100644 index 0000000..831ecc6 --- /dev/null +++ b/src/logstashui/Management/templates/api_tokens.html @@ -0,0 +1,153 @@ + + +{% extends "base.html" %} + +{% block content %} + +
+
+

API Tokens

+ +
+

+ Authenticate scripted requests with Authorization: ApiKey <token>. + A token acts as the user who created it and inherits that user's role. +

+ +
+ + + + + + + + + + + + + + {% for token in tokens %} + {% include 'components/api_token_row.html' %} + {% empty %} + + + + {% endfor %} + +
NameTokenOwnerStatusLast UsedExpiresActions
+ No API tokens yet. +
+
+
+ + + + + +{% endblock %} diff --git a/src/logstashui/Management/templates/components/api_token_created.html b/src/logstashui/Management/templates/components/api_token_created.html new file mode 100644 index 0000000..0cc3da1 --- /dev/null +++ b/src/logstashui/Management/templates/components/api_token_created.html @@ -0,0 +1,51 @@ + + +
+
+ Token {{ token.name }} created. Copy it now — it cannot be shown again. +
+ +
+
+ Token + +
+ {{ raw_token }} +
+ +
+ Example + 
curl -X POST \
+  -H "Authorization: ApiKey {{ raw_token }}" \
+  -d "connection_type=CENTRALIZED&name=my-cluster&host=https://es.example.com&port=443&api_key=<es-api-key>" \
+  https://localhost:8443/ConnectionManager/AddConnection
+
+ +
+ +
+
+ + diff --git a/src/logstashui/Management/templates/components/api_token_row.html b/src/logstashui/Management/templates/components/api_token_row.html new file mode 100644 index 0000000..c2b8854 --- /dev/null +++ b/src/logstashui/Management/templates/components/api_token_row.html @@ -0,0 +1,84 @@ + + + + + {{ token.name|default:"—" }} + + + {{ token.masked }} + + + {{ token.user.username }} + {% if token.user.profile.role != 'admin' %} + + readonly + + {% endif %} + + + {% if token.revoked_at %} + + Revoked + + {% elif token.is_expired %} + + Expired + + {% else %} + + Active + + {% endif %} + + + {{ token.last_used_at|date:"Y-m-d H:i"|default:"never" }} + + + {{ token.expires_at|date:"Y-m-d"|default:"never" }} + + +
+ + +
+ + diff --git a/src/logstashui/Management/templates/management.html b/src/logstashui/Management/templates/management.html index f120024..8dd8bda 100644 --- a/src/logstashui/Management/templates/management.html +++ b/src/logstashui/Management/templates/management.html @@ -36,6 +36,19 @@

Logs

+ + +
+
+ + + +
+

API Tokens

+

Issue tokens for scripted access to LogstashUI

+
+ +
diff --git a/src/logstashui/Management/urls.py b/src/logstashui/Management/urls.py index a147fbb..464da7b 100644 --- a/src/logstashui/Management/urls.py +++ b/src/logstashui/Management/urls.py @@ -15,6 +15,7 @@ path("Logs/", views.Logs, name="Logs"), path("Logs/filter", views.LogsFilter, name="LogsFilter"), path("Logs/download", views.LogsDownload, name="LogsDownload"), + path("ApiTokens/", views.ApiTokens, name="ApiTokens"), path("Settings/", views.SettingsView, name="Settings"), path("Settings/Tls/", views.SettingsTlsUpload, name="SettingsTlsUpload"), path("Settings/Tls/Revert/", views.SettingsTlsRevert, name="SettingsTlsRevert"), diff --git a/src/logstashui/Management/views.py b/src/logstashui/Management/views.py index 40506d0..2575091 100644 --- a/src/logstashui/Management/views.py +++ b/src/logstashui/Management/views.py @@ -13,8 +13,10 @@ from django.template.loader import render_to_string from django.conf import settings from django.db import transaction +from django.utils import timezone from .models import UserProfile, Settings from django.http import JsonResponse +from datetime import timedelta import logging import json import os @@ -457,4 +459,107 @@ def SettingsTlsRevert(request): }) except Exception as e: logger.error(f"Error reverting TLS certificate: {e}", exc_info=True) - return JsonResponse({'success': False, 'message': f'Error: {e}'}, status=500) \ No newline at end of file + return JsonResponse({'success': False, 'message': f'Error: {e}'}, status=500) + + +# --------------------------------------------------------------------------- +# API tokens +# --------------------------------------------------------------------------- + +def _token_error(message): + return HttpResponse( + '
{escape(message)}
' + ) + + +def _generate_token_table_rows(tokens, request): + """Render the token table body, reused for htmx swaps after revoke/delete.""" + rows_html = '' + for token in tokens: + rows_html += render_to_string('components/api_token_row.html', { + 'token': token, + 'csrf_token': request.META.get('CSRF_COOKIE', ''), + }, request=request) + return rows_html + + +@require_admin_role +def ApiTokens(request): + """Mint, list, revoke and delete admin API tokens. + + A token acts as its owning user, so the caller's own account is the owner — + that keeps audit lines like "User 'x' added connection" meaningful, and + means a readonly user's token is readonly. + """ + from PipelineManager.models import ApiKey + + def _all_tokens(): + return ( + ApiKey.objects.filter(user__isnull=False) + .select_related('user') + .order_by('-created_at', '-id') + ) + + if request.method == 'POST': + action = request.POST.get('action') + + if action == 'create': + name = (request.POST.get('name') or '').strip() + if not name: + return _token_error('A token name is required.') + if len(name) > 100: + return _token_error('Token name must be 100 characters or fewer.') + + expires_at = None + raw_days = (request.POST.get('expires_days') or '').strip() + if raw_days: + try: + days = int(raw_days) + except ValueError: + return _token_error('Expiry must be a whole number of days.') + if days < 1: + return _token_error('Expiry must be at least 1 day.') + expires_at = timezone.now() + timedelta(days=days) + + token, raw = ApiKey.issue_for_user( + request.user, name=name, expires_at=expires_at + ) + # Deliberately not logged — this is the only time the secret exists. + logger.info( + f"User '{request.user.username}' created API token '{name}' " + f"(prefix {token.prefix})" + ) + return render(request, 'components/api_token_created.html', { + 'token': token, + 'raw_token': raw, + }) + + if action in ('revoke', 'delete'): + token_id = request.POST.get('token_id') + token = ApiKey.objects.filter( + id=token_id, user__isnull=False + ).first() + if token is None: + return _token_error('Token not found.') + + if action == 'revoke': + if token.revoked_at is None: + token.revoked_at = timezone.now() + token.save() + logger.warning( + f"User '{request.user.username}' revoked API token " + f"'{token.name}' (prefix {token.prefix})" + ) + else: + logger.warning( + f"User '{request.user.username}' deleted API token " + f"'{token.name}' (prefix {token.prefix})" + ) + token.delete() + + return HttpResponse(_generate_token_table_rows(_all_tokens(), request)) + + return _token_error('Unknown action.') + + return render(request, 'api_tokens.html', {'tokens': _all_tokens()}) \ No newline at end of file diff --git a/src/logstashui/PipelineManager/migrations/0028_apikey_admin_tokens.py b/src/logstashui/PipelineManager/migrations/0028_apikey_admin_tokens.py new file mode 100644 index 0000000..b3a8047 --- /dev/null +++ b/src/logstashui/PipelineManager/migrations/0028_apikey_admin_tokens.py @@ -0,0 +1,80 @@ +# Generated by Django 6.0.3 on 2026-09-03 21:00 + +import django.db.models.deletion +from django.conf import settings +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('PipelineManager', '0027_packaged_managed_default_ports'), + migrations.swappable_dependency(settings.AUTH_USER_MODEL), + ] + + operations = [ + migrations.AddField( + model_name='apikey', + name='created_at', + field=models.DateTimeField(auto_now_add=True, null=True), + ), + migrations.AddField( + model_name='apikey', + name='expires_at', + field=models.DateTimeField(blank=True, null=True), + ), + migrations.AddField( + model_name='apikey', + name='last_used_at', + field=models.DateTimeField(blank=True, null=True), + ), + migrations.AddField( + model_name='apikey', + name='name', + field=models.CharField(blank=True, default='', help_text='Human-readable label for this token', max_length=100), + ), + migrations.AddField( + model_name='apikey', + name='prefix', + field=models.CharField(blank=True, db_index=True, help_text='Unhashed lookup key for admin tokens; null for agent keys', max_length=12, null=True, unique=True), + ), + migrations.AddField( + model_name='apikey', + name='revoked_at', + field=models.DateTimeField(blank=True, null=True), + ), + migrations.AddField( + model_name='apikey', + name='user', + field=models.ForeignKey(blank=True, help_text='User this API token acts as (admin tokens only)', null=True, on_delete=django.db.models.deletion.CASCADE, related_name='api_tokens', to=settings.AUTH_USER_MODEL), + ), + migrations.AlterField( + model_name='apikey', + name='connection', + field=models.ForeignKey(blank=True, help_text='Connection this API key belongs to (agent keys only)', null=True, on_delete=django.db.models.deletion.CASCADE, related_name='api_keys', to='PipelineManager.connection'), + ), + # The four AlterFields below are unrelated pre-existing drift: help_text + # was edited in 638acfb without regenerating a migration. They emit no + # SQL, and absorbing them here stops the drift trailing into the next + # unrelated migration. + migrations.AlterField( + model_name='connection', + name='agent_api_port', + field=models.PositiveIntegerField(blank=True, help_text='Agent FastAPI port for this connection (packaged 9550; managed 9550+N; simulate 9500+N; embedded 9500)', null=True), + ), + migrations.AlterField( + model_name='connection', + name='logstash_api_port', + field=models.PositiveIntegerField(blank=True, help_text='Logstash HTTP API port for this connection (packaged 9600; managed 9700+N; simulate 9560+N; embedded 9560)', null=True), + ), + migrations.AlterField( + model_name='policy', + name='agent_api_port', + field=models.PositiveIntegerField(default=9500, help_text='Policy default agent FastAPI port (Packaged 9550 as-is; Managed 9550+N; Simulate/Embedded 9500+N or 9500)'), + ), + migrations.AlterField( + model_name='policy', + name='logstash_api_port', + field=models.PositiveIntegerField(default=9560, help_text='Policy default Logstash HTTP API port (Packaged 9600 as-is; Managed 9700+N; Simulate/Embedded 9560+N or 9560)'), + ), + ] diff --git a/src/logstashui/PipelineManager/models.py b/src/logstashui/PipelineManager/models.py index 90610eb..96a1017 100644 --- a/src/logstashui/PipelineManager/models.py +++ b/src/logstashui/PipelineManager/models.py @@ -2,11 +2,14 @@ #or more contributor license agreements. Licensed under the Elastic License; #you may not use this file except in compliance with the Elastic License. +from django.conf import settings from django.db import models from Common.encryption import encrypt_credential, decrypt_credential from django.core.exceptions import ValidationError -from django.contrib.auth.hashers import make_password, check_password +from django.contrib.auth.hashers import make_password, check_password, identify_hasher +from django.utils import timezone import hashlib +import secrets from Common import logstash_config_parse @@ -701,35 +704,150 @@ def __str__(self): return f"{self.policy.name} - {self.name}" +#: Namespace marker on admin API tokens. Agent keys carry no prefix, so the +#: token middleware can tell the two apart from the header alone. +API_TOKEN_SCHEME = 'lsui' + + class ApiKey(models.Model): """ - Represents an API key used by an enrolled agent for authenticated polling/check-in. - Each API key belongs to a specific connection. + A hashed bearer credential. Two flavours share this table: + + * **Agent keys** — issued at enrollment, scoped to a ``connection``. The + agent sends ``connection_id`` in the request body, so the row is found + before the hash is ever checked and no lookup column is needed. These + rows have ``prefix=None``. + * **Admin API tokens** — issued from Management, scoped to a ``user``, and + presented as ``Authorization: ApiKey lsui__``. Here the + header is the *only* identifier, so ``prefix`` is stored unhashed and + indexed; without it, resolving a token would mean a PBKDF2 comparison + against every row in the table on every request. + + Exactly one of ``connection`` / ``user`` is set. """ connection = models.ForeignKey( Connection, on_delete=models.CASCADE, related_name='api_keys', - help_text="Connection this API key belongs to" + null=True, + blank=True, + help_text="Connection this API key belongs to (agent keys only)" + ) + user = models.ForeignKey( + settings.AUTH_USER_MODEL, + on_delete=models.CASCADE, + related_name='api_tokens', + null=True, + blank=True, + help_text="User this API token acts as (admin tokens only)" + ) + name = models.CharField( + max_length=100, + blank=True, + default='', + help_text="Human-readable label for this token" + ) + prefix = models.CharField( + max_length=12, + null=True, + blank=True, + unique=True, + db_index=True, + help_text="Unhashed lookup key for admin tokens; null for agent keys" ) api_key = models.CharField( max_length=512, help_text="Hashed API key for agent authentication" ) - + created_at = models.DateTimeField(null=True, blank=True, auto_now_add=True) + last_used_at = models.DateTimeField(null=True, blank=True) + revoked_at = models.DateTimeField(null=True, blank=True) + expires_at = models.DateTimeField(null=True, blank=True) + class Meta: verbose_name = 'API Key' verbose_name_plural = 'API Keys' - + def __str__(self): - return f"{self.connection.name} - API Key" - + if self.connection_id: + return f"{self.connection.name} - API Key" + return f"{self.name or 'unnamed'} - API Token" + + def clean(self): + # Not a DB CheckConstraint: constraint enforcement is uneven across the + # MySQL 8.0 floor, and this is only ever violated by our own code. + if bool(self.connection_id) == bool(self.user_id): + raise ValidationError( + "An ApiKey must belong to exactly one of connection or user." + ) + def save(self, *args, **kwargs): - # Always hash the API key before saving - if self.api_key: + # Hash on the way in, but only once. Renaming or revoking a token + # re-saves the row, and re-running make_password on a stored hash + # would silently invalidate the credential. + if self.api_key and not self._is_hashed(self.api_key): self.api_key = make_password(self.api_key) super().save(*args, **kwargs) - + + @staticmethod + def _is_hashed(value): + try: + identify_hasher(value) + except ValueError: + return False + return True + def verify_api_key(self, raw_api_key): """Verify a raw API key against the stored hash""" return check_password(raw_api_key, self.api_key) + + # -- admin API tokens --------------------------------------------------- + + @classmethod + def issue_for_user(cls, user, name='', expires_at=None): + """Mint an admin API token. Returns ``(instance, raw_token)``. + + The raw token is the only time the secret exists in plaintext — it is + not recoverable afterwards. + """ + # token_hex, not token_urlsafe: the prefix must contain no '_' so that + # split('_', 2) on the wire format is unambiguous. + prefix = secrets.token_hex(6) + secret = secrets.token_urlsafe(32) + token = cls( + user=user, + name=name, + prefix=prefix, + api_key=secret, + expires_at=expires_at, + ) + token.full_clean(exclude=['api_key'], validate_unique=False) + token.save() + return token, f"{API_TOKEN_SCHEME}_{prefix}_{secret}" + + @staticmethod + def parse_token(raw): + """Split a wire-format token into ``(prefix, secret)``. + + Returns ``(None, None)`` for anything that is not an admin token, + including agent keys, which carry no scheme marker. + """ + parts = (raw or '').split('_', 2) + if len(parts) != 3 or parts[0] != API_TOKEN_SCHEME: + return None, None + if not parts[1] or not parts[2]: + return None, None + return parts[1], parts[2] + + @property + def masked(self): + """Display form for the token list — prefix only, never the secret.""" + return f"{API_TOKEN_SCHEME}_{self.prefix}_…" if self.prefix else '' + + @property + def is_expired(self): + return self.expires_at is not None and self.expires_at <= timezone.now() + + @property + def is_active(self): + return self.revoked_at is None and not self.is_expired diff --git a/tests/Common/unit/test_api_token_middleware.py b/tests/Common/unit/test_api_token_middleware.py new file mode 100644 index 0000000..6ab3bbc --- /dev/null +++ b/tests/Common/unit/test_api_token_middleware.py @@ -0,0 +1,263 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +"""API token authentication middleware. + +The endpoint under test is ``/ConnectionManager/AddConnection`` because it +exercises all three gates a scripted caller has to clear: CsrfViewMiddleware, +LoginRequiredMiddleware, and ``require_admin_role``. +""" + +from datetime import timedelta + +from django.contrib.auth.models import User +from django.test import Client +from django.utils import timezone + +import pytest + +from Management.models import UserProfile +from PipelineManager.models import ApiKey + + +ADD_CONNECTION = '/ConnectionManager/AddConnection' +FORM_CT = 'application/x-www-form-urlencoded' +BODY = ( + 'connection_type=CENTRALIZED&name=api-created' + '&host=https://es.invalid&port=443&api_key=abc' +) + + +@pytest.fixture +def csrf_client(): + """A client that enforces CSRF, so exemption is actually observable.""" + return Client(enforce_csrf_checks=True) + + +@pytest.fixture +def admin_token(test_user): + return ApiKey.issue_for_user(test_user, name='ci') + + +@pytest.fixture +def readonly_token(db): + user = User.objects.create_user(username='ro', password='ropass123') + UserProfile.objects.update_or_create(user=user, defaults={'role': 'readonly'}) + return ApiKey.issue_for_user(user, name='ro-token') + + +def _post(client, raw=None): + kwargs = {} + if raw is not None: + kwargs['HTTP_AUTHORIZATION'] = f'ApiKey {raw}' + return client.post(ADD_CONNECTION, BODY, content_type=FORM_CT, **kwargs) + + +@pytest.mark.django_db +class TestTokenAccepted: + def test_valid_token_reaches_the_view(self, csrf_client, admin_token, monkeypatch): + """No CSRF token, no session — the request still reaches AddConnection.""" + monkeypatch.setattr( + 'PipelineManager.manager_views.test_connectivity', + lambda cid: (True, 'ok'), + ) + _token, raw = admin_token + + response = _post(csrf_client, raw) + + assert response.status_code == 200 + payload = response.json() + assert payload['success'] is True + assert payload['connection_id'] + + def test_token_acts_as_its_owner(self, request_factory, admin_token): + """Both halves of the middleware pair, driven directly. + + The split exists because AuthenticationMiddleware sits between them and + would overwrite request.user, so assert the two effects separately. + """ + from django.contrib.auth.models import AnonymousUser + from Common.middleware import ( + ApiTokenCsrfMiddleware, ApiTokenUserMiddleware, + ) + + _token, raw = admin_token + request = request_factory.post( + ADD_CONNECTION, BODY, content_type=FORM_CT, + HTTP_AUTHORIZATION=f'ApiKey {raw}', + ) + request.user = AnonymousUser() + + ApiTokenCsrfMiddleware(lambda r: None)(request) + assert request._dont_enforce_csrf_checks is True + + ApiTokenUserMiddleware(lambda r: None)(request) + assert request.user.username == 'testuser' + + def test_absent_token_leaves_csrf_alone(self, request_factory): + from Common.middleware import ApiTokenCsrfMiddleware + + request = request_factory.post(ADD_CONNECTION, BODY, content_type=FORM_CT) + ApiTokenCsrfMiddleware(lambda r: None)(request) + + assert not hasattr(request, '_dont_enforce_csrf_checks') + assert not hasattr(request, '_api_token') + + def test_last_used_at_is_recorded(self, csrf_client, admin_token, monkeypatch): + monkeypatch.setattr( + 'PipelineManager.manager_views.test_connectivity', + lambda cid: (True, 'ok'), + ) + token, raw = admin_token + assert token.last_used_at is None + + _post(csrf_client, raw) + + token.refresh_from_db() + assert token.last_used_at is not None + + +@pytest.mark.django_db +class TestTokenRejected: + def test_unknown_prefix(self, csrf_client): + response = _post(csrf_client, 'lsui_deadbeefcafe_bogus') + assert response.status_code == 401 + assert response.json()['success'] is False + + def test_wrong_secret(self, csrf_client, admin_token): + token, _raw = admin_token + response = _post(csrf_client, f'lsui_{token.prefix}_wrongsecret') + assert response.status_code == 401 + + def test_revoked(self, csrf_client, admin_token): + token, raw = admin_token + token.revoked_at = timezone.now() + token.save() + assert _post(csrf_client, raw).status_code == 401 + + def test_expired(self, csrf_client, admin_token): + token, raw = admin_token + token.expires_at = timezone.now() - timedelta(seconds=1) + token.save() + assert _post(csrf_client, raw).status_code == 401 + + def test_inactive_owner(self, csrf_client, admin_token, test_user): + _token, raw = admin_token + test_user.is_active = False + test_user.save() + assert _post(csrf_client, raw).status_code == 401 + + def test_readonly_owner_gets_json_403(self, csrf_client, readonly_token): + """require_admin_role must answer a script in JSON, not an HX-Trigger toast.""" + _token, raw = readonly_token + response = _post(csrf_client, raw) + assert response.status_code == 403 + assert response.json()['success'] is False + assert 'HX-Trigger' not in response + + +@pytest.mark.django_db +class TestCsrfStillEnforced: + def test_session_post_without_csrf_token_is_rejected(self, test_user): + """The regression guard: a logged-in browser POST still needs CSRF. + + A token must not be the thing that switches CSRF off for everyone. + """ + client = Client(enforce_csrf_checks=True) + client.login(username='testuser', password='testpass123') + + response = client.post(ADD_CONNECTION, BODY, content_type=FORM_CT) + + assert response.status_code == 403 + + def test_forged_header_does_not_disable_csrf(self, test_user): + """An invalid token is refused outright, never silently CSRF-exempted.""" + client = Client(enforce_csrf_checks=True) + client.login(username='testuser', password='testpass123') + + response = client.post( + ADD_CONNECTION, BODY, content_type=FORM_CT, + HTTP_AUTHORIZATION='ApiKey lsui_deadbeefcafe_bogus', + ) + + assert response.status_code == 401 + + def test_anonymous_post_is_rejected(self, csrf_client): + """Redirected to login by LoginRequiredMiddleware, which runs before the + CSRF check in process_view.""" + response = _post(csrf_client) + assert response.status_code in (302, 403) + + +@pytest.mark.django_db +class TestAgentKeysUnaffected: + def test_agent_key_header_passes_through(self, csrf_client): + """Agent keys carry no lsui_ marker; the middleware must ignore them and + leave authentication to the agent views.""" + response = csrf_client.post( + '/ConnectionManager/CheckIn/', '{}', + content_type='application/json', + HTTP_AUTHORIZATION='ApiKey some-agent-key-without-prefix', + ) + # 400 from check_in's own missing-connection_id path, not 401 from us. + assert response.status_code == 400 + + def test_non_apikey_scheme_ignored(self, csrf_client, test_user): + client = Client(enforce_csrf_checks=True) + client.login(username='testuser', password='testpass123') + response = client.post( + ADD_CONNECTION, BODY, content_type=FORM_CT, + HTTP_AUTHORIZATION='Bearer lsui_deadbeefcafe_bogus', + ) + # Falls through to normal session handling -> CSRF rejection. + assert response.status_code == 403 + + +@pytest.mark.django_db +class TestApiKeyModel: + def test_issue_returns_parseable_token(self, test_user): + token, raw = ApiKey.issue_for_user(test_user, name='x') + prefix, secret = ApiKey.parse_token(raw) + assert prefix == token.prefix + assert token.verify_api_key(secret) + + def test_prefix_has_no_underscore(self, test_user): + """split('_', 2) on the wire format depends on this.""" + token, _raw = ApiKey.issue_for_user(test_user, name='x') + assert '_' not in token.prefix + + def test_resaving_does_not_double_hash(self, test_user): + """Rename and revoke both re-save the row; the secret must survive.""" + token, raw = ApiKey.issue_for_user(test_user, name='x') + _prefix, secret = ApiKey.parse_token(raw) + + token.name = 'renamed' + token.save() + token.revoked_at = timezone.now() + token.save() + + token.refresh_from_db() + assert token.verify_api_key(secret) + + def test_agent_key_still_hashes_on_create(self, test_connection): + raw = 'agent-raw-key' + key = ApiKey.objects.create(connection=test_connection, api_key=raw) + assert key.api_key != raw + assert key.verify_api_key(raw) + + @pytest.mark.parametrize('raw', [ + '', 'nope', 'lsui_only', 'other_prefix_secret', 'lsui__secret', 'lsui_prefix_', + ]) + def test_parse_rejects_non_tokens(self, raw): + assert ApiKey.parse_token(raw) == (None, None) + + def test_clean_requires_exactly_one_owner(self, test_user, test_connection): + from django.core.exceptions import ValidationError + + with pytest.raises(ValidationError): + ApiKey(api_key='x').clean() + with pytest.raises(ValidationError): + ApiKey( + api_key='x', user=test_user, connection=test_connection + ).clean() diff --git a/tests/Management/unit/test_api_tokens.py b/tests/Management/unit/test_api_tokens.py new file mode 100644 index 0000000..68b85a0 --- /dev/null +++ b/tests/Management/unit/test_api_tokens.py @@ -0,0 +1,166 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +"""Management -> API Tokens page.""" + +from django.contrib.auth.models import User + +import pytest + +from Management.models import UserProfile +from PipelineManager.models import ApiKey + + +URL = '/Management/ApiTokens/' + + +@pytest.fixture +def readonly_client(client, db): + user = User.objects.create_user(username='ro', password='ropass123') + UserProfile.objects.update_or_create(user=user, defaults={'role': 'readonly'}) + client.login(username='ro', password='ropass123') + return client + + +@pytest.mark.django_db +class TestCreate: + def test_create_shows_raw_token_once(self, authenticated_client): + response = authenticated_client.post( + URL, {'action': 'create', 'name': 'ci'} + ) + + assert response.status_code == 200 + token = ApiKey.objects.get(name='ci') + body = response.content.decode() + assert f'lsui_{token.prefix}_' in body + # The stored value is a hash, so the page is the only source of the raw + # secret — and the list view must never render it. + assert token.api_key not in body + + def test_created_token_is_usable(self, authenticated_client): + import re + + response = authenticated_client.post( + URL, {'action': 'create', 'name': 'ci'} + ) + raw = re.search(r'lsui_[0-9a-f]{12}_[A-Za-z0-9_\-]+', + response.content.decode()).group(0) + + prefix, secret = ApiKey.parse_token(raw) + assert ApiKey.objects.get(prefix=prefix).verify_api_key(secret) + + def test_owner_is_the_creator(self, authenticated_client, test_user): + authenticated_client.post(URL, {'action': 'create', 'name': 'ci'}) + assert ApiKey.objects.get(name='ci').user == test_user + + def test_expiry_in_days(self, authenticated_client): + authenticated_client.post( + URL, {'action': 'create', 'name': 'ci', 'expires_days': '30'} + ) + assert ApiKey.objects.get(name='ci').expires_at is not None + + @pytest.mark.parametrize('payload,fragment', [ + ({'action': 'create', 'name': ''}, 'name is required'), + ({'action': 'create', 'name': 'ci', 'expires_days': 'soon'}, 'whole number'), + ({'action': 'create', 'name': 'ci', 'expires_days': '0'}, 'at least 1 day'), + ]) + def test_validation(self, authenticated_client, payload, fragment): + response = authenticated_client.post(URL, payload) + assert fragment in response.content.decode() + assert not ApiKey.objects.exists() + + def test_no_expiry_by_default(self, authenticated_client): + authenticated_client.post(URL, {'action': 'create', 'name': 'ci'}) + assert ApiKey.objects.get(name='ci').expires_at is None + + +@pytest.mark.django_db +class TestRevokeAndDelete: + def test_revoke_sets_timestamp(self, authenticated_client, test_user): + token, _raw = ApiKey.issue_for_user(test_user, name='ci') + + response = authenticated_client.post( + URL, {'action': 'revoke', 'token_id': token.id} + ) + + assert response.status_code == 200 + token.refresh_from_db() + assert token.revoked_at is not None + + def test_revoke_preserves_the_hash(self, authenticated_client, test_user): + """Revoke re-saves the row; without the double-hash guard the stored + secret would be silently rewritten.""" + token, raw = ApiKey.issue_for_user(test_user, name='ci') + stored = token.api_key + + authenticated_client.post(URL, {'action': 'revoke', 'token_id': token.id}) + + token.refresh_from_db() + assert token.api_key == stored + assert token.verify_api_key(ApiKey.parse_token(raw)[1]) + + def test_delete_removes_row(self, authenticated_client, test_user): + token, _raw = ApiKey.issue_for_user(test_user, name='ci') + + authenticated_client.post(URL, {'action': 'delete', 'token_id': token.id}) + + assert not ApiKey.objects.filter(pk=token.pk).exists() + + def test_cannot_touch_agent_keys(self, authenticated_client, test_connection): + """The page manages admin tokens only; agent keys are not addressable.""" + key = ApiKey.objects.create(connection=test_connection, api_key='raw') + + response = authenticated_client.post( + URL, {'action': 'delete', 'token_id': key.id} + ) + + assert 'not found' in response.content.decode() + assert ApiKey.objects.filter(pk=key.pk).exists() + + def test_unknown_action(self, authenticated_client): + response = authenticated_client.post(URL, {'action': 'nope'}) + assert 'Unknown action' in response.content.decode() + + +@pytest.mark.django_db +class TestListing: + def test_lists_only_admin_tokens(self, authenticated_client, test_user, + test_connection): + ApiKey.issue_for_user(test_user, name='mine') + ApiKey.objects.create(connection=test_connection, api_key='agent-raw') + + response = authenticated_client.get(URL) + + body = response.content.decode() + assert response.status_code == 200 + assert 'mine' in body + assert 'agent-raw' not in body + + def test_empty_state(self, authenticated_client): + response = authenticated_client.get(URL) + assert 'No API tokens yet' in response.content.decode() + + def test_secret_never_rendered_in_list(self, authenticated_client, test_user): + token, raw = ApiKey.issue_for_user(test_user, name='mine') + _prefix, secret = ApiKey.parse_token(raw) + + body = authenticated_client.get(URL).content.decode() + + assert secret not in body + assert token.api_key not in body + + +@pytest.mark.django_db +class TestReadonlyUserBlocked: + def test_get_denied(self, readonly_client): + assert readonly_client.get(URL).status_code == 403 + + def test_create_denied(self, readonly_client): + response = readonly_client.post(URL, {'action': 'create', 'name': 'x'}) + assert response.status_code == 403 + assert not ApiKey.objects.exists() + + def test_anonymous_redirected(self, client): + response = client.get(URL) + assert response.status_code in (302, 403) From 51c9e0acfef7d14b85de499f6487dca689715961 Mon Sep 17 00:00:00 2001 From: Aaron Mildenstein Date: Thu, 3 Sep 2026 20:48:56 -0600 Subject: [PATCH 33/62] Massive collection of repairs and new functionality MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - **The LS pill now tracks the Logstash version actually running on the host.** It was frozen at whatever version happened to be recorded first, for two reasons. `resolve_running_logstash_version()` consulted the `Connection.logstash_version_resolved` column *before* the current check-in's `status_blob`, and that column is only ever written on a truthy value and never cleared — so one stored version permanently shadowed every later one. The check-in handler also never read `status_blob.logstash_api.version`, the version the running instance reports through its own API, so on hosts that report it only there the column was never refreshed at all. The blob now leads and the column is the fallback, which keeps the last known version on screen while Logstash is stopped or its API is unreachable. - The pill also updates without a page reload. The Connections page adds `logstash_version` to the existing agent-status SSE payload — free, since the stream already selects `status_blob` — and the Policies → Agents table, which had no live channel at all, polls every 10s while that tab is visible and pauses when the browser tab is hidden. - **LogstashUI can now cache Logstash release tarballs and serve them to agents.** A `MANAGED` or `SIMULATE` policy pinned to **VERSION** previously made every agent pull its own ~450 MB tarball from `artifacts.elastic.co`. Tick **Download the tarball from LogstashUI** on the policy and LogstashUI fetches each release once, verifies its SHA-512, and serves it to every agent. Required for air-gapped sites, where the direct download is impossible. See [Logstash Tarball Proxy](docs/docs/logstashui/configuration/logstash_proxy.md). - **Compatibility — the agent artifact URL gained a `connection_id` segment.** `GET /ConnectionManager/LogstashArtifact/{filename}` is now `GET /ConnectionManager/LogstashArtifact/{connection_id}/{filename}`. A GET has no body, and an agent key is a bare hash with no lookup column, so the header alone cannot identify the caller — the path is what narrows the lookup to one row before `check_password` runs. Agents older than the paired release cannot use the proxy; the boolean simply stays off for them and they continue downloading from Elastic. - The proxy is exposed to agents in three places: `logstash_via_ui` in the enrollment `policy_config`, `logstash_via_ui` in the check-in response, and `logstash_runtime.via_ui` in the config delta. `via_ui` participates in the `runtime_changed` comparison, so flipping the checkbox alone triggers a re-materialize — no separate Deploy for binary-only changes. - **Management → Logstash Tarballs** manages the cache: download by version + architecture (or from a full URL for an internal mirror), delete, retry, and **Import from disk** for air-gapped operators who copy a tarball into the cache directory by hand. Import hashes the file, checks a supplied `.sha512` and fails on mismatch, and writes one when none was supplied. There is no browser upload — 450 MB through a form is not viable. Progress polls over htmx only while a row is actively fetching. - **Single-flight is a conditional database `UPDATE`, not an in-process lock.** `settings.py` defines no `CACHES`, so Django falls back to per-process `LocMemCache`, which does not coordinate across gunicorn workers. The first agent to ask claims the row and gets a `503`; everyone else gets a `503` until the file lands, across all workers. A fetch killed by a restart leaves a stale claim that the next request reclaims after the heartbeat window, and the orphaned `.part` is swept at startup — crash recovery is the normal path, not an edge case, because a 450 MB fetch will always be killed by `graceful_timeout`. - Nothing partial is ever served: downloads land in a `.part` file and are `os.replace()`d into place only after the SHA-512 verifies. Agents get `200`/`206` when the file is ready, `503` while it is being fetched, `429` at the concurrent-serve cap, `502` on an upstream failure, `404` for an unrecognized filename, and `401` for a bad key or mismatched `connection_id`. `503`/`429`/`502` all carry `Retry-After`; agents honour it, backing off to a 5-minute ceiling, and never fall back to `artifacts.elastic.co` while the proxy is enabled. - Range requests are supported (single range only) so an interrupted transfer resumes instead of re-pulling the whole tarball. Files under 1 MiB — the `.sha512` companions — are exempt from the serve semaphore, so a burst of checksum fetches cannot consume download slots. - New env vars: `LOGSTASHUI_LOGSTASH_DIR` (default `/logstashes`), `LOGSTASHUI_ARTIFACT_MAX_UPSTREAM` (default `2`, cluster-wide), and `LOGSTASHUI_ARTIFACT_MAX_SERVE_PER_WORKER` (default `4`; effective total is this × `LOGSTASHUI_WORKERS`). The serve cap is deliberately per-worker — dividing a global limit by the worker count truncates to zero and makes the knob lie. The upstream base URL is **Management → Settings → Logstash tarball source**, blank meaning `https://artifacts.elastic.co/downloads/logstash`. - Tarballs are stored in `/logstashes`, **not** under `staticfiles/` — `STATIC_ROOT` is served by WhiteNoise at `/static/`, which is in `LOGIN_REQUIRED_IGNORE_PATHS`, so every tarball would have been an unauthenticated public download, and `collectstatic` would churn over them on every `serve`. - **Optional OpenTelemetry export** (`LOGSTASHUI_OTEL=true` plus the new `LogstashUI[otel]` extra, included in the offline wheel archive) sends traces and metrics over OTLP/HTTP. Four custom instruments answer the capacity question Django spans cannot: `logstashui.gevent.hub.lag`, `logstashui.artifact.downloads.active`, `logstashui.artifact.requests`, and `logstashui.artifact.serve.bytes_per_second`. Only the HTTP/protobuf exporter is supported — the gRPC exporter's native threads are not gevent-patchable. - Migrations: `PipelineManager/0029_logstash_artifacts` (the `LogstashArtifact` model and `Policy.logstash_via_ui`) and `Management/0004_settings_logstash_artifact_base_url`. - New tests: `tests/PipelineManager/unit/test_logstash_artifacts.py` and `tests/Management/unit/test_logstash_artifact_page.py`. --- CHANGELOG.md | 17 + bin/freeze_logstashui.sh | 10 +- docs/docs/logstashagent/general/roles.md | 4 +- .../logstashui/configuration/host_mode.md | 4 +- docs/docs/logstashui/configuration/index.md | 7 + .../configuration/logstash_proxy.md | 137 +++ .../logstashui/configuration/simulation.md | 2 +- pyproject.toml | 8 + src/logstashui/Documentation/views.py | 1 + src/logstashui/LogstashUI/paths.py | 19 +- src/logstashui/LogstashUI/settings.py | 29 +- src/logstashui/LogstashUI/telemetry.py | 141 +++ src/logstashui/LogstashUI/wsgi.py | 19 + ...004_settings_logstash_artifact_base_url.py | 29 + src/logstashui/Management/models.py | 10 + .../components/logstash_artifact_row.html | 100 ++ .../components/logstash_artifact_tbody.html | 26 + .../templates/logstash_artifacts.html | 172 ++++ .../Management/templates/management.html | 13 + .../templates/management_settings.html | 26 +- src/logstashui/Management/urls.py | 1 + src/logstashui/Management/views.py | 164 +++- src/logstashui/PipelineManager/agent_api.py | 27 +- src/logstashui/PipelineManager/agent_modes.py | 23 + .../PipelineManager/agent_versions.py | 33 +- .../PipelineManager/artifact_metrics.py | 134 +++ src/logstashui/PipelineManager/artifacts.py | 745 +++++++++++++++ .../PipelineManager/manager_views.py | 24 +- .../migrations/0029_logstash_artifacts.py | 45 + src/logstashui/PipelineManager/models.py | 203 ++++ .../PipelineManager/policies_crud.py | 7 + .../static/js/agent_policies.js | 74 +- .../static/js/agent_status_sse.js | 29 +- .../pipeline_manager/agent_policies.html | 13 + .../templates/pipeline_manager.html | 8 +- src/logstashui/PipelineManager/urls.py | 10 +- .../unit/test_logstash_artifact_page.py | 297 ++++++ tests/PipelineManager/unit/test_agent_api.py | 65 ++ .../unit/test_agent_versions.py | 45 +- .../unit/test_logstash_artifacts.py | 891 ++++++++++++++++++ .../unit/test_manager_views.py | 71 ++ uv.lock | 192 +++- 42 files changed, 3820 insertions(+), 55 deletions(-) create mode 100644 docs/docs/logstashui/configuration/logstash_proxy.md create mode 100644 src/logstashui/LogstashUI/telemetry.py create mode 100644 src/logstashui/Management/migrations/0004_settings_logstash_artifact_base_url.py create mode 100644 src/logstashui/Management/templates/components/logstash_artifact_row.html create mode 100644 src/logstashui/Management/templates/components/logstash_artifact_tbody.html create mode 100644 src/logstashui/Management/templates/logstash_artifacts.html create mode 100644 src/logstashui/PipelineManager/artifact_metrics.py create mode 100644 src/logstashui/PipelineManager/artifacts.py create mode 100644 src/logstashui/PipelineManager/migrations/0029_logstash_artifacts.py create mode 100644 tests/Management/unit/test_logstash_artifact_page.py create mode 100644 tests/PipelineManager/unit/test_logstash_artifacts.py diff --git a/CHANGELOG.md b/CHANGELOG.md index e3510aa..e38b5cd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -68,6 +68,23 @@ SQLite does not scale under gunicorn/gevent. Operators can now run LogstashUI on - ConnectionManager and Policy editor Agents tab show a cyan **LS X.Y.Z** pill for the Logstash version the agent reported (`logstash_version_resolved`, else Logstash API version). Hidden until known. - Policy Source **VERSION** live-fills and persists Binary Path as `{download_dir}/logstash-{version}/bin` (default `/opt/logstash-agent/logstash-versions/logstash-X.Y.Z/bin`). Custom paths are kept. Switching back to SYSTEM restores `/usr/share/logstash/bin` when the field still looks derived. - Agent newer than preferred (`__PREFERRED_LS_AGENT_VERSION__`, currently 0.5.1) shows **unreleased version** instead of a backwards Upgrade button. Older agents still get Upgrade. Unparseable versions still get Upgrade. +- **The LS pill now tracks the Logstash version actually running on the host.** It was frozen at whatever version happened to be recorded first, for two reasons. `resolve_running_logstash_version()` consulted the `Connection.logstash_version_resolved` column *before* the current check-in's `status_blob`, and that column is only ever written on a truthy value and never cleared — so one stored version permanently shadowed every later one. The check-in handler also never read `status_blob.logstash_api.version`, the version the running instance reports through its own API, so on hosts that report it only there the column was never refreshed at all. The blob now leads and the column is the fallback, which keeps the last known version on screen while Logstash is stopped or its API is unreachable. +- The pill also updates without a page reload. The Connections page adds `logstash_version` to the existing agent-status SSE payload — free, since the stream already selects `status_blob` — and the Policies → Agents table, which had no live channel at all, polls every 10s while that tab is visible and pauses when the browser tab is hidden. + +### Logstash tarball proxy + +- **LogstashUI can now cache Logstash release tarballs and serve them to agents.** A `MANAGED` or `SIMULATE` policy pinned to **VERSION** previously made every agent pull its own ~450 MB tarball from `artifacts.elastic.co`. Tick **Download the tarball from LogstashUI** on the policy and LogstashUI fetches each release once, verifies its SHA-512, and serves it to every agent. Required for air-gapped sites, where the direct download is impossible. See [Logstash Tarball Proxy](docs/docs/logstashui/configuration/logstash_proxy.md). +- **Compatibility — the agent artifact URL gained a `connection_id` segment.** `GET /ConnectionManager/LogstashArtifact/{filename}` is now `GET /ConnectionManager/LogstashArtifact/{connection_id}/{filename}`. A GET has no body, and an agent key is a bare hash with no lookup column, so the header alone cannot identify the caller — the path is what narrows the lookup to one row before `check_password` runs. Agents older than the paired release cannot use the proxy; the boolean simply stays off for them and they continue downloading from Elastic. +- The proxy is exposed to agents in three places: `logstash_via_ui` in the enrollment `policy_config`, `logstash_via_ui` in the check-in response, and `logstash_runtime.via_ui` in the config delta. `via_ui` participates in the `runtime_changed` comparison, so flipping the checkbox alone triggers a re-materialize — no separate Deploy for binary-only changes. +- **Management → Logstash Tarballs** manages the cache: download by version + architecture (or from a full URL for an internal mirror), delete, retry, and **Import from disk** for air-gapped operators who copy a tarball into the cache directory by hand. Import hashes the file, checks a supplied `.sha512` and fails on mismatch, and writes one when none was supplied. There is no browser upload — 450 MB through a form is not viable. Progress polls over htmx only while a row is actively fetching. +- **Single-flight is a conditional database `UPDATE`, not an in-process lock.** `settings.py` defines no `CACHES`, so Django falls back to per-process `LocMemCache`, which does not coordinate across gunicorn workers. The first agent to ask claims the row and gets a `503`; everyone else gets a `503` until the file lands, across all workers. A fetch killed by a restart leaves a stale claim that the next request reclaims after the heartbeat window, and the orphaned `.part` is swept at startup — crash recovery is the normal path, not an edge case, because a 450 MB fetch will always be killed by `graceful_timeout`. +- Nothing partial is ever served: downloads land in a `.part` file and are `os.replace()`d into place only after the SHA-512 verifies. Agents get `200`/`206` when the file is ready, `503` while it is being fetched, `429` at the concurrent-serve cap, `502` on an upstream failure, `404` for an unrecognized filename, and `401` for a bad key or mismatched `connection_id`. `503`/`429`/`502` all carry `Retry-After`; agents honour it, backing off to a 5-minute ceiling, and never fall back to `artifacts.elastic.co` while the proxy is enabled. +- Range requests are supported (single range only) so an interrupted transfer resumes instead of re-pulling the whole tarball. Files under 1 MiB — the `.sha512` companions — are exempt from the serve semaphore, so a burst of checksum fetches cannot consume download slots. +- New env vars: `LOGSTASHUI_LOGSTASH_DIR` (default `/logstashes`), `LOGSTASHUI_ARTIFACT_MAX_UPSTREAM` (default `2`, cluster-wide), and `LOGSTASHUI_ARTIFACT_MAX_SERVE_PER_WORKER` (default `4`; effective total is this × `LOGSTASHUI_WORKERS`). The serve cap is deliberately per-worker — dividing a global limit by the worker count truncates to zero and makes the knob lie. The upstream base URL is **Management → Settings → Logstash tarball source**, blank meaning `https://artifacts.elastic.co/downloads/logstash`. +- Tarballs are stored in `/logstashes`, **not** under `staticfiles/` — `STATIC_ROOT` is served by WhiteNoise at `/static/`, which is in `LOGIN_REQUIRED_IGNORE_PATHS`, so every tarball would have been an unauthenticated public download, and `collectstatic` would churn over them on every `serve`. +- **Optional OpenTelemetry export** (`LOGSTASHUI_OTEL=true` plus the new `LogstashUI[otel]` extra, included in the offline wheel archive) sends traces and metrics over OTLP/HTTP. Four custom instruments answer the capacity question Django spans cannot: `logstashui.gevent.hub.lag`, `logstashui.artifact.downloads.active`, `logstashui.artifact.requests`, and `logstashui.artifact.serve.bytes_per_second`. Only the HTTP/protobuf exporter is supported — the gRPC exporter's native threads are not gevent-patchable. +- Migrations: `PipelineManager/0029_logstash_artifacts` (the `LogstashArtifact` model and `Policy.logstash_via_ui`) and `Management/0004_settings_logstash_artifact_base_url`. +- New tests: `tests/PipelineManager/unit/test_logstash_artifacts.py` and `tests/Management/unit/test_logstash_artifact_page.py`. ## [0.5.1] - Agent control plane, SNMP NMS, dual HTTPS - 08/31/2026 diff --git a/bin/freeze_logstashui.sh b/bin/freeze_logstashui.sh index 5667589..4a31568 100755 --- a/bin/freeze_logstashui.sh +++ b/bin/freeze_logstashui.sh @@ -152,11 +152,13 @@ freeze_wheels() { [[ -n "$whl" && -f "$whl" ]] || die "uv build did not produce dist/logstashui-${VERSION}-*.whl" cp "$whl" "$wheels/" - echo "==> uv export --frozen --extra databases" - (cd "$ROOT" && uv export --frozen --no-dev --extra databases --no-emit-project \ + # otel rides along so an air-gapped site can still enable tracing with + # LOGSTASHUI_OTEL=true. It is inert until then. + echo "==> uv export --frozen --extra databases --extra otel" + (cd "$ROOT" && uv export --frozen --no-dev --extra databases --extra otel --no-emit-project \ -o "$req" >/dev/null) local req_plain="$OUT/requirements-offline.nohash.txt" - (cd "$ROOT" && uv export --frozen --no-dev --extra databases --no-emit-project --no-hashes \ + (cd "$ROOT" && uv export --frozen --no-dev --extra databases --extra otel --no-emit-project --no-hashes \ -o "$req_plain" >/dev/null) echo "==> download manylinux cp312 wheels (pure-python sdists → wheel on builder)" @@ -175,7 +177,7 @@ freeze_wheels() { echo "git ${GIT_SHA}" echo "python CPython 3.12" echo "platform linux-x86_64" - echo "extra databases" + echo "extras databases otel" echo echo "wheels:" (cd "$wheels" && ls -1 *.whl | sort) diff --git a/docs/docs/logstashagent/general/roles.md b/docs/docs/logstashagent/general/roles.md index 19e7935..006f0cf 100644 --- a/docs/docs/logstashagent/general/roles.md +++ b/docs/docs/logstashagent/general/roles.md @@ -140,7 +140,9 @@ Policies can set **Logstash binary source**: 1. Save the policy in LogstashUI (Source = VERSION, pin e.g. `9.4.3`). 2. **No separate Deploy is required for binary-only changes** — the next agent check-in detects runtime drift. -3. Agent downloads (if needed) into `/opt/logstash-agent/logstash-versions//`. +3. Agent downloads (if needed) into `/opt/logstash-agent/logstash-versions//` — from Elastic + artifacts, or from LogstashUI when the policy enables + [the tarball proxy](/docs/docs/logstashui/configuration/logstash_proxy.md). 4. Writes `LOGSTASH_BINARY=` into the instance `env` file. 5. Restarts the Logstash unit when the binary path or pin changes. 6. Reports resolved version on check-in (`status_blob.logstash_version_resolved`). diff --git a/docs/docs/logstashui/configuration/host_mode.md b/docs/docs/logstashui/configuration/host_mode.md index 1a08e23..1d6eb7e 100644 --- a/docs/docs/logstashui/configuration/host_mode.md +++ b/docs/docs/logstashui/configuration/host_mode.md @@ -22,7 +22,9 @@ - Root for `logstash-agent install --enroll …` (or non-root enroll + `sudo logstash-agent setup-simulate`) - Logstash binary available either: - **SYSTEM** — package or tarball already on the host, or - - **VERSION** — agent downloads from Elastic artifacts into `/opt/logstash-agent/logstash-versions/` + - **VERSION** — agent downloads into `/opt/logstash-agent/logstash-versions/`, from Elastic + artifacts or, with [the tarball proxy](/docs/docs/logstashui/configuration/logstash_proxy.md) + enabled, from LogstashUI - Reachable LogstashUI URL from the agent host ## Install a simulate agent diff --git a/docs/docs/logstashui/configuration/index.md b/docs/docs/logstashui/configuration/index.md index db8931e..772add7 100644 --- a/docs/docs/logstashui/configuration/index.md +++ b/docs/docs/logstashui/configuration/index.md @@ -16,6 +16,13 @@ Database engines and `LOGSTASHUI_DB_*`: **[Database](/docs/docs/logstashui/datab `logstashui.yml` is [removed](/docs/docs/logstashui/configuration/logstashui.yml.md). +### **[Logstash tarball proxy](/docs/docs/logstashui/configuration/logstash_proxy.md)** + +Cache each Logstash release once and serve it to every agent, instead of each agent pulling ~450 MB +from Elastic. Required for air-gapped sites. + +**📖 [View the tarball proxy guide →](/docs/docs/logstashui/configuration/logstash_proxy.md)** + --- ## Simulation Configuration diff --git a/docs/docs/logstashui/configuration/logstash_proxy.md b/docs/docs/logstashui/configuration/logstash_proxy.md new file mode 100644 index 0000000..6b97dea --- /dev/null +++ b/docs/docs/logstashui/configuration/logstash_proxy.md @@ -0,0 +1,137 @@ +# Logstash Tarball Proxy + +A `MANAGED` or `SIMULATE` policy pinned to **VERSION** makes every agent download its own copy of +the Logstash release tarball — roughly 450 MB each. At two agents that is wasteful; at fifty it is a +bandwidth event, and in an air-gapped site it is impossible. + +With the proxy enabled, LogstashUI fetches each release **once**, verifies its SHA-512, and serves it +to every agent that needs it. + +## Enabling it + +1. **Cache the tarball.** Management → **Logstash Tarballs** → *Download Tarball*. Pick a version and + architecture, or paste a full URL for an internal mirror. +2. **Point the policy at LogstashUI.** In the policy editor, set the Logstash source to **VERSION**, + enter the version, and tick **Download the tarball from LogstashUI**. + +The checkbox only appears for **Managed** and **Simulate** policies using **VERSION**. Packaged +policies use the OS package and Embedded runs Logstash in-process, so neither ever downloads a +tarball. + +Agents pick up the change on their next check-in. No separate Deploy is needed for binary-only +changes. + +## Managing the cache + +**Management → Logstash Tarballs** lists everything cached, with live progress while a download is +running. + +| Action | What it does | +| --- | --- | +| Download Tarball | Fetch by version + architecture, or from a full URL | +| Import from disk | Register tarballs copied into the cache directory by hand | +| Delete | Remove the tarball, its checksum, and the row | +| Retry download | Re-attempt a failed fetch | + +Tarballs live in `LOGSTASHUI_LOGSTASH_DIR`, default `/logstashes`. There is no upload +action — 450 MB through a browser is not viable. + +**Served** counts full tarball downloads. An agent fetches the `.sha512` sidecar as well, and may +resume an interrupted transfer with a range request; neither adds to the count, so the number tracks +how many agents actually pulled the release. + +### Air-gapped sites + +Copy the tarball (and its `.sha512`, if you have it) into the cache directory, then click **Import +from disk**. LogstashUI hashes the file and publishes it. If you supply a `.sha512` it is checked and +a mismatch fails the import; if you do not, LogstashUI writes one from what it computed so the +agent's own verification step still has something to check. + +## Configuration + +The upstream source is **Management → Settings → Logstash tarball source**. Blank means +`https://artifacts.elastic.co/downloads/logstash`. Point it at an internal mirror to keep fetches +inside your network. + +Everything else is environment, because the concurrency limits interact with `LOGSTASHUI_WORKERS` +and take effect at startup: + +| Variable | Default | Notes | +| --- | --- | --- | +| `LOGSTASHUI_LOGSTASH_DIR` | `/logstashes` | Cache root | +| `LOGSTASHUI_ARTIFACT_MAX_UPSTREAM` | `2` | Concurrent fetches from upstream, across all workers | +| `LOGSTASHUI_ARTIFACT_MAX_SERVE_PER_WORKER` | `4` | Concurrent agent downloads **per worker**; effective total is this × `LOGSTASHUI_WORKERS` | + +The serve limit is a fairness knob, not a throughput knob. Capping does not reduce the total bytes +moved — it decides whether twenty agents each crawl at a twentieth of your bandwidth, or eight finish +quickly and twelve retry shortly after. For a rollout, the second is better. + +## What agents see + +`GET /ConnectionManager/LogstashArtifact//` with +`Authorization: ApiKey `, over the product CA. + +The `connection_id` is in the path because a GET has no body to carry it, and an agent key is a bare +hash with no lookup column — the header alone cannot identify which agent is calling. + +| Status | Meaning | Agent behaviour | +| --- | --- | --- | +| `200` | The file, with `Content-Length` and `Accept-Ranges: bytes` | Verify the SHA-512 as usual | +| `206` | Partial content, if the agent sent a `Range` header | Append and continue | +| `503` | Not cached yet; a fetch has started | Sleep for `Retry-After`, retry | +| `429` | At the concurrent-serve cap | Sleep for `Retry-After`, retry | +| `502` | The upstream fetch failed | Retry, and check the Logstash Tarballs page | +| `404` | Filename not recognized | Fail the deployment | +| `401` | Bad key or wrong `connection_id` | Re-enroll | + +Agents honour `Retry-After`, backing off exponentially to a 5-minute ceiling. When the proxy is +enabled they **never** fall back to `artifacts.elastic.co` — silently reaching the internet would +defeat both the bandwidth saving and air-gapped operation. + +Range requests let an agent resume an interrupted transfer instead of re-pulling 450 MB, which is +exactly the load the cache exists to prevent. + +## How it behaves under load + +- **One download per release, ever.** The first agent to ask triggers the fetch and gets a 503; + everyone else gets a 503 until it lands. This holds across gunicorn workers — the claim is a + conditional database update, not an in-process lock. +- **Nothing partial is ever served.** Downloads land in a `.part` file and are moved into place only + after the SHA-512 verifies. +- **Restarts are safe.** A download killed by a restart leaves a stale claim, which the next request + reclaims; the orphaned `.part` is swept at startup. +- **Check-ins do not starve.** Transfers are chunked so they yield frequently. With the default caps + the practical ceiling is your network, not the server. + +## Measuring it + +Set `LOGSTASHUI_OTEL=true` and install the `otel` extra (`pip install 'LogstashUI[otel]'`; already +bundled in the offline wheel archive) to export traces and metrics over OTLP/HTTP to +`OTEL_EXPORTER_OTLP_ENDPOINT`. + +Four instruments answer the capacity question that request traces cannot: + +| Instrument | What it tells you | +| --- | --- | +| `logstashui.gevent.hub.lag` | The important one. Flat under load means you are network-bound and more workers will not help; rising means requests are starving and more workers or cores will | +| `logstashui.artifact.downloads.active` | If this never reaches the cap, the cap is not your constraint | +| `logstashui.artifact.requests` | The 429/503 rate — the direct "raise the cap" signal | +| `logstashui.artifact.serve.bytes_per_second` | Per-stream throughput. Falling while the aggregate stays flat means you are at the network ceiling | + +Only the HTTP/protobuf OTLP exporter is supported. The gRPC exporter is incompatible with the gevent +worker LogstashUI runs under. + +## Troubleshooting + +**Agents keep getting 503.** Check the Logstash Tarballs page. A row stuck on *Downloading* with no +progress usually means the upstream is unreachable; a *Failed* row shows the reason. + +**"SHA-512 mismatch".** The upstream published a checksum that does not match what was downloaded. +Nothing is published, so agents are unaffected. Retry — this is usually a truncated transfer. + +**Agents still hit artifacts.elastic.co.** The checkbox only takes effect for **Managed**/**Simulate** +policies with source **VERSION**. Confirm the policy saved, then confirm the agent has checked in +since. + +**A version disappeared.** Deleting a tarball a policy still pins sends those agents into a retry +loop until it is cached again. The table marks those rows **in use**. diff --git a/docs/docs/logstashui/configuration/simulation.md b/docs/docs/logstashui/configuration/simulation.md index 89f251c..a82359b 100644 --- a/docs/docs/logstashui/configuration/simulation.md +++ b/docs/docs/logstashui/configuration/simulation.md @@ -41,7 +41,7 @@ You can run **default** and **simulate** agents on the same host; simulate never #### Multi-version testing -Clone the Simulate Policy, set `logstash_source=VERSION` and a version (e.g. `9.4.3`). The agent downloads that release into `/opt/logstash-agent/logstash-versions/` when materializing. Pick the instance in the editor to compare pipeline behavior across releases. +Clone the Simulate Policy, set `logstash_source=VERSION` and a version (e.g. `9.4.3`). The agent downloads that release into `/opt/logstash-agent/logstash-versions/` when materializing — from Elastic artifacts, or from LogstashUI if the policy also enables [the tarball proxy](/docs/docs/logstashui/configuration/logstash_proxy.md). Pick the instance in the editor to compare pipeline behavior across releases. #### Keystore variables diff --git a/pyproject.toml b/pyproject.toml index 9ef6de6..7f76d8d 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -46,6 +46,14 @@ databases = [ "psycopg[binary]>=3.2.0", "PyMySQL>=1.1.1", ] +# Deliberately HTTP/protobuf, never gRPC: grpcio's native C-core threads cannot +# be monkey-patched by gevent and deadlock under the gunicorn worker class. +otel = [ + "opentelemetry-sdk>=1.30.0", + "opentelemetry-exporter-otlp-proto-http>=1.30.0", + "opentelemetry-instrumentation-django>=0.51b0", + "opentelemetry-instrumentation-requests>=0.51b0", +] [tool.uv] package = true diff --git a/src/logstashui/Documentation/views.py b/src/logstashui/Documentation/views.py index eceadaf..54c5d5d 100644 --- a/src/logstashui/Documentation/views.py +++ b/src/logstashui/Documentation/views.py @@ -25,6 +25,7 @@ 'logstashui.yml': 'logstashui.yml', 'SNMP': 'SNMP', 'api_access': 'API Access', + 'logstash_proxy': 'Logstash Tarball Proxy', 'tsds_implementation': 'TSDS Implementation', 'data_overview': 'Data Overview', 'pipeline_generation': 'Pipeline Generation', diff --git a/src/logstashui/LogstashUI/paths.py b/src/logstashui/LogstashUI/paths.py index 437e274..62a4ec4 100644 --- a/src/logstashui/LogstashUI/paths.py +++ b/src/logstashui/LogstashUI/paths.py @@ -4,7 +4,8 @@ """Resolve runtime data / logs directories (no Django import). -Precedence: LOGSTASHUI_DATA_DIR / LOGSTASHUI_LOGS_DIR → default. +Precedence: LOGSTASHUI_DATA_DIR / LOGSTASHUI_LOGS_DIR / LOGSTASHUI_LOGSTASH_DIR +→ default. Default data root is ``$(pwd)/logstashui_data``. Pytest keeps using ``/data`` so test runs do not touch a checkout bind-mount. @@ -78,6 +79,22 @@ def resolve_logs_dir(data_dir: Optional[Path] = None) -> Path: return chosen +def resolve_logstash_dir(data_dir: Optional[Path] = None) -> Path: + """Cache root for proxied Logstash release tarballs. + + Deliberately a sibling of ``staticfiles``, never a child: STATIC_ROOT is + served by WhiteNoise at ``/static/``, which is in LOGIN_REQUIRED_IGNORE_PATHS, + so anything under it is an unauthenticated public download. ``collectstatic`` + also runs on every ``serve`` and would churn over half-gigabyte files. + """ + env = os.environ.get("LOGSTASHUI_LOGSTASH_DIR") + chosen = _coerce_path(env, relative_to=Path.cwd()) + if chosen is None: + root = data_dir if data_dir is not None else resolve_data_dir() + chosen = root / "logstashes" + return chosen + + def maybe_migrate_legacy_data(dest: Path) -> None: """Copy src/logstashui/data → dest when dest has no sqlite and legacy does.""" try: diff --git a/src/logstashui/LogstashUI/settings.py b/src/logstashui/LogstashUI/settings.py index 8dac2b2..a651282 100644 --- a/src/logstashui/LogstashUI/settings.py +++ b/src/logstashui/LogstashUI/settings.py @@ -21,7 +21,12 @@ from .config import CONFIG, merge_allowed_hosts from .database import build_databases from .logging_config import resolve_django_log_levels, resolve_log_level -from .paths import resolve_data_dir, resolve_docs_dir, resolve_logs_dir +from .paths import ( + resolve_data_dir, + resolve_docs_dir, + resolve_logs_dir, + resolve_logstash_dir, +) # Build paths inside the project like this: BASE_DIR / 'subdir'. BASE_DIR = Path(__file__).resolve().parent.parent @@ -252,6 +257,9 @@ def _get_version(): "/ConnectionManager/GetConfigChanges", "/ConnectionManager/IssueServerCert/", "/ConnectionManager/IssueServerCert", + # Prefix match: covers /LogstashArtifact//. + # The view authenticates the agent key itself. + "/ConnectionManager/LogstashArtifact/", ] # Session Configuration @@ -325,6 +333,25 @@ def _get_version(): 'LOGSTASH_AGENT_URL', 'https://logstashagent:9500' ) +# Logstash tarball proxy +# Cache of Logstash release tarballs served to agents whose policy sets +# logstash_via_ui. See PipelineManager/artifacts.py. +LOGSTASH_DIR = resolve_logstash_dir(DATA_DIR) +LOGSTASH_DIR.mkdir(parents=True, exist_ok=True) + +# Upstream fetches in flight cluster-wide, counted in the DB (the only state +# shared between gunicorn workers -- there is no CACHES backend configured). +LOGSTASH_ARTIFACT_MAX_UPSTREAM = int( + os.environ.get('LOGSTASHUI_ARTIFACT_MAX_UPSTREAM', '2') +) +# Concurrent agent downloads *per worker*. Effective total is this value times +# LOGSTASHUI_WORKERS. Deliberately not a global divided by worker count: that +# truncates to zero and makes the knob lie. +LOGSTASH_ARTIFACT_MAX_SERVE_PER_WORKER = int( + os.environ.get('LOGSTASHUI_ARTIFACT_MAX_SERVE_PER_WORKER', '4') +) +LOGSTASH_ARTIFACT_DEFAULT_BASE_URL = 'https://artifacts.elastic.co/downloads/logstash' + # Logging Configuration # https://docs.djangoproject.com/en/5.2/topics/logging/ diff --git a/src/logstashui/LogstashUI/telemetry.py b/src/logstashui/LogstashUI/telemetry.py new file mode 100644 index 0000000..6d9dfbe --- /dev/null +++ b/src/logstashui/LogstashUI/telemetry.py @@ -0,0 +1,141 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +"""Optional OpenTelemetry bootstrap. + +Enabled by ``LOGSTASHUI_OTEL=true`` and the ``LogstashUI[otel]`` extra. Absent +either, every entry point here is a no-op — LogstashUI ships as an air-gapped +wheel bundle and an offline Docker image, so tracing cannot be a hard dependency. + +Two constraints come from running under gunicorn's gevent worker, and both are +easy to get wrong: + +* **The gRPC exporter is unusable.** ``grpcio`` runs its own C-core native + threads, which gevent cannot monkey-patch; it deadlocks or silently drops + spans. This module only ever configures the HTTP/protobuf exporter, which goes + through ``requests`` and is fully cooperative. +* **``BatchSpanProcessor`` is required, not optional.** Its worker is a + ``threading.Thread``, which under gevent is a greenlet, and it blocks on a + patched ``Condition.wait`` — so it is safe. ``SimpleSpanProcessor`` would put a + synchronous OTLP round-trip inside every request, including agent check-ins. + +Initialization belongs in ``wsgi.py``: it runs once per worker, after the fork +and after monkey-patching. ``settings.py`` is imported by ``manage.py``, every +migration, and every ``cli.py`` management command, so initializing there would +spin up a tracer provider for ``collectstatic``. +""" + +import logging +import os +import threading +import time + +logger = logging.getLogger(__name__) + +_initialized = False + +#: How often the hub-lag probe samples. Cheap: one greenlet, one sleep. +_LAG_INTERVAL = 1.0 + + +def _enabled(): + return os.environ.get('LOGSTASHUI_OTEL', 'false').lower() in ('true', '1', 'yes') + + +def init_telemetry(): + """Set up tracing and metrics. Returns True when instrumentation is live.""" + global _initialized + if _initialized or not _enabled(): + return False + + try: + from opentelemetry import metrics, trace + from opentelemetry.exporter.otlp.proto.http.metric_exporter import ( + OTLPMetricExporter, + ) + from opentelemetry.exporter.otlp.proto.http.trace_exporter import ( + OTLPSpanExporter, + ) + from opentelemetry.instrumentation.django import DjangoInstrumentor + from opentelemetry.instrumentation.requests import RequestsInstrumentor + from opentelemetry.sdk.metrics import MeterProvider + from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader + from opentelemetry.sdk.resources import Resource + from opentelemetry.sdk.trace import TracerProvider + from opentelemetry.sdk.trace.export import BatchSpanProcessor + except ImportError: + logger.info( + "LOGSTASHUI_OTEL is set but OpenTelemetry is not installed; " + "tracing disabled. Install the 'otel' extra to enable it." + ) + return False + + try: + resource = Resource.create({ + 'service.name': os.environ.get('OTEL_SERVICE_NAME', 'logstashui'), + }) + + tracer_provider = TracerProvider(resource=resource) + # Batch, never Simple -- see the module docstring. + tracer_provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter())) + trace.set_tracer_provider(tracer_provider) + + meter_provider = MeterProvider( + resource=resource, + metric_readers=[PeriodicExportingMetricReader(OTLPMetricExporter())], + ) + metrics.set_meter_provider(meter_provider) + + # Mutates settings.MIDDLEWARE, so it has to run before the handler is built. + DjangoInstrumentor().instrument() + RequestsInstrumentor().instrument() + + from PipelineManager import artifact_metrics + + artifact_metrics.init(meter_provider) + _start_hub_lag_probe(meter_provider) + + _initialized = True + logger.info("OpenTelemetry instrumentation enabled (OTLP over HTTP)") + return True + except Exception: + # Telemetry must never be the reason a worker fails to boot. + logger.exception("OpenTelemetry initialization failed; continuing without it") + return False + + +def _start_hub_lag_probe(meter_provider): + """Measure how late a greenlet wakes up from a 1-second sleep. + + This is the single most useful signal for capacity decisions and the only one + that sees the problems a span cannot. A greenlet that asks to sleep 1.0 s and + wakes at 1.4 s spent 0.4 s waiting behind something that would not yield -- + a TLS write, a SQLite writer blocked on ``busy_timeout``, a large disk read. + + Reading it: lag stays flat as load rises => the NIC is the ceiling and adding + workers will not help. Lag climbs => greenlets are starving, so more workers + or cores will. + """ + meter = meter_provider.get_meter('logstashui.runtime') + state = {'lag': 0.0} + + def _observe(_options): + from opentelemetry.metrics import Observation + + return [Observation(state['lag'])] + + meter.create_observable_gauge( + 'logstashui.gevent.hub.lag', + callbacks=[_observe], + unit='s', + description='How far past its deadline a sleeping greenlet actually woke', + ) + + def _probe(): + while True: + start = time.monotonic() + time.sleep(_LAG_INTERVAL) + state['lag'] = max(0.0, time.monotonic() - start - _LAG_INTERVAL) + + threading.Thread(target=_probe, daemon=True).start() diff --git a/src/logstashui/LogstashUI/wsgi.py b/src/logstashui/LogstashUI/wsgi.py index 743ed77..1a99156 100644 --- a/src/logstashui/LogstashUI/wsgi.py +++ b/src/logstashui/LogstashUI/wsgi.py @@ -25,6 +25,25 @@ application = get_wsgi_application() ensure_psycopg_gevent() +# Runs once per worker, post-fork and post-monkey-patch. No-op unless +# LOGSTASHUI_OTEL is set and the optional 'otel' extra is installed. +try: + from LogstashUI.telemetry import init_telemetry + + init_telemetry() +except Exception: + pass + +# A tarball download dies with its worker, so a restart can leave a .part behind. +# Clear them here rather than at import of settings, which also runs for every +# management command. +try: + from PipelineManager.artifacts import sweep_partials + + sweep_partials() +except Exception: + pass + # Quiet gevent/gunicorn spam: clients that reject the product CA (browser # probes, scanners, default-trust Python) abort the handshake with diff --git a/src/logstashui/Management/migrations/0004_settings_logstash_artifact_base_url.py b/src/logstashui/Management/migrations/0004_settings_logstash_artifact_base_url.py new file mode 100644 index 0000000..c3aacc7 --- /dev/null +++ b/src/logstashui/Management/migrations/0004_settings_logstash_artifact_base_url.py @@ -0,0 +1,29 @@ +#Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +#or more contributor license agreements. Licensed under the Elastic License; +#you may not use this file except in compliance with the Elastic License. + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('Management', '0003_settings_agent_ui_url'), + ] + + operations = [ + migrations.AddField( + model_name='settings', + name='logstash_artifact_base_url', + field=models.CharField( + blank=True, + default='', + help_text=( + 'Upstream source for Logstash release tarballs. Blank uses ' + 'https://artifacts.elastic.co/downloads/logstash. Point this at an ' + 'internal mirror to keep tarball fetches inside your network.' + ), + max_length=512, + ), + ), + ] diff --git a/src/logstashui/Management/models.py b/src/logstashui/Management/models.py index 1fdfa7d..32639f9 100644 --- a/src/logstashui/Management/models.py +++ b/src/logstashui/Management/models.py @@ -41,6 +41,16 @@ class Settings(models.Model): "May differ from the browser reverse-proxy URL." ), ) + logstash_artifact_base_url = models.CharField( + max_length=512, + blank=True, + default="", + help_text=( + "Upstream source for Logstash release tarballs. Blank uses " + "https://artifacts.elastic.co/downloads/logstash. Point this at an " + "internal mirror to keep tarball fetches inside your network." + ), + ) class Meta: db_table = 'settings' diff --git a/src/logstashui/Management/templates/components/logstash_artifact_row.html b/src/logstashui/Management/templates/components/logstash_artifact_row.html new file mode 100644 index 0000000..58046ca --- /dev/null +++ b/src/logstashui/Management/templates/components/logstash_artifact_row.html @@ -0,0 +1,100 @@ + + + + + {{ artifact.filename }} + + + {{ artifact.version }} + {% if artifact.in_use %} + + in use + + {% endif %} + + + {% if artifact.status == 'READY' %} + + Ready + + {% elif artifact.status == 'FAILED' %} + + Failed + + {% if artifact.error %} +

{{ artifact.error|truncatechars:160 }}

+ {% endif %} + {% elif artifact.status == 'PENDING' %} + + Queued + + {% else %} + + {% if artifact.status == 'IMPORTING' %}Verifying{% else %}Downloading{% endif %} + +
+
+
+

+ {% if artifact.percent is not None %}{{ artifact.percent }}%{% else %}starting…{% endif %} +

+ {% endif %} + + + {% if artifact.size_bytes %}{{ artifact.size_bytes|filesizeformat }}{% else %}—{% endif %} + + + {{ artifact.serve_count }}× + {% if artifact.last_served_at %} + {{ artifact.last_served_at|date:"Y-m-d H:i" }} + {% endif %} + + +
+ + +
+ + diff --git a/src/logstashui/Management/templates/components/logstash_artifact_tbody.html b/src/logstashui/Management/templates/components/logstash_artifact_tbody.html new file mode 100644 index 0000000..2f06d17 --- /dev/null +++ b/src/logstashui/Management/templates/components/logstash_artifact_tbody.html @@ -0,0 +1,26 @@ + + + {% for artifact in artifacts %} + {% include 'components/logstash_artifact_row.html' %} + {% empty %} + + + No tarballs cached yet. Use Download Tarball, + or copy one into the cache directory and click + Import from disk. + + + {% endfor %} + diff --git a/src/logstashui/Management/templates/logstash_artifacts.html b/src/logstashui/Management/templates/logstash_artifacts.html new file mode 100644 index 0000000..90bbc7b --- /dev/null +++ b/src/logstashui/Management/templates/logstash_artifacts.html @@ -0,0 +1,172 @@ + + +{% extends "base.html" %} + +{% block content %} + +
+
+

Logstash Tarballs

+
+ + + +
+
+

+ LogstashUI fetches each release once and serves it to every agent whose policy has + Download the tarball from LogstashUI enabled, + instead of each agent pulling ~450 MB from Elastic. +

+

+ Cache: {{ artifact_dir }} + · Source: {{ base_url }} + (change under Settings) +

+ +
+ +
+ + + + + + + + + + + + {% include 'components/logstash_artifact_tbody.html' %} +
TarballVersionStatusSizeServedActions
+
+
+ + + + + +{% endblock %} diff --git a/src/logstashui/Management/templates/management.html b/src/logstashui/Management/templates/management.html index 8dd8bda..e3d5c7a 100644 --- a/src/logstashui/Management/templates/management.html +++ b/src/logstashui/Management/templates/management.html @@ -49,6 +49,19 @@

API Tokens

+ + +
+
+ + + +
+

Logstash Tarballs

+

Cache Logstash releases once and serve them to agents

+
+ +
diff --git a/src/logstashui/Management/templates/management_settings.html b/src/logstashui/Management/templates/management_settings.html index 2bc72de..05423ee 100644 --- a/src/logstashui/Management/templates/management_settings.html +++ b/src/logstashui/Management/templates/management_settings.html @@ -70,6 +70,26 @@

Agent callback URL

{% if app_settings.agent_ui_url %}{{ app_settings.agent_ui_url }}{% else %}<ui_url>{% endif %}/.well-known/logstashui/ca.crt

+ + +
+

Logstash tarball source

+

+ Where LogstashUI fetches Logstash release tarballs for the + tarball proxy. + Leave blank to use Elastic's artifact host. Point it at an internal mirror to + keep fetches inside your network. +

+ +

+ Tarballs are read as <base>/logstash-<version>-linux-x86_64.tar.gz, + with the matching .sha512 alongside. +

+
@@ -174,13 +194,16 @@

Use custom certificate

{% endblock %} From 674761198c27d1377afecd7ba7eb377409471b6a Mon Sep 17 00:00:00 2001 From: Aaron Mildenstein Date: Fri, 4 Sep 2026 16:52:41 -0600 Subject: [PATCH 54/62] docs: insecure HTTP is not the Compose or Kubernetes default --- CHANGELOG.md | 2 ++ docs/docs/logstashui/configuration/environment.md | 2 ++ .../logstashui/kubernetes/examples/mysql/statefulset.yaml | 4 ++++ .../kubernetes/examples/postgresql/statefulset.yaml | 4 ++++ .../logstashui/kubernetes/examples/sqlite/statefulset.yaml | 4 ++++ 5 files changed, 16 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1d31230..1d0320a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -51,6 +51,8 @@ SQLite does not scale under gunicorn/gevent. Operators can now run LogstashUI on ### Insecure HTTP (escape hatch) - `LOGSTASHUI_INSECURE_HTTP=true` forces plain HTTP for the UI and every UI→agent URL, skips product CA and certificate generation, and overrides `LOGSTASHUI_TLS`. This is **not** best practice; automatic TLS remains the supported default. It is **not** the same as `LOGSTASHUI_TLS=false` (TLS-terminating ingress). See [Environment](docs/docs/logstashui/configuration/environment.md). +- Settings TLS **Upload** and **Revert** are hidden in this mode and return **409** before any filesystem change. Leftover `$DATA_DIR/tls/` custom leaves are left intact (`save_custom_ui_certificate` / `revert_ui_certificate_to_product_default` raise `ProductCADisabled` first). +- Default Compose and Kubernetes examples remain HTTPS. The hatch is native/`logstashui serve` (or a hand-edited EnvironmentFile). StatefulSet examples comment that probes stay `scheme: HTTPS`. ### Fixes diff --git a/docs/docs/logstashui/configuration/environment.md b/docs/docs/logstashui/configuration/environment.md index 6411311..7b3e26c 100644 --- a/docs/docs/logstashui/configuration/environment.md +++ b/docs/docs/logstashui/configuration/environment.md @@ -67,6 +67,8 @@ LogstashAgent needs a matching TLS-off flag on its side. That flag is not config **This is not `LOGSTASHUI_TLS=false`.** That setting is only for a TLS-terminating ingress: gunicorn speaks HTTP, agents still use HTTPS, and the product CA is still issued. Kubernetes should keep `LOGSTASHUI_TLS=true` and skip-verify at the Gateway/Ingress. See [Kubernetes](/docs/docs/logstashui/kubernetes/index.md). +**Standard Compose and Kubernetes examples stay HTTPS.** Setting `LOGSTASHUI_INSECURE_HTTP` in the host shell does not enter the UI container. Injecting it without changing the Compose healthcheck (today `https://127.0.0.1:8443/.well-known/logstashui/ca.crt`) leaves the service unhealthy — that endpoint is 404 in this mode. Kubernetes example probes use `scheme: HTTPS`; uncommenting the ConfigMap key without changing probe (and Ingress/HTTPRoute backend) scheme fails the pod. Use this hatch with native `logstashui serve` or a hand-edited systemd EnvironmentFile. + --- ## Observability diff --git a/docs/docs/logstashui/kubernetes/examples/mysql/statefulset.yaml b/docs/docs/logstashui/kubernetes/examples/mysql/statefulset.yaml index d614205..6f08755 100644 --- a/docs/docs/logstashui/kubernetes/examples/mysql/statefulset.yaml +++ b/docs/docs/logstashui/kubernetes/examples/mysql/statefulset.yaml @@ -59,6 +59,10 @@ spec: volumeMounts: - name: data mountPath: /var/lib/logstashui + # Probes are HTTPS. If you uncomment LOGSTASHUI_INSECURE_HTTP in the + # ConfigMap, gunicorn speaks HTTP and these probes will fail until + # scheme is HTTP (and Ingress/HTTPRoute backend protocol too). + # Do not enable that flag without cause. Automatic TLS is supported. startupProbe: httpGet: path: / diff --git a/docs/docs/logstashui/kubernetes/examples/postgresql/statefulset.yaml b/docs/docs/logstashui/kubernetes/examples/postgresql/statefulset.yaml index d614205..6f08755 100644 --- a/docs/docs/logstashui/kubernetes/examples/postgresql/statefulset.yaml +++ b/docs/docs/logstashui/kubernetes/examples/postgresql/statefulset.yaml @@ -59,6 +59,10 @@ spec: volumeMounts: - name: data mountPath: /var/lib/logstashui + # Probes are HTTPS. If you uncomment LOGSTASHUI_INSECURE_HTTP in the + # ConfigMap, gunicorn speaks HTTP and these probes will fail until + # scheme is HTTP (and Ingress/HTTPRoute backend protocol too). + # Do not enable that flag without cause. Automatic TLS is supported. startupProbe: httpGet: path: / diff --git a/docs/docs/logstashui/kubernetes/examples/sqlite/statefulset.yaml b/docs/docs/logstashui/kubernetes/examples/sqlite/statefulset.yaml index d614205..6f08755 100644 --- a/docs/docs/logstashui/kubernetes/examples/sqlite/statefulset.yaml +++ b/docs/docs/logstashui/kubernetes/examples/sqlite/statefulset.yaml @@ -59,6 +59,10 @@ spec: volumeMounts: - name: data mountPath: /var/lib/logstashui + # Probes are HTTPS. If you uncomment LOGSTASHUI_INSECURE_HTTP in the + # ConfigMap, gunicorn speaks HTTP and these probes will fail until + # scheme is HTTP (and Ingress/HTTPRoute backend protocol too). + # Do not enable that flag without cause. Automatic TLS is supported. startupProbe: httpGet: path: / From 5a41499f51fb5975535636ea374f4bca828eaf31 Mon Sep 17 00:00:00 2001 From: Aaron Mildenstein Date: Fri, 4 Sep 2026 17:33:54 -0600 Subject: [PATCH 55/62] docs: add Kubernetes embedded-agent overlay --- .../kubernetes/examples/embedded-agent.yaml | 112 ++++++++++++++++++ 1 file changed, 112 insertions(+) create mode 100644 docs/docs/logstashui/kubernetes/examples/embedded-agent.yaml diff --git a/docs/docs/logstashui/kubernetes/examples/embedded-agent.yaml b/docs/docs/logstashui/kubernetes/examples/embedded-agent.yaml new file mode 100644 index 0000000..75461a8 --- /dev/null +++ b/docs/docs/logstashui/kubernetes/examples/embedded-agent.yaml @@ -0,0 +1,112 @@ +# Optional embedded simulation agent (compose --profile embedded analog). +# Apply a DB tree first (sqlite/ postgresql/ or mysql/), uncomment +# LOGSTASH_AGENT_URL in ConfigMap/logstashui and LOGSTASHUI_AGENT_CSR_SECRET +# in Secret/logstashui, then: +# kubectl apply -f docs/docs/logstashui/kubernetes/examples/embedded-agent.yaml +# ClusterIP only. Lab node. Prefer enrolled Simulate agents for serious work. +# Missing CSR secret → CreateContainerConfigError (intentional). +apiVersion: v1 +kind: ConfigMap +metadata: + name: logstashagent + namespace: logstashui +data: + LOGSTASH_UI_URL: https://logstashui:8443 + LOGSTASH_URL: https://logstashui:8443 + SIMULATION_MODE: "true" + # Agent API TLS (default true). Automatic product TLS works out of the box. + # Uncomment false only if the UI is also plain HTTP (LOGSTASHUI_INSECURE_HTTP) + # and you have rewritten LOGSTASH_UI_URL / LOGSTASH_URL / LOGSTASH_AGENT_URL + # to http://. Do not enable without cause. + # LOGSTASH_AGENT_TLS: "false" + # Skip verifying the UI HTTPS certificate (default false). + # Not the same as plain HTTP. Product CA pinning works OOTB; leave commented. + # LOGSTASH_UI_TLS_INSECURE: "true" +--- +apiVersion: v1 +kind: Service +metadata: + name: logstashagent + namespace: logstashui + labels: + app: logstashagent +spec: + selector: + app: logstashagent + ports: + - name: agent + port: 9500 + targetPort: agent + - name: logstash-api + port: 9560 + targetPort: logstash-api + - name: http-input + port: 9449 + targetPort: http-input +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: logstashagent + namespace: logstashui +spec: + replicas: 1 + selector: + matchLabels: + app: logstashagent + template: + metadata: + labels: + app: logstashagent + spec: + initContainers: + - name: wait-ui-ca + image: codyjackson032/logstashui:latest + imagePullPolicy: IfNotPresent + command: + - python + - -c + - | + import ssl, time, urllib.request + ctx = ssl._create_unverified_context() + url = "https://logstashui:8443/.well-known/logstashui/ca.crt" + while True: + try: + urllib.request.urlopen(url, context=ctx, timeout=5) + break + except Exception: + time.sleep(2) + containers: + - name: logstashagent + image: codyjackson032/logstash-agent:latest + imagePullPolicy: IfNotPresent + ports: + - name: agent + containerPort: 9500 + - name: logstash-api + containerPort: 9560 + - name: http-input + containerPort: 9449 + envFrom: + - configMapRef: + name: logstashagent + env: + - name: LOGSTASHUI_AGENT_CSR_SECRET + valueFrom: + secretKeyRef: + name: logstashui + key: LOGSTASHUI_AGENT_CSR_SECRET + startupProbe: + tcpSocket: + port: agent + periodSeconds: 5 + failureThreshold: 60 + readinessProbe: + tcpSocket: + port: agent + periodSeconds: 10 + livenessProbe: + tcpSocket: + port: agent + periodSeconds: 30 + failureThreshold: 6 From 1eba16af1f89783030271a947ba59d7b8e082e09 Mon Sep 17 00:00:00 2001 From: Aaron Mildenstein Date: Fri, 4 Sep 2026 17:34:07 -0600 Subject: [PATCH 56/62] docs: comment LOGSTASH_AGENT_URL on K8s example ConfigMaps --- docs/docs/logstashui/kubernetes/examples/mysql/configmap.yaml | 4 ++++ .../logstashui/kubernetes/examples/postgresql/configmap.yaml | 4 ++++ .../docs/logstashui/kubernetes/examples/sqlite/configmap.yaml | 4 ++++ 3 files changed, 12 insertions(+) diff --git a/docs/docs/logstashui/kubernetes/examples/mysql/configmap.yaml b/docs/docs/logstashui/kubernetes/examples/mysql/configmap.yaml index 36578b2..6035134 100644 --- a/docs/docs/logstashui/kubernetes/examples/mysql/configmap.yaml +++ b/docs/docs/logstashui/kubernetes/examples/mysql/configmap.yaml @@ -33,3 +33,7 @@ data: # LOGSTASHUI_OTEL: "true" # OTEL_SERVICE_NAME: logstashui # OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector.observability.svc:4318 + # Embedded simulation agent (optional). Apply examples/embedded-agent.yaml + # after uncommenting this key and LOGSTASHUI_AGENT_CSR_SECRET in the Secret. + # Leave commented if you are not running the in-cluster embedded node. + # LOGSTASH_AGENT_URL: https://logstashagent:9500 diff --git a/docs/docs/logstashui/kubernetes/examples/postgresql/configmap.yaml b/docs/docs/logstashui/kubernetes/examples/postgresql/configmap.yaml index 3650d11..7d07c00 100644 --- a/docs/docs/logstashui/kubernetes/examples/postgresql/configmap.yaml +++ b/docs/docs/logstashui/kubernetes/examples/postgresql/configmap.yaml @@ -34,3 +34,7 @@ data: # LOGSTASHUI_OTEL: "true" # OTEL_SERVICE_NAME: logstashui # OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector.observability.svc:4318 + # Embedded simulation agent (optional). Apply examples/embedded-agent.yaml + # after uncommenting this key and LOGSTASHUI_AGENT_CSR_SECRET in the Secret. + # Leave commented if you are not running the in-cluster embedded node. + # LOGSTASH_AGENT_URL: https://logstashagent:9500 diff --git a/docs/docs/logstashui/kubernetes/examples/sqlite/configmap.yaml b/docs/docs/logstashui/kubernetes/examples/sqlite/configmap.yaml index eb62a37..54430dc 100644 --- a/docs/docs/logstashui/kubernetes/examples/sqlite/configmap.yaml +++ b/docs/docs/logstashui/kubernetes/examples/sqlite/configmap.yaml @@ -27,3 +27,7 @@ data: # LOGSTASHUI_OTEL: "true" # OTEL_SERVICE_NAME: logstashui # OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector.observability.svc:4318 + # Embedded simulation agent (optional). Apply examples/embedded-agent.yaml + # after uncommenting this key and LOGSTASHUI_AGENT_CSR_SECRET in the Secret. + # Leave commented if you are not running the in-cluster embedded node. + # LOGSTASH_AGENT_URL: https://logstashagent:9500 From d10a2bc441566c1fd60d9112ff5e02a9b7d7f395 Mon Sep 17 00:00:00 2001 From: Aaron Mildenstein Date: Fri, 4 Sep 2026 17:34:23 -0600 Subject: [PATCH 57/62] docs: comment LOGSTASHUI_AGENT_CSR_SECRET on K8s example Secrets --- docs/docs/logstashui/kubernetes/examples/mysql/secret.yaml | 4 ++++ .../logstashui/kubernetes/examples/postgresql/secret.yaml | 4 ++++ docs/docs/logstashui/kubernetes/examples/sqlite/secret.yaml | 4 ++++ 3 files changed, 12 insertions(+) diff --git a/docs/docs/logstashui/kubernetes/examples/mysql/secret.yaml b/docs/docs/logstashui/kubernetes/examples/mysql/secret.yaml index 19ef69e..4e97b5c 100644 --- a/docs/docs/logstashui/kubernetes/examples/mysql/secret.yaml +++ b/docs/docs/logstashui/kubernetes/examples/mysql/secret.yaml @@ -7,3 +7,7 @@ type: Opaque stringData: SECRET_KEY: CHANGE-ME-generate-a-django-secret-key LOGSTASHUI_DB_PASSWORD: CHANGE-ME + # Shared with Deployment/logstashagent so the embedded node can CSR without enroll. + # Uncomment when applying examples/embedded-agent.yaml. Must match what the + # agent reads. Do not use the compose default in a real cluster. + # LOGSTASHUI_AGENT_CSR_SECRET: CHANGE-ME-shared-with-the-agent diff --git a/docs/docs/logstashui/kubernetes/examples/postgresql/secret.yaml b/docs/docs/logstashui/kubernetes/examples/postgresql/secret.yaml index 19ef69e..4e97b5c 100644 --- a/docs/docs/logstashui/kubernetes/examples/postgresql/secret.yaml +++ b/docs/docs/logstashui/kubernetes/examples/postgresql/secret.yaml @@ -7,3 +7,7 @@ type: Opaque stringData: SECRET_KEY: CHANGE-ME-generate-a-django-secret-key LOGSTASHUI_DB_PASSWORD: CHANGE-ME + # Shared with Deployment/logstashagent so the embedded node can CSR without enroll. + # Uncomment when applying examples/embedded-agent.yaml. Must match what the + # agent reads. Do not use the compose default in a real cluster. + # LOGSTASHUI_AGENT_CSR_SECRET: CHANGE-ME-shared-with-the-agent diff --git a/docs/docs/logstashui/kubernetes/examples/sqlite/secret.yaml b/docs/docs/logstashui/kubernetes/examples/sqlite/secret.yaml index 1e9afcc..150aaa2 100644 --- a/docs/docs/logstashui/kubernetes/examples/sqlite/secret.yaml +++ b/docs/docs/logstashui/kubernetes/examples/sqlite/secret.yaml @@ -6,3 +6,7 @@ metadata: type: Opaque stringData: SECRET_KEY: CHANGE-ME-generate-a-django-secret-key + # Shared with Deployment/logstashagent so the embedded node can CSR without enroll. + # Uncomment when applying examples/embedded-agent.yaml. Must match what the + # agent reads. Do not use the compose default in a real cluster. + # LOGSTASHUI_AGENT_CSR_SECRET: CHANGE-ME-shared-with-the-agent From d7cb471a9b7d24b312e19ebd195eaf8b1674fed7 Mon Sep 17 00:00:00 2001 From: Aaron Mildenstein Date: Fri, 4 Sep 2026 17:34:42 -0600 Subject: [PATCH 58/62] docs: document Kubernetes embedded agent overlay --- .../logstashui/kubernetes/examples/README.md | 25 +++++++++++++++++++ docs/docs/logstashui/kubernetes/index.md | 8 ++++++ 2 files changed, 33 insertions(+) diff --git a/docs/docs/logstashui/kubernetes/examples/README.md b/docs/docs/logstashui/kubernetes/examples/README.md index 1ab324e..8e4fbae 100644 --- a/docs/docs/logstashui/kubernetes/examples/README.md +++ b/docs/docs/logstashui/kubernetes/examples/README.md @@ -15,3 +15,28 @@ Envoy Gateway instead of Ingress: [envoy-gateway.md](../envoy-gateway.md) (enabl ```bash kubectl apply -f docs/docs/logstashui/kubernetes/examples/sqlite/ ``` + +## Embedded agent (optional) + +Same role as compose `--profile embedded`: an in-cluster LogstashAgent for the pipeline editor Sim target (`embedded · docker`). No enroll. Lab only — prefer an enrolled Simulate agent for serious work. + +1. Apply **one** DB tree above. +2. Uncomment `LOGSTASH_AGENT_URL` in that tree's ConfigMap and `LOGSTASHUI_AGENT_CSR_SECRET` in its Secret. Replace the secret placeholder. Do not reuse the compose default. +3. Apply the overlay: + +```bash +kubectl apply -f docs/docs/logstashui/kubernetes/examples/embedded-agent.yaml +``` + +`Service/logstashagent` is ClusterIP **9500** (agent API), **9560** (Logstash API), **9449** (HTTP input). Nothing is published outside the cluster. Image: `codyjackson032/logstash-agent:latest`. + +If the CSR secret key is still commented, the agent pod is `CreateContainerConfigError` until you uncomment it. + +Agent ConfigMap `logstashagent` already sets `LOGSTASH_UI_URL` / `LOGSTASH_URL` to `https://logstashui:8443` and `SIMULATION_MODE=true`. Two extra keys are **commented** (LogstashAgent env, not UI): + +| Env | Default | Do not enable without cause | +|---|---|---| +| `LOGSTASH_AGENT_TLS` | `true` | Uncomment `"false"` only with `LOGSTASHUI_INSECURE_HTTP` and `http://` URLs. Serves the agent API over HTTP. | +| `LOGSTASH_UI_TLS_INSECURE` | `false` | Uncomment `"true"` to skip verifying the UI HTTPS cert. Not the same as plain HTTP. Product CA pinning works out of the box. | + +See the LogstashAgent README TLS table. Standard examples stay HTTPS. diff --git a/docs/docs/logstashui/kubernetes/index.md b/docs/docs/logstashui/kubernetes/index.md index 5e63471..a6458ac 100644 --- a/docs/docs/logstashui/kubernetes/index.md +++ b/docs/docs/logstashui/kubernetes/index.md @@ -103,11 +103,19 @@ Default engine is SQLite on the PVC. For Postgres or MySQL/MariaDB, set the disc --- +## Embedded agent (optional) + +Compose `--profile embedded` analog: [embedded-agent.yaml](examples/embedded-agent.yaml). Apply a DB tree, uncomment `LOGSTASH_AGENT_URL` and `LOGSTASHUI_AGENT_CSR_SECRET`, then apply the overlay. ClusterIP only (`9500` / `9560` / `9449`). Details: [examples README](examples/README.md#embedded-agent-optional). + +--- + ## Apply (generic) ```bash kubectl apply -f docs/docs/logstashui/kubernetes/examples/sqlite/ # or postgresql/ or mysql/ +# optional embedded sim node: +# kubectl apply -f docs/docs/logstashui/kubernetes/examples/embedded-agent.yaml ``` Replace `logstashui.example.com` in ConfigMap, Secret, and Ingress. Replace `SECRET_KEY`. For Postgres/MySQL, create the empty database first ([SQL examples](/docs/docs/logstashui/database/examples/)), then apply the StatefulSet. First start runs `migrate`. From 03c3e79fcfbe75c8a0c0c29e0b177a0ce0dffe3e Mon Sep 17 00:00:00 2001 From: Aaron Mildenstein Date: Fri, 4 Sep 2026 17:34:54 -0600 Subject: [PATCH 59/62] docs: point agent URL and CSR secret at K8s embedded overlay --- docs/docs/logstashui/configuration/environment.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/docs/logstashui/configuration/environment.md b/docs/docs/logstashui/configuration/environment.md index 7b3e26c..46158fa 100644 --- a/docs/docs/logstashui/configuration/environment.md +++ b/docs/docs/logstashui/configuration/environment.md @@ -92,11 +92,13 @@ Only the HTTP/protobuf OTLP exporter is supported. The gRPC exporter is incompat | `LOGSTASHUI_NO_AUTH` | `false` | Bypass login (**sandbox only**) | | `LOGSTASHUI_AGENT_UI_URL` | empty | Prefill `--logstash-ui-url` (DB Settings wins if set) | | `LOGSTASHUI_INCLUDE_CA_FINGERPRINT` | `true` | Embed product CA fingerprint in enrollment tokens | -| `LOGSTASH_AGENT_URL` | debug: `http://127.0.0.1:9500`; else `https://logstashagent:9500` | Embedded/compose agent API | +| `LOGSTASH_AGENT_URL` | debug: `http://127.0.0.1:9500`; else `https://logstashagent:9500` | Embedded/compose agent API. Kubernetes examples comment this; apply [embedded-agent.yaml](/docs/docs/logstashui/kubernetes/examples/embedded-agent.yaml) after uncommenting. | | `LOGSTASHUI_HOST_HOSTNAME` / `LOGSTASHUI_HOST_IPS` / `LOGSTASHUI_TLS_SANS` | empty | Extra SANs on the product UI cert. Kubernetes: set `LOGSTASHUI_HOST_IPS` from `status.podIP` (Downward API). IPs are also appended to `ALLOWED_HOSTS` unless that list is `*`. | -| `LOGSTASHUI_AGENT_CSR_SECRET` | empty | Compose/embedded agent CSR without enroll | +| `LOGSTASHUI_AGENT_CSR_SECRET` | empty | Compose/embedded agent CSR without enroll. Kubernetes examples comment this on Secret `logstashui`; the overlay `secretKeyRef` requires it uncommented. | | `LOGSTASHUI_DOCS_DIR` | checkout `docs/` or packaged copy | In-app documentation root | +Agent-side (not UI ConfigMap): `LOGSTASH_AGENT_TLS` (default `true`) and `LOGSTASH_UI_TLS_INSECURE` (default `false`) are LogstashAgent env vars. The Kubernetes overlay comments both on ConfigMap `logstashagent`. See the LogstashAgent README TLS table. Do not enable without cause. + Booleans accept `true`/`false`, `1`/`0`, `yes`/`no`, `on`/`off`. --- From 8cb9ffbfc7679401ddfc11706ee4da6b80e95a1a Mon Sep 17 00:00:00 2001 From: Aaron Mildenstein Date: Fri, 4 Sep 2026 17:35:00 -0600 Subject: [PATCH 60/62] docs: CHANGELOG Kubernetes embedded-agent overlay --- CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1d0320a..18e7e91 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -46,6 +46,7 @@ SQLite does not scale under gunicorn/gevent. Operators can now run LogstashUI on - Kubernetes subsection: StatefulSet + one PVC at `/var/lib/logstashui`, TLS kept on `:8443`, Ingress-nginx skip backend verify, Envoy Gateway `Backend` `insecureSkipVerify` (enable Backend API), CloudNativePG Cluster in the app namespace. - K8s probes send `Host: logstashui` (kubelet otherwise uses the pod IP and Django returns 400). Downward API `status.podIP` → `LOGSTASHUI_HOST_IPS` for the product leaf; those IPs are appended to `ALLOWED_HOSTS` unless the list is `*`. - Example manifests under `docs/docs/logstashui/kubernetes/examples/{sqlite,postgresql,mysql}/`. +- Optional embedded simulation agent overlay: `docs/docs/logstashui/kubernetes/examples/embedded-agent.yaml` (compose `--profile embedded` analog). ClusterIP 9500 / 9560 / 9449. Uncomment `LOGSTASH_AGENT_URL` and `LOGSTASHUI_AGENT_CSR_SECRET` on the UI examples before apply. Agent ConfigMap comments `LOGSTASH_AGENT_TLS` (default true) and `LOGSTASH_UI_TLS_INSECURE` (default false). - Database subsection: engines, every `LOGSTASHUI_DB_*` default, offline dump/load, BETA `migrate-engine`, CREATE DATABASE scripts (`utf8mb4_bin` for MySQL/MariaDB), schema snapshots from 0.5.2 `migrate`. ### Insecure HTTP (escape hatch) From cfb9eee4fa7d4a24314aa8ce8105a60eb1b2cde3 Mon Sep 17 00:00:00 2001 From: Aaron Mildenstein Date: Fri, 4 Sep 2026 18:17:12 -0600 Subject: [PATCH 61/62] Update the k8s example values to be consistent. --- .../docs/logstashui/kubernetes/examples/embedded-agent.yaml | 5 +++-- .../logstashui/kubernetes/examples/mysql/configmap.yaml | 6 ++---- docs/docs/logstashui/kubernetes/examples/mysql/secret.yaml | 5 ++--- .../kubernetes/examples/postgresql/configmap.yaml | 6 ++---- .../logstashui/kubernetes/examples/postgresql/secret.yaml | 5 ++--- .../logstashui/kubernetes/examples/sqlite/configmap.yaml | 6 ++---- docs/docs/logstashui/kubernetes/examples/sqlite/secret.yaml | 5 ++--- docs/docs/logstashui/kubernetes/index.md | 2 +- 8 files changed, 16 insertions(+), 24 deletions(-) diff --git a/docs/docs/logstashui/kubernetes/examples/embedded-agent.yaml b/docs/docs/logstashui/kubernetes/examples/embedded-agent.yaml index 75461a8..595f4a0 100644 --- a/docs/docs/logstashui/kubernetes/examples/embedded-agent.yaml +++ b/docs/docs/logstashui/kubernetes/examples/embedded-agent.yaml @@ -13,12 +13,13 @@ metadata: data: LOGSTASH_UI_URL: https://logstashui:8443 LOGSTASH_URL: https://logstashui:8443 - SIMULATION_MODE: "true" + # # Agent API TLS (default true). Automatic product TLS works out of the box. - # Uncomment false only if the UI is also plain HTTP (LOGSTASHUI_INSECURE_HTTP) + # Uncomment false ONLY if the UI is also plain HTTP (LOGSTASHUI_INSECURE_HTTP) # and you have rewritten LOGSTASH_UI_URL / LOGSTASH_URL / LOGSTASH_AGENT_URL # to http://. Do not enable without cause. # LOGSTASH_AGENT_TLS: "false" + # # Skip verifying the UI HTTPS certificate (default false). # Not the same as plain HTTP. Product CA pinning works OOTB; leave commented. # LOGSTASH_UI_TLS_INSECURE: "true" diff --git a/docs/docs/logstashui/kubernetes/examples/mysql/configmap.yaml b/docs/docs/logstashui/kubernetes/examples/mysql/configmap.yaml index 6035134..6cbe91d 100644 --- a/docs/docs/logstashui/kubernetes/examples/mysql/configmap.yaml +++ b/docs/docs/logstashui/kubernetes/examples/mysql/configmap.yaml @@ -22,10 +22,12 @@ data: LOGSTASHUI_DB_USER: logstashui # Optional CA file path if the server requires TLS: # LOGSTASHUI_DB_SSL_CA: /etc/ssl/certs/mysql-ca.pem + # # NOT recommended. Automatic product TLS works out of the box. # If you must run the UI and all agent connections over plain HTTP # (no CA, no certs), uncomment. Do not enable without cause. # LOGSTASHUI_INSECURE_HTTP: "true" + # # OpenTelemetry (optional). The image already installs LogstashUI[otel]. # Uncomment to export traces/metrics over OTLP/HTTP (not gRPC — gevent # cannot patch grpcio). Point at a collector that accepts HTTP/protobuf @@ -33,7 +35,3 @@ data: # LOGSTASHUI_OTEL: "true" # OTEL_SERVICE_NAME: logstashui # OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector.observability.svc:4318 - # Embedded simulation agent (optional). Apply examples/embedded-agent.yaml - # after uncommenting this key and LOGSTASHUI_AGENT_CSR_SECRET in the Secret. - # Leave commented if you are not running the in-cluster embedded node. - # LOGSTASH_AGENT_URL: https://logstashagent:9500 diff --git a/docs/docs/logstashui/kubernetes/examples/mysql/secret.yaml b/docs/docs/logstashui/kubernetes/examples/mysql/secret.yaml index 4e97b5c..fc774a3 100644 --- a/docs/docs/logstashui/kubernetes/examples/mysql/secret.yaml +++ b/docs/docs/logstashui/kubernetes/examples/mysql/secret.yaml @@ -8,6 +8,5 @@ stringData: SECRET_KEY: CHANGE-ME-generate-a-django-secret-key LOGSTASHUI_DB_PASSWORD: CHANGE-ME # Shared with Deployment/logstashagent so the embedded node can CSR without enroll. - # Uncomment when applying examples/embedded-agent.yaml. Must match what the - # agent reads. Do not use the compose default in a real cluster. - # LOGSTASHUI_AGENT_CSR_SECRET: CHANGE-ME-shared-with-the-agent + # Must match what the agent reads. Do not use the default in a real cluster. + LOGSTASHUI_AGENT_CSR_SECRET: CHANGE-ME-shared-with-the-agent diff --git a/docs/docs/logstashui/kubernetes/examples/postgresql/configmap.yaml b/docs/docs/logstashui/kubernetes/examples/postgresql/configmap.yaml index 7d07c00..d19d24c 100644 --- a/docs/docs/logstashui/kubernetes/examples/postgresql/configmap.yaml +++ b/docs/docs/logstashui/kubernetes/examples/postgresql/configmap.yaml @@ -23,10 +23,12 @@ data: # PASSWORD is in secret.yaml. For CloudNativePG, prefer secretKeyRef from # -app instead of these HOST/USER/NAME keys — see cnpg.yaml and # ../../cnpg.md. + # # NOT recommended. Automatic product TLS works out of the box. # If you must run the UI and all agent connections over plain HTTP # (no CA, no certs), uncomment. Do not enable without cause. # LOGSTASHUI_INSECURE_HTTP: "true" + # # OpenTelemetry (optional). The image already installs LogstashUI[otel]. # Uncomment to export traces/metrics over OTLP/HTTP (not gRPC — gevent # cannot patch grpcio). Point at a collector that accepts HTTP/protobuf @@ -34,7 +36,3 @@ data: # LOGSTASHUI_OTEL: "true" # OTEL_SERVICE_NAME: logstashui # OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector.observability.svc:4318 - # Embedded simulation agent (optional). Apply examples/embedded-agent.yaml - # after uncommenting this key and LOGSTASHUI_AGENT_CSR_SECRET in the Secret. - # Leave commented if you are not running the in-cluster embedded node. - # LOGSTASH_AGENT_URL: https://logstashagent:9500 diff --git a/docs/docs/logstashui/kubernetes/examples/postgresql/secret.yaml b/docs/docs/logstashui/kubernetes/examples/postgresql/secret.yaml index 4e97b5c..fc774a3 100644 --- a/docs/docs/logstashui/kubernetes/examples/postgresql/secret.yaml +++ b/docs/docs/logstashui/kubernetes/examples/postgresql/secret.yaml @@ -8,6 +8,5 @@ stringData: SECRET_KEY: CHANGE-ME-generate-a-django-secret-key LOGSTASHUI_DB_PASSWORD: CHANGE-ME # Shared with Deployment/logstashagent so the embedded node can CSR without enroll. - # Uncomment when applying examples/embedded-agent.yaml. Must match what the - # agent reads. Do not use the compose default in a real cluster. - # LOGSTASHUI_AGENT_CSR_SECRET: CHANGE-ME-shared-with-the-agent + # Must match what the agent reads. Do not use the default in a real cluster. + LOGSTASHUI_AGENT_CSR_SECRET: CHANGE-ME-shared-with-the-agent diff --git a/docs/docs/logstashui/kubernetes/examples/sqlite/configmap.yaml b/docs/docs/logstashui/kubernetes/examples/sqlite/configmap.yaml index 54430dc..edb6b55 100644 --- a/docs/docs/logstashui/kubernetes/examples/sqlite/configmap.yaml +++ b/docs/docs/logstashui/kubernetes/examples/sqlite/configmap.yaml @@ -16,10 +16,12 @@ data: LOGSTASHUI_AGENT_UI_URL: https://logstashui.example.com LOGSTASHUI_DB_ENGINE: sqlite # db.sqlite3 is created at $LOGSTASHUI_DATA_DIR/db.sqlite3 on the PVC. + # # NOT recommended. Automatic product TLS works out of the box. # If you must run the UI and all agent connections over plain HTTP # (no CA, no certs), uncomment. Do not enable without cause. # LOGSTASHUI_INSECURE_HTTP: "true" + # # OpenTelemetry (optional). The image already installs LogstashUI[otel]. # Uncomment to export traces/metrics over OTLP/HTTP (not gRPC — gevent # cannot patch grpcio). Point at a collector that accepts HTTP/protobuf @@ -27,7 +29,3 @@ data: # LOGSTASHUI_OTEL: "true" # OTEL_SERVICE_NAME: logstashui # OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector.observability.svc:4318 - # Embedded simulation agent (optional). Apply examples/embedded-agent.yaml - # after uncommenting this key and LOGSTASHUI_AGENT_CSR_SECRET in the Secret. - # Leave commented if you are not running the in-cluster embedded node. - # LOGSTASH_AGENT_URL: https://logstashagent:9500 diff --git a/docs/docs/logstashui/kubernetes/examples/sqlite/secret.yaml b/docs/docs/logstashui/kubernetes/examples/sqlite/secret.yaml index 150aaa2..13e02d4 100644 --- a/docs/docs/logstashui/kubernetes/examples/sqlite/secret.yaml +++ b/docs/docs/logstashui/kubernetes/examples/sqlite/secret.yaml @@ -7,6 +7,5 @@ type: Opaque stringData: SECRET_KEY: CHANGE-ME-generate-a-django-secret-key # Shared with Deployment/logstashagent so the embedded node can CSR without enroll. - # Uncomment when applying examples/embedded-agent.yaml. Must match what the - # agent reads. Do not use the compose default in a real cluster. - # LOGSTASHUI_AGENT_CSR_SECRET: CHANGE-ME-shared-with-the-agent + # Must match what the agent reads. Do not use the default in a real cluster. + LOGSTASHUI_AGENT_CSR_SECRET: CHANGE-ME-shared-with-the-agent diff --git a/docs/docs/logstashui/kubernetes/index.md b/docs/docs/logstashui/kubernetes/index.md index a6458ac..fc9b083 100644 --- a/docs/docs/logstashui/kubernetes/index.md +++ b/docs/docs/logstashui/kubernetes/index.md @@ -105,7 +105,7 @@ Default engine is SQLite on the PVC. For Postgres or MySQL/MariaDB, set the disc ## Embedded agent (optional) -Compose `--profile embedded` analog: [embedded-agent.yaml](examples/embedded-agent.yaml). Apply a DB tree, uncomment `LOGSTASH_AGENT_URL` and `LOGSTASHUI_AGENT_CSR_SECRET`, then apply the overlay. ClusterIP only (`9500` / `9560` / `9449`). Details: [examples README](examples/README.md#embedded-agent-optional). +This uses the embedded agent image [embedded-agent.yaml](examples/embedded-agent.yaml). Apply a DB tree, set `LOGSTASHUI_AGENT_CSR_SECRET`, then apply the overlay. ClusterIP only (`9500` / `9560` / `9449`). Details: [examples README](examples/README.md#embedded-agent-optional). --- From 93635611ab269841a5a92266adc6110d6a8e634f Mon Sep 17 00:00:00 2001 From: Aaron Mildenstein Date: Fri, 4 Sep 2026 18:18:09 -0600 Subject: [PATCH 62/62] Missed the README in the last commit --- docs/docs/logstashui/kubernetes/examples/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/docs/logstashui/kubernetes/examples/README.md b/docs/docs/logstashui/kubernetes/examples/README.md index 8e4fbae..a226707 100644 --- a/docs/docs/logstashui/kubernetes/examples/README.md +++ b/docs/docs/logstashui/kubernetes/examples/README.md @@ -21,7 +21,7 @@ kubectl apply -f docs/docs/logstashui/kubernetes/examples/sqlite/ Same role as compose `--profile embedded`: an in-cluster LogstashAgent for the pipeline editor Sim target (`embedded · docker`). No enroll. Lab only — prefer an enrolled Simulate agent for serious work. 1. Apply **one** DB tree above. -2. Uncomment `LOGSTASH_AGENT_URL` in that tree's ConfigMap and `LOGSTASHUI_AGENT_CSR_SECRET` in its Secret. Replace the secret placeholder. Do not reuse the compose default. +2. Set `LOGSTASHUI_AGENT_CSR_SECRET` in its Secret. DO NOT use the secret default. 3. Apply the overlay: ```bash @@ -32,7 +32,7 @@ kubectl apply -f docs/docs/logstashui/kubernetes/examples/embedded-agent.yaml If the CSR secret key is still commented, the agent pod is `CreateContainerConfigError` until you uncomment it. -Agent ConfigMap `logstashagent` already sets `LOGSTASH_UI_URL` / `LOGSTASH_URL` to `https://logstashui:8443` and `SIMULATION_MODE=true`. Two extra keys are **commented** (LogstashAgent env, not UI): +Agent ConfigMap `logstashagent` already sets `LOGSTASH_UI_URL` / `LOGSTASH_URL` to `https://logstashui:8443`. Two extra keys are **commented** (LogstashAgent env, not UI): | Env | Default | Do not enable without cause | |---|---|---|