Fix test core dump, add producer metadata, real-time dashboard plugin

.github/instructions/tests.instructions.md

@@ -29,10 +29,20 @@ TEST_CASE("descriptive name", "[tag]") {
 ```
 
 ## Running Tests
-- Build: `ninja -C build`
+- Build: `cmake -B build -G Ninja -DBUILD_TESTS=ON -DPLEXDB_LOG_ENABLED=ON -DPLEXDB_DEBUG=ON && ninja -C build`
 - Run all: `./build/plexdb/plexdb_tests --skip-benchmarks` or `./build/objstore/objstore_tests --skip-benchmarks`
 - Run specific tags: `./build/objstore/objstore_tests --skip-benchmarks "[tagname]"`
 
+## Debugging Test Failures
+- Build with `-DPLEXDB_LOG_ENABLED=ON` so log messages appear in failing tests.
+- All `plexdb::log` messages are routed to Catch2 `UNSCOPED_INFO` by the test log consumer (`test/log_consumer_helper.test.cpp`). When a test fails, all log messages emitted during that test case are printed alongside the failure.
+- To add diagnostic logging in production code, create a `plexdb::log::Producer` and use `plexdb::log::message(producer, Level::Debug, str8)` where `str8` is a `String8`. This is zero-overhead when logging is disabled at build time. Registration is lazy — the producer registers itself on first use.
+- For numeric metrics without string formatting overhead, create a `plexdb::log::Stat` with a `StatType` (`Counter` or `Gauge`) and use `plexdb::log::stat(s, value)`. The stat's name and type are registered automatically on first fire. Default type is `Gauge`.
+- When a consumer registers, all known producers and stat metadata are replayed (catch-up).
+- Custom log consumers can be registered via the plugin ABI (`plexdb_log_register_consumer` in `plexdb/log/log_abi.h`). Write a consumer to filter by producer ID, log level, or stat ID.
+- The OTLP plugin (`objstore/plugins/log_otel/log_otel_plugin.cpp`) exports metrics via OpenTelemetry (OTLP/HTTP JSON).
+- Parse errors are reported via `UNSCOPED_INFO` through `objstore/test/parsers_error_reporter.helper.cppm` and also through the log system (`objstore::log::cql_parse_error` at `Level::Error`).
+
 ## Benchmarks
 - Catch2 supports benchmarks but use them sparingly
 - Always use `--skip-benchmarks` for normal test runs

.github/workflows/copilot-setup-steps.yml

@@ -22,14 +22,22 @@ jobs:
       - name: Install dependencies
         run: |
           sudo apt-get update
-          sudo apt-get install -y ninja-build liburing-dev clang-19 clang-tools-19 nodejs npm
+          sudo apt-get install -y ninja-build liburing-dev clang-19 clang-tools-19 nodejs npm \
+            libgrpc++-dev libprotobuf-dev protobuf-compiler protobuf-compiler-grpc
 
       - name: Configure CMake
         run: cmake -B build -G Ninja -DBUILD_TESTS=ON -DPLEXDB_LOG_ENABLED=ON -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_COMPILER=clang++-19 -DCMAKE_C_COMPILER=clang-19
 
       - name: Build
         run: ninja -C build
 
+      - name: Build OTLP plugin
+        run: |
+          cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Debug \
+            -DCMAKE_CXX_COMPILER=clang++-19 -DCMAKE_C_COMPILER=clang-19
+          ninja -C build
+        working-directory: objstore/plugins/log_otel
+
       - name: Install test_cql.js dependencies
         run: npm install
         working-directory: extra/node

.github/workflows/release.yml

@@ -15,7 +15,8 @@ jobs:
       - name: Install dependencies
         run: |
           sudo apt-get update
-          sudo apt-get install -y ninja-build liburing-dev clang-19 clang-tools-19
+          sudo apt-get install -y ninja-build liburing-dev clang-19 clang-tools-19 \
+            libgrpc++-dev libprotobuf-dev protobuf-compiler protobuf-compiler-grpc
 
       - name: Configure
         run: |
@@ -41,10 +42,20 @@ jobs:
           cp macros/macros.h pkg/plexdb/macros/
           tar -czf plexdb-linux-x64.tar.gz -C pkg plexdb
 
+      - name: Build OTLP plugin
+        run: |
+          cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release \
+            -DCMAKE_CXX_COMPILER=clang++-19 -DCMAKE_C_COMPILER=clang-19
+          ninja -C build
+        working-directory: objstore/plugins/log_otel
+
       - name: Package objstore
         run: |
           cp build/objstore/objstore_server objstore-linux-x64
 
+      - name: Package OTLP plugin
+        run: cp objstore/plugins/log_otel/build/libobjstore_log_otel.so .
+
       - name: Upload objstore
         uses: actions/upload-artifact@v4
         with:
@@ -57,6 +68,12 @@ jobs:
           name: plexdb-linux-x64
           path: plexdb-linux-x64.tar.gz
 
+      - name: Upload OTLP plugin
+        uses: actions/upload-artifact@v4
+        with:
+          name: libobjstore_log_otel
+          path: libobjstore_log_otel.so
+
   release:
     needs: build
     runs-on: ubuntu-latest
@@ -74,9 +91,15 @@ jobs:
         with:
           name: plexdb-linux-x64
 
+      - name: Download OTLP plugin
+        uses: actions/download-artifact@v4
+        with:
+          name: libobjstore_log_otel
+
       - name: Create release
         uses: softprops/action-gh-release@v2
         with:
           files: |
             objstore-linux-x64
             plexdb-linux-x64.tar.gz
+            libobjstore_log_otel.so

.github/workflows/test.yml

@@ -16,14 +16,29 @@ jobs:
       - name: Install dependencies
         run: |
           sudo apt-get update
-          sudo apt-get install -y ninja-build liburing-dev clang-19 clang-tools-19
+          sudo apt-get install -y ninja-build liburing-dev clang-19 clang-tools-19 \
+            libgrpc++-dev libprotobuf-dev protobuf-compiler protobuf-compiler-grpc
 
       - name: Configure
-        run: cmake -B build -G Ninja -DBUILD_TESTS=ON -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_COMPILER=clang++-19 -DCMAKE_C_COMPILER=clang-19
+        run: cmake -B build -G Ninja -DBUILD_TESTS=ON -DPLEXDB_LOG_ENABLED=ON -DPLEXDB_DEBUG=ON -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_COMPILER=clang++-19 -DCMAKE_C_COMPILER=clang-19
 
       - name: Build
         run: ninja -C build
 
+      - name: Build OTLP plugin
+        run: |
+          cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Debug \
+            -DCMAKE_CXX_COMPILER=clang++-19 -DCMAKE_C_COMPILER=clang-19
+          ninja -C build
+        working-directory: objstore/plugins/log_otel
+
+      - name: Build file plugin
+        run: |
+          cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Debug \
+            -DCMAKE_CXX_COMPILER=clang++-19 -DCMAKE_C_COMPILER=clang-19
+          ninja -C build
+        working-directory: objstore/plugins/log_file
+
       - name: Run plexdb tests
         run: ./build/plexdb/plexdb_tests --skip-benchmarks
 

.gitignore

@@ -4,6 +4,8 @@ build/
 out/
 _codeql_detected_source_root
 *.db
+*.stats
 objstore.pid
 **/.venv
-**/node_modules
+**/node_modules
+**/__pycache__

AGENTS.md

@@ -14,4 +14,16 @@
 
 ## Testing
 - Tests use [Catch2](https://github.com/catchorg/Catch2/blob/devel/docs/). Create tests for your changes.
-- Test executables: `build/plexdb/plexdb_tests`, `build/objstore/objstore_tests`. Use `--skip-benchmarks` for normal runs.
+- Test executables: `build/plexdb/plexdb_tests`, `build/objstore/objstore_tests`. Use `--skip-benchmarks` for normal runs.
+
+## Logging & debugging test failures
+- Build with `-DPLEXDB_LOG_ENABLED=ON` to enable the structured logging system.
+- All log messages are routed to Catch2's `UNSCOPED_INFO` via the test log consumer (`test/log_consumer_helper.test.cpp`). They appear automatically in the output of any failing test case.
+- The log system uses levels: `Trace`, `Debug`, `Info`, `Warn`, `Error` (see `plexdb::log::Level`).
+- Registration is lazy: `Producer` and `Stat` objects register themselves on first use, so construction order does not matter.
+- To add diagnostic logging, create a `plexdb::log::Producer` and call `plexdb::log::message(producer, Level::Debug, str8)` where `str8` is a `String8`.
+- For structured numeric metrics, create a `plexdb::log::Stat` with a `StatType` (`Counter` or `Gauge`) and call `plexdb::log::stat(s, value)` — no string formatting overhead. The stat's name and type are registered automatically on first fire.
+- Stat types: `StatType::Counter` for monotonically increasing cumulative values, `StatType::Gauge` for point-in-time measurements. Default is `Gauge`.
+- When a consumer registers, all known producers and stat metadata (including stat types) are replayed (catch-up), so the consumer always has a complete view.
+- The plugin system (`plexdb/log/log_abi.h`) allows custom consumers. Register via `plexdb_log_register_consumer` to filter or redirect logs. See `objstore/plugins/log_file/log_file_plugin.cpp` for a reference plugin.
+- The OTLP plugin (`objstore/plugins/log_otel/log_otel_plugin.cpp`) exports metrics via OpenTelemetry (OTLP/gRPC). Built as a standalone project in `objstore/plugins/log_otel/`; load via `LD_PRELOAD`. Uses plaintext gRPC (`use_ssl_credentials = false`).

README.md

@@ -10,6 +10,7 @@ Pre-built binaries and the plexdb static library are attached to each [GitHub re
 |---|---|
 | `objstore` | Server executable (Linux x86-64) |
 | `plexdb-linux-x64.tar.gz` | Static library + C++20 module sources |
+| `libobjstore_log_otel.so` | OTLP/gRPC metrics plugin (load via `LD_PRELOAD`) |
 
 ## Usage
 
@@ -32,3 +33,29 @@ cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release \
       -DCMAKE_CXX_COMPILER=clang++-19 -DCMAKE_C_COMPILER=clang-19
 ninja -C build
 ```
+
+## Plugins
+
+### OTLP metrics plugin
+
+Exports structured log stats to an OpenTelemetry collector via OTLP/gRPC. Requires gRPC and protobuf system libraries.
+
+```sh
+cd objstore/plugins/log_otel
+cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
+ninja -C build
+```
+
+Load at server startup:
+
+```sh
+LD_PRELOAD=objstore/plugins/log_otel/build/libobjstore_log_otel.so \
+  PLEXDB_OTLP_ENDPOINT=localhost:4317 \
+  ./build/objstore/objstore_server <db_path>
+```
+
+| Env var | Default | Description |
+|---|---|---|
+| `PLEXDB_OTLP_ENDPOINT` | `localhost:4317` | OTLP/gRPC collector endpoint |
+| `PLEXDB_OTLP_INTERVAL_MS` | `10000` | Export interval in milliseconds |
+| `PLEXDB_OTLP_SERVICE` | `plexdb` | Service name reported to collector |

TODO.md

@@ -1,6 +1,5 @@
 # TODO
-- Parse CQL in objstore with lexy
-    - Parse cache with string hash as key
+- Parse cache with string hash as key
 - Query planning
     - Very simple for basic CQL commands
     - Again cache planning result if complex