This repository performs automated daily system tests of ReGaHss - the HomeMatic CCU "Logic Layer" engine. It uses a mocha-based node.js testing framework to test the ReGaHss binaries published in OpenCCU-Base, the component base layer of OpenCCU. Beside testing common corner cases of the embedded scripting language of ReGaHss, this testing framework also tests program/timer execution (including DST and leap year boundaries via libfaketime), the XML-RPC interface and checks for known security vulnerabilities.
- The required parts of an OpenCCU-Base revision (
bin/<arch>/ReGaHss,lib/<arch>/libXmlRpc.so,lib/<arch>/libxmlparser.so,www/and the CMake build system plus library sources) are fetched via a shallow sparse checkout (scripts/fetch-openccu-base.sh). - ReGaHss is either run with the prebuilt
libXmlRpc/libxmlparserof OpenCCU-Base or with these libraries rebuilt from the OpenCCU-Base sources (scripts/build-libs.sh), optionally instrumented with AddressSanitizer/UndefinedBehaviorSanitizer and gcov coverage. - ReGaHss and its runtime environment (
rega.conf,InterfacesList.xml, the prebuilt testhomematic.regadom, dummy hook scripts) are installed into a disposable docker image (scripts/install-regahss.sh). - Each test file (
test/*.js) starts its own ReGaHss process (optionally together with the simulated CCU interface processes or under libfaketime) and interacts with it via the ReGa script interface, its XML-RPC/BIN-RPC server and its log output. lib/ccu-simulator.jssimulates the interface processes of a CCU:BidCos-RF(rfd, BIN-RPC),HmIP-RF(HmIPServer, XML-RPC) andVirtualDevices(XML-RPC,/groups). They accept theinitof ReGaHss, announce their devices (test/fixtures/devices.json) vialistDevices/newDevices, answergetParamsetDescription/getLinks/setValueetc. and allow tests to send events, add/delete devices or restart an interface.- Every ReGaHss instance runs with its own working directory (
rega.conf,homematic.regadom,InterfacesList.xml) and its own ports (lib/rega-instance.js), so that the test files can be run in parallel worker processes (REGA_JOBS). - ReGaHss is started directly (without wrapper processes), so the test harness notices crashes immediately: pending tests fail at once with the exit signal and the last log lines of ReGaHss instead of running into timeouts.
# build the test image for the current OpenCCU-Base main branch
docker build -t regahss-test --build-arg REGA_ARCH=x86_64-linux-gnu --build-arg BASE_REF=main .
# run the complete test suite, a summary (results/summary.md), a JUnit report
# (results/junit.xml) and all other reports are written to results/
docker run --rm --init -v "$PWD/results:/results" regahss-test
# run the test files in 4 parallel worker processes and the timer tests in
# real time (default: 10 times accelerated clock)
docker run --rm --init -e REGA_JOBS=4 -e REGA_FAKETIME_RATE=1 -v "$PWD/results:/results" regahss-test
# test ReGaHss with sanitizer instrumented libXmlRpc/libxmlparser built from source
# (results/sanitizer: ASan/UBSan reports, results/coverage: gcov line coverage)
docker build -t regahss-test:asan --build-arg REGA_LIBS=asan .
docker run --rm --init -v "$PWD/results:/results" regahss-test:asan
# test another ReGaHss binary (e.g. an own build, also instrumented with
# ASan/UBSan) instead of the one of OpenCCU-Base (path in the build context)
docker build -t regahss-test:custom --build-arg REGA_LIBS=asan --build-arg REGA_BINARY=regahss-binary/ReGaHss .
# run only specific test files
docker run --rm --init regahss-test npx mocha test/02-script-doku-teil1.js test/13-fixed-bugs.js
# additionally compare the results of the script corpus with the ReGaHss of
# another OpenCCU-Base revision (results/differential.md, informational)
docker build -t regahss-test --build-arg REF_BASE_REF=<sha> .
# show the ReGaHss output while running the tests
docker run --rm --init -e REGA_OUTPUT=1 regahss-test npx mocha test/01-rega-startup.jsBuild arguments:
| Argument | Default | Description |
|---|---|---|
REGA_ARCH |
x86_64-linux-gnu |
ReGaHss architecture to test: x86_64-linux-gnu, i686-linux-gnu, aarch64-linux-gnu or arm-linux-gnueabihf (the ARM binaries run with the qemu user mode emulator, prebuilt libraries only) |
BASE_REF |
main |
OpenCCU-Base branch, tag or commit SHA to take ReGaHss from |
BASE_REPO |
https://github.com/OpenCCU/OpenCCU-Base.git |
OpenCCU-Base repository (e.g. a fork) |
REF_BASE_REF |
– | OpenCCU-Base revision with a reference ReGaHss for differential tests of the script corpus (the CI uses the revision before the last change of the ReGaHss binary) |
REGA_LIBS |
prebuilt |
libXmlRpc/libxmlparser to run ReGaHss with: prebuilt (shipped with OpenCCU-Base), source (release build from the OpenCCU-Base sources, incl. informational abidiff report vs. the prebuilt libraries) or asan (debug build with ASan/UBSan and gcov coverage, x86_64-linux-gnu only) |
REGA_BINARY |
– | ReGaHss binary in the build context to test instead of the one of OpenCCU-Base, e.g. a build of a ReGaHss pull request. A binary instrumented with ASan/UBSan (x86_64-linux-gnu only) is run like the asan variant (sanitizer reports fail the test run, faketime based timer tests are pending) |
For all variants the build verifies that ReGaHss resolves all its symbols with the selected libraries (ldd -r). With asan the sanitizer runtimes are only preloaded into the ReGaHss process (REGA_PRELOAD), any ASan/UBSan report makes the test run fail and ReGaHss is stopped gracefully (SIGTERM) to let it write its coverage data. As libfaketime and the preloaded ASan runtime deadlock at ReGaHss startup (the ASan allocator calls clock_gettime() with its lock held, which triggers the lazy initialization of libfaketime, which in turn allocates memory), the faketime based timer tests are reported as pending for asan (they are run by all other variants).
ReGaHss sporadically aborts when it is stopped with SIGTERM (terminate called without an active exception: the forced unwind of a thread cancelled by Halt() ends in std::terminate()). This known shutdown race only causes a warning in the summary when ReGaHss is stopped at the end of a test file (its core dump is kept as shutdown-abort.core*, but does not fail the test run), whereas test/15-rega-lifecycle.js checks the shutdown on SIGTERM strictly.
The output of every ReGaHss instance started by the tests is kept in results/logs.tar.gz. If ReGaHss crashes with a core dump (requires docker run --ulimit core=-1 and a relative kernel.core_pattern of the host such as core.%e.%p), the core dumps are moved to results/cores together with a backtrace (gdb) and the ReGaHss binary, and the test run fails.
The timer tests (test/07*-timer-faketime-*.js) run with a 10 times accelerated faked clock by default (REGA_FAKETIME_RATE), which reduces the runtime of the whole test suite from ~28 to ~3 minutes. For the 32-bit ReGaHss (i686-linux-gnu, arm-linux-gnueabihf) they run in real time, as libfaketime does not accelerate the waits of its timer thread (glibc time64 ABI) and the timers would fire too late. The nightly CI run executes them in real time on all architectures (REGA_FAKETIME_RATE=1). Real time runs benefit from parallel execution (REGA_JOBS=4), as the timer tests are split into four files. Each timer test starts ReGaHss 12 (faked) seconds before the first expected execution; if ReGaHss calculates its timers less than 2 seconds before it (e.g. a slow cold start of an emulated ReGaHss with the accelerated clock while other test files start as well), it skips this execution correctly, so the test restarts ReGaHss at the same faked time instead (at most twice, reported as warning in the summary).
The ReGaHss binaries of OpenCCU-Base for aarch64-linux-gnu and arm-linux-gnueabihf are tested with the qemu user mode emulator (qemu-user-static). ReGaInstance detects the architecture from the ELF header and runs a binary of a foreign architecture through the matching emulator (REGA_EMULATOR overrides it, none disables it); the preload libraries (line buffering, libfaketime) are built for the architecture of ReGaHss and passed to the emulated process. The docker image installs qemu-user-static and the runtime libraries of the architecture; only the prebuilt libraries are supported for the ARM architectures. Emulated runs are several times slower, so the timeouts of the tests are scaled accordingly.
Images built by the CI for the master branch (prebuilt libraries) are published as ghcr.io/openccu/regahss-test:<arch>-<main|release|commit>.
scripts/install-regahss.sh installs ReGaHss to /bin, /etc, /www and /usr/local/lib/regahss and therefore must only be used in disposable environments (VM, container). Requires node.js >= 22, libfaketime >= 0.9.13 built for the ReGaHss architecture (the 32-bit ReGaHss uses the glibc time64 ABI), the timezone Europe/Berlin and a library making the output of ReGaHss line buffered: for x86_64-linux-gnu the libstdbuf.so of coreutils is used automatically, for the other architectures build src/linebuf.c (gcc -m32 -shared -fPIC -o /usr/local/lib/regahss-test/liblinebuf.so src/linebuf.c, or with the cross compiler of the target architecture). The ARM binaries additionally need qemu-user-static and the runtime libraries of their architecture. The Y2038 tests use src/timeshift.c (REGA_TIMESHIFT_LIB):
scripts/fetch-openccu-base.sh https://github.com/OpenCCU/OpenCCU-Base.git main x86_64-linux-gnu /tmp/openccu-base
sudo scripts/install-regahss.sh /tmp/openccu-base x86_64-linux-gnu
# or with libraries built from source (requires cmake and g++)
scripts/build-libs.sh /tmp/openccu-base x86_64-linux-gnu source /tmp/regahss-libs
sudo scripts/install-regahss.sh /tmp/openccu-base x86_64-linux-gnu /tmp/regahss-libs/lib
npm ci
sudo env "PATH=$PATH" TZ=Europe/Berlin npm test| Variable | Default | Description |
|---|---|---|
REGA_BIN |
/bin/ReGaHss |
ReGaHss binary to test |
REGA_LABEL |
<arch>@<commit>/<libs> |
label shown in the test titles |
REGA_JOBS |
1 |
number of test files to run in parallel (mocha --parallel) |
REGA_FAKETIME_RATE |
10 (1 for 32-bit) |
speed of the faked clock for the timer tests (1: real time) |
REGA_EMULATOR |
auto | emulator to run ReGaHss with (qemu user mode for foreign architectures, none to force native execution) |
REGA_TIMESHIFT_LIB |
auto | preload library shifting the realtime clock (src/timeshift.c, used by the Y2038 tests beyond 2038 on 32-bit platforms) |
REGA_FUZZ_ITERATIONS |
200 |
number of fuzzing inputs per interface (test/21-fuzzing.js) |
REGA_FUZZ_SEED |
1 |
seed of the fuzzing inputs (reproducible) |
REGA_FUZZ_DIR |
– | directory to save the last fuzzing inputs to if ReGaHss crashes (for reproduction) |
REGA_SOAK_SECONDS |
30 |
duration of the stability/soak test (test/22-stability.js) |
REGA_PRELOAD |
– | libraries to preload into the ReGaHss process only (set automatically for asan) |
REGA_STOP_SIGNAL |
KILL |
signal to stop ReGaHss with after each test file (TERM for asan) |
REGA_LOG_DIR |
– | directory to write the output of each ReGaHss instance to (results/logs in the docker image) |
REGA_JUNIT_FILE |
– | file to write a JUnit XML report to (results/junit.xml in the docker image) |
REGA_WARNINGS_FILE |
– | file to append warnings of the test run to, e.g. known aborts of ReGaHss while stopping it (results/warnings.md in the docker image, shown in the summary) |
REGA_SKIPPED_FILE |
– | file to append the tests to which were skipped because the ReGaHss (or libXmlRpc) under test is too old (results/skipped.md in the docker image, shown in the summary) |
REGA_WORK_DIR |
$TMPDIR/regahss-test |
base directory of the working directories of the ReGaHss instances |
REGA_PORT_BASE |
20000 |
first port used by the ReGaHss instances (10 ports per parallel worker) |
REGA_LINEBUF_LIB |
auto | preload library making the ReGaHss output line buffered (src/linebuf.c or coreutils' libstdbuf.so) |
REGA_FAKETIME_LIB |
auto | libfaketime.so.1 to fake the time of ReGaHss with |
REGA_RESULTS_DIR |
/results |
directory for the summary, JUnit report, ReGaHss logs, core dumps, sanitizer reports and coverage (docker image) |
REGA_OUTPUT |
– | set to 1 to show the ReGaHss output |
SIM_OUTPUT |
– | set to 1 to show the output of the interface simulator (RPC calls in both directions) |
REGA_CORPUS |
– | only run the script corpus files matching this regular expression |
REGA_CORPUS_RECORD |
– | set to 1 to write the results of the ReGaHss under test into the script corpus files |
REGA_REF_BIN |
– | reference ReGaHss to execute the script corpus with as well (set automatically in the docker image if built with REF_BASE_REF) |
REGA_REF_LIB_DIR |
– | directory with the libXmlRpc/libxmlparser of the reference ReGaHss |
REGA_DIFF_REPORT |
– | markdown file to write the differences between ReGaHss and the reference ReGaHss to |
Each test file starts and stops its ReGaHss instance via initTest() and cleanupTest() of lib/helper.js. Tests use async functions and fail immediately if ReGaHss crashes:
const {rega, regaLabel, initTest, cleanupTest, waitForRega, waitForSim} = require('../lib/helper.js');
describe('Running my-test.js [' + regaLabel + ']', function () {
// start ReGaHss (options: sim, faketime, rate, rpc, nocopy)
initTest({sim: true});
describe('running tests', function () {
it('should execute a script', async function () {
const {output, objects} = await rega.exec('string s = "hello"; WriteLine(s);');
output.should.equal('hello\r\n');
objects.s.should.equal('hello');
});
it('should trigger a program', async function () {
// wait for log output caused by the script (not for earlier lines)
await Promise.all([
waitForSim(/setValue rfd BidCoS-RF:12 PRESS_LONG true/, {buffered: false}),
rega.exec('dom.GetObject(1237).State(true);')
]);
});
});
cleanupTest();
});Regression tests of a ReGaHss fix declare the first ReGaHss version containing the fix with requireRegaVersion(). With an older ReGaHss (e.g. the release revision of OpenCCU-Base until OpenCCU updates it) they are skipped and listed in the summary instead of failing:
// a single test
it('should not crash on a huge index', async function () {
requireRegaVersion(this, 'R1.00.0388.0257');
// ...
});
// all tests of a suite (incl. nested suites)
describe('DayProfileEntry() index range tests', function () {
before(function () {
requireRegaVersion(this, 'R1.00.0388.0257');
});
// ...
});after() hooks of a skipped suite still run (before() hooks registered after the skipping one do not).
Fixes in the runtime libraries of ReGaHss are tested accordingly with requireXmlRpcVersion(). It compares the libXmlRpc version which the running ReGaHss reports in the Server header of its XML-RPC responses (XMLRPC++ 0.12), so it can only be called in a test (not in before()) of a test file which has started ReGaHss:
it('should not crash on deeply nested XML-RPC parameters', async function () {
await requireXmlRpcVersion(this, '0.12');
// ...
});test/corpus/*.rega contains ReGa scripts together with their expected output, the resulting variables and the script errors. Each file is executed by a ReGaHss instance of its own (test/16-script-corpus.js):
!! fixed-time: 2024-06-15 12:34:56 CEST
#### string concatenation
string s = "a" # "b";
WriteLine(s);
---- output
ab
---- vars
s=ab
----
!! fixed-time: <date>freezes the clock of ReGaHss (libfaketime) for all cases of the file- each case starts with
#### <name>followed by the script, the expectedoutput(every line ends with CR/LF;\r,\n,\tand\\are escaped and a trailing\cmeans that the output does not end with a line break), the expectedvars(values of the script variables) and, only if there are any, the expectederrors(script errors logged by ReGaHss, normalized) - new cases can be added without expectations and recorded with
REGA_CORPUS_RECORD=1 npx mocha test/16-script-corpus.js; the recorded results must be reviewed before committing them
With REGA_REF_BIN every case is executed by a reference ReGaHss as well and the differences are written to a markdown report (informational, they do not make the tests fail). The CI uses the ReGaHss of the OpenCCU-Base revision before the last change of the binary as reference and adds the report to the job summary.
lib/program-builder.js generates ReGa scripts creating programs (conditions on system variables, datapoints and time modules, destinations with delays, else-if/else rules) and time modules (incl. astro time modules with offsets to sunrise/sunset) the same way the WebUI does, e.g.:
const {programScript} = require('../lib/program-builder.js');
await rega.exec(programScript({
name: 'Key pressed',
rules: [{
// conditions: OR of AND groups
conditions: [[{datapoint: 'BidCos-RF.BidCoS-RF:40.PRESS_SHORT', value: true, trigger: 'update'}]],
destinations: [{sysvar: 'VarString1', value: 'key pressed', delay: 3}]
}]
}));lib/fuzzer.js generates deterministic (seeded) inputs for the interfaces of ReGaHss — BIN-RPC and XML-RPC requests to its RPC server (libXmlRpc/libxmlparser), HTTP requests to its web server and ReGa scripts — from valid via slightly broken to heavily byte-mutated. test/21-fuzzing.js sends them to a running ReGaHss and checks after every batch that it is still alive and answers on both servers; a crash is detected immediately and the last inputs are saved for reproduction (REGA_FUZZ_DIR). With the sanitizer instrumented libraries (asan) memory errors in the libraries are reported as well. The same seed always produces the same inputs (REGA_FUZZ_SEED), so a finding is reproducible.
The fuzzer immediately found an unbounded recursion in libXmlRpc: a system.multicall which contains another system.multicall makes XmlRpcServerConnection::executeMulticall() recurse until the stack is exhausted and ReGaHss crashes (SIGSEGV), reachable unauthenticated via the RPC port (OpenCCU/OpenCCU#4384). test/21-fuzzing.js therefore fails until ReGaHss is fixed.
test/20-y2038.js starts ReGaHss shortly before 2038-01-19 03:14:07 UTC (2³¹ seconds since the epoch, the overflow of a signed 32-bit time_t) and checks that time modules, programs, timestamps of system variables, the communication with the interfaces and the persistence keep working past the overflow — relevant for the 32-bit platforms (i686, armhf), whose ReGaHss is built for the 64-bit time_t of the glibc time64 ABI. As libfaketime keeps the faked time in a 32-bit time_t for 32-bit programs and cannot fake a time beyond 2038, the realtime clock is shifted with src/timeshift.c instead (the monotonic clock is left untouched and absolute realtime timeouts are shifted back, so waits still take as long as intended).
test/22-stability.js is a soak test: it executes scripts, sends device events and lets a program run for REGA_SOAK_SECONDS while sampling the resident memory, the number of threads and open file descriptors of ReGaHss (ReGaInstance.resourceUsage()), and checks that ReGaHss stays responsive and does not leak. The nightly CI run uses a longer duration.
test/23-script-cache.js checks the caches of the script engine of ReGaHss: Call() of functions in .fn files together with system.ClearCache() and system.CacheMode(), system.ClearCache() called by cached code itself, and concurrent first requests of a page, which must all return the complete page (OpenCCU/OpenCCU#2638, OpenCCU/OpenCCU#1467). The required .fn/.htm files (test/fixtures/www/cache-test) are served from a document root of their own (the WebUI files linked plus the fixtures, ReGaInstance option httpDocPath). The tests are skipped for ReGaHss versions before R1.00.0388.0264.
The CI workflow runs on every push/pull request and nightly. It tests the ReGaHss binaries for x86_64-linux-gnu, i686-linux-gnu, aarch64-linux-gnu and arm-linux-gnueabihf (the ARM architectures with qemu) of
main- the current HEAD of OpenCCU-Base andrelease- the OpenCCU-Base revision OpenCCU currently builds its firmware with (OPENCCU_BASE_VERSIONin openccu-base.mk),
whereas identical revisions are only tested once. Each revision is tested with the prebuilt libraries on all four architectures, the source built libraries on x86_64-linux-gnu/i686-linux-gnu and the asan variant on x86_64-linux-gnu. Every test job adds a summary (versions, results per test file and failed tests, core dumps, ABI changes, sanitizer reports, library coverage) to the GitHub job summary and uploads all results (incl. JUnit report, ReGaHss logs and core dumps) as artifact. The test files run in four parallel worker processes and the timer tests with a 10 times accelerated clock (64-bit), the nightly run executes them in real time and fuzzes/soaks more thoroughly. A manual run (workflow_dispatch) allows to test an additional OpenCCU-Base ref (base_ref) and to additionally run the legacy tests against the binaries of the old OCCU repository (legacy_occu, see legacy/).
The test job is a reusable workflow which can be called from OpenCCU-Base to test a pull request (including the libXmlRpc/libxmlparser sources it changes) before it is merged:
# .github/workflows/regahss-test.yml in OpenCCU-Base
name: ReGaHss-Test
on:
pull_request:
permissions:
contents: read
jobs:
regahss-test:
strategy:
fail-fast: false
matrix:
include:
- {arch: x86_64-linux-gnu, libs: source}
- {arch: x86_64-linux-gnu, libs: asan}
- {arch: i686-linux-gnu, libs: source}
uses: OpenCCU/ReGaHss-Test/.github/workflows/regahss-test.yml@master
with:
base_repo: ${{ github.event.pull_request.head.repo.clone_url }}
base_ref: ${{ github.event.pull_request.head.sha }}
base_name: pr-${{ github.event.pull_request.number }}
arch: ${{ matrix.arch }}
libs: ${{ matrix.libs }}
test_repo: OpenCCU/ReGaHss-Test
test_ref: masterThe reusable workflow can also test a ReGaHss binary built by the calling workflow run instead of the one of OpenCCU-Base, e.g. a ReGaHss pull request built with ASan/UBSan. The calling workflow uploads the binary (file ReGaHss) as artifact and passes its name as rega_artifact. As a called workflow runs in the context of the calling run, the binary and all results (logs, core dumps, sanitizer reports) are only accessible there, i.e. a non-public repository does not disclose them:
jobs:
build-asan:
runs-on: ubuntu-24.04
steps:
# ... build ReGaHss with -fsanitize=address,undefined ...
- uses: actions/upload-artifact@v7
with:
name: ReGaHss-asan
path: ReGaHss
regahss-test:
needs: build-asan
uses: OpenCCU/ReGaHss-Test/.github/workflows/regahss-test.yml@<sha>
with:
base_ref: main
arch: x86_64-linux-gnu
libs: asan
rega_artifact: ReGaHss-asan
test_repo: OpenCCU/ReGaHss-Test
test_ref: <sha>ReGaHss is started with a prebuilt homematic.regadom which contains the following variables, programs and objects:
- VarBool1 - id: 1237
- VarEnum1
- VarNum1
- VarString1
- Bool1OnFalse - BidCoS-RF:13 PRESS_LONG
- Bool1OnTrue - BidCoS-RF:12 PRESS_LONG
- Bool1OnFalseUpdate - BidCoS-RF:14 PRESS_LONG
- Bool1OnTrueUpdate - BidCoS-RF:15 PRESS_LONG
- TimeEveryMinute - id: 1302 - BidCoS-RF:50 PRESS_LONG
- Time0100 - id: 1314 - BidCoS-RF:11 PRESS_LONG
- Time0130 - id: 1430 - BidCoS-RF:20 PRESS_LONG
- Time0155 - id: 1458 - BidCoS-RF:21 PRESS_LONG
- Time0200 - id: 1470 - BidCoS-RF:22 PRESS_LONG
- Time0205 - id: 1498 - BidCoS-RF:23 PRESS_LONG
- Time0230 - id: 1510 - BidCoS-RF:24 PRESS_LONG
- Time0255 - id: 1522 - BidCoS-RF:25 PRESS_LONG
- Time0300 - id: 1534 - BidCoS-RF:26 PRESS_LONG
- Time0305 - id: 1546 - BidCoS-RF:27 PRESS_LONG
- Time0330 - id: 1558 - BidCoS-RF:28 PRESS_LONG
- Key16Key17 - on BidCoS-RF:16 PRESS_LONG => BidCoS-RF:17 PRESS_LONG
- Key1 - on BidCos-RF:1 PRESS_SHORT => BidCoS-RF:2 PRESS_LONG
- OpenCCU
- OpenCCU-Base
- OCCU (legacy)
- hm-simulator (formerly used rfd/HmIPServer simulator, source of the HmIP device descriptions in
test/fixtures/devices.json) - ccu x86 docker image (used for creation of the prebuilt homematic.regadom)
Help and Feedback highly appreciated, Pull Requests Welcome! :-)
MIT (c) 2017-2026 Jens Maus, Sebastian Raff