Running parallel instances using namespaces

@hdiethelm

The list of tests is getting very long. We can either change the code to allow port/key/address shifting or, for example, Linux namespaces (using unshare).

@hdiethelm has tested a bit already. Any suggestions and links are welcome.

See also #2722

What I managed is to get two linuxcnc instances running using:
sudo unshare --net --ipc su $USER -c "../scripts/linuxcnc"
and removing the lockfile check in scripts/linuxcnc.in.

--net separates the network namespaces and also the named fifo in rtapi_app. The named fifo can also be set with RTAPI_FIFO_PATH
--ipc separates the shared memory namespaces.

Interestingly, if I close the second axis gui, the first one closes also. There is some hidden connection somewhere, might be close window by name or a PID file.

Using overlayfs and mount namespaces, it should be possible to reuse existing files but new files like the lockfile and possibly others are volatile and per instance. But I did not continue investigating future.

The real problem is the sudo. I never allow a build/test process to use sudo.

The linuxcnc script kills processes by name and kills everything it finds. No process currently creates PID files. That was one thing I was working on because you only want to kill what you created.

The real problem is the sudo. I never allow a build/test process to use sudo.

You can get rid of the sudo by creating a user namespace where you are root inside and from there creating the net and ipc namespace. This is how podman works. I need to dig up the doc again.

The linuxcnc script kills processes by name and kills everything it finds. No process currently creates PID files. That was one thing I was working on because you only want to kill what you created.

This can be isolated with a pid namespace.

So it should all be possible somehow to create container like separation for the tests but it will be a bit complex. Also test scheduling and result retrieval needs some effort. That's why I did not continue.

Finalizing #2722 sounds also not simple but it could help for more than only the tests.

That sounds a bit like what docker is doing, maybe that could be a way forward that doesn't involve dealing with all the details of namespace handling?

I've been running parallel instances locally for a while to test different branches side by side. I use bubblewrap, which works as a normal user with no sudo:

#!/bin/bash
exec bwrap --dev-bind / / --unshare-ipc --unshare-pid --unshare-net \
    --proc /proc --tmpfs /tmp --tmpfs /dev/shm --die-with-parent -- "$@"

Then sandbox.sh linuxcnc -r config.ini or sandbox.sh halrun ....

--unshare-ipc plus a private /dev/shm separates the HAL/RTAPI shared memory.
--unshare-net separates the rtapi_app FIFO and any abstract sockets.
--unshare-pid means the kill-by-name in the linuxcnc script only sees its own processes. That should also cover the "closing one AXIS closes the other" effect.
A private /tmp gives each instance its own lockfile, so scripts/linuxcnc.in needs no changes.

Limits: my runs are headless. The private /tmp hides /tmp/.X11-unix, so GUI tests would need that socket bound back in or an xvfb inside the sandbox. It isn't realtime either, because setuid gets no privileges inside. It also needs unprivileged user namespaces, which some distros restrict (Ubuntu 24.04 does it through AppArmor).

@grandixximo

That sounds a bit like what docker is doing, maybe that could be a way forward that doesn't involve dealing with all the details of namespace handling?

Exactly, but with docker, you can not share the root file system. Normally, this is what you want. But with linuxcnc and all the dependency's, the image is ~2.5GB in size, so it could be slower at the end instead of faster due to the image has to be built or downloaded in CI.

@grandixximo Thanks for the tip with bwrap, it's way easier than using unshare.

The following command does something similar with unshare:
unshare --user --map-users=1000,1000,1 --map-groups=1000,1000,1 --fork --pid --net --ipc --mount-proc ../scripts/linuxcnc

By using user namespace, sudo is not needed. With overlays, tmp can be isolated for new files but shared for existing ones. This works with X11 and the lockfile:
bwrap --dev-bind / / --overlay-src /tmp --tmp-overlay /tmp --dev-bind /tmp/.X11-unix /tmp/.X11-unix --unshare-ipc --unshare-pid --unshare-net --proc /proc --die-with-parent -- ../scripts/linuxcnc

Somehow, the second linuxcnc still manages to kill the first one. There is still something interfering.

Found it: axis-remote --quit called from linuxcnc at shutdown goes trough the X server. This can not be isolated or there is just no GUI.

For tests, this should be no issue, X must be isolated anyway to use an xvfb for the gui tests.

But it is annoying otherwise, it kills just an axis gui running on the same X server. Sometimes the right one, sometimes the wrong one.

Found it: axis-remote --quit called from linuxcnc at shutdown goes trough the X server. This can not be isolated or there is just no GUI.
But it is annoying otherwise, it kills just an axis gui running on the same X server. Sometimes the right one, sometimes the wrong one.

It is actually wrong in a parallel program context way. This should connect to a socket for the instance you want to talk to and not the container.

For tests, this should be no issue, X must be isolated anyway to use an xvfb for the gui tests.

Yes, for tests it works fine because they use a dedicated xvfb instance, but otherwise, it needs to be separated out.

Found it: axis-remote --quit called from linuxcnc at shutdown goes trough the X server. This can not be isolated or there is just no GUI.
But it is annoying otherwise, it kills just an axis gui running on the same X server. Sometimes the right one, sometimes the wrong one.

It is actually wrong in a parallel program context way. This should connect to a socket for the instance you want to talk to and not the container.

Indeed, it just operates on the window named axis trough X11. If I have 2 running and change to axis #2 in axis-remote, it closes the second window.

xwininfo -tree -root | grep axis
0x5200019 "axis.ngc - AXIS 2.10.0~pre1 auf LinuxCNC-HAL-SIM-AXIS": ("axis #2" "Axis")  847x752+5+29  +3169+391
0x4c00019 "axis.ngc - AXIS 2.10.0~pre1 auf LinuxCNC-HAL-SIM-AXIS": ("axis" "Axis")  847x752+5+29  +2463+386

Looks like linuxcnc is designed from the ground up to run one instance. Using bwrap should work but it feels more like a workaround than a real solution.

It's probably still worth integrating if there are no other show stoppers going trough some files. The parallel test run will need more or less the same framework, independent of the bwrap isolation. As soon as LINUXCNC_INSTANCE is available, the bwrap isolation can just be removed.

So, first prototype: https://github.com/hdiethelm/linuxcnc-fork/tree/tests_parallel

time scripts/runtests -p -j 128
...
real	1m12.127s
user	0m1.634s
sys	0m1.694s

Most tests spend most time waiting, so it's a massive speedup and given enough ram, all tests can pretty much be run in parallel...
It seams to work. However, I had to isolate $HOME. If I don't do this, tests randomly fail. There are probably some files there also. Only isolating the linuxcnc-src folder is not enough.

Still missing: A way to collect all the results to create a summary and some cleanup. It is just a quick&dirty POC for now.

@hdiethelm

@hdiethelm the file in $HOME that makes parallel tests fail at random is ~/.tool.mmap. io opens it with O_CREAT|O_TRUNC at startup (src/emc/tooldata/tooldata_mmap.cc) and refills it from its own tool table, and task, the interpreter and the GUIs of every running instance map that same file. So a second instance starting while the first runs replaces the first one's tool table. I reproduced it by starting a second sim a few seconds into tests/ptp-machine-frame: its G43.4 H1 lost the 50 mm tool length and the test failed, while it passes every time on its own. Giving each instance its own HOME fixes it, as you found. A per-instance file name would fix it in LinuxCNC itself; tool_mmap_set_fname() already exists for rs274.

@hdiethelm

There are more troubles. I have been hacking a version based on @hdiethelm that isolates home and the tests directory. That works and requires /tmp and /var/tmp to be separate. Needed to fix tests/overrun, which still isn't completely solved (calls the runtests script recursively). There are also random failures in the ui-smoke tests. Other that that it works fine.

There are more troubles. I have been hacking a version based on @hdiethelm that isolates home and the tests directory.

I have my linuxcnc-src in $HOME so it is also isolated, but you are right, it does not have to be that way.

That works and requires /tmp and /var/tmp to be separate.

Is there also something in /var/tmp used by linuxcnc? I had random failures in m98m99/12-M99-endless-main-program but isolating also /var/tmp, this is gone.

Needed to fix tests/overrun, which still isn't completely solved (calls the runtests script recursively). There are also random failures in the ui-smoke tests. Other that that it works fine.

Hmm, this should work, overrun starts runtest without any arguments, so it should run single threaded as before but isolated from the parent parallel runtests.

On my PC ui-smoke tests pass. Probably a os / source location dependent issue or a file not isolated.

Just checked: You can use --overlay-src /tmp --tmp-overlay /tmp followed by a --bind /tmp/results/ /tmp/results/ to bind a subfolder inside an overlayed folder. This could be used to store results but it needs one file per test.

Pushed a new commits that summarizes the result and also isolates /var/tmp. Now 4/4 runs pass all tests with 200/64/32 jobs.

And another commit: By binding all that is not overlayed RO, there can be no files that interfere. If you miss one folder with an overlay, there should be an error in runtests due to the file can not be written. For example if I don't add a $HOME overlay:
tool_mmap_creator(): file open fail: Read-only file system
cp: cannot create regular file '/home/user/linuxcnc_debug.txt': Read-only file system

Can be closed, #4648 was merged

Navigation Menu

Repository navigation

Running parallel instances using namespaces #4588

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Running parallel instances using namespaces #4588

Description

Activity

BsAtHome commented on Sep 27, 2026

hdiethelm commented on Sep 27, 2026

BsAtHome commented on Sep 27, 2026

hdiethelm commented on Sep 27, 2026

rmu75 commented on Sep 28, 2026

grandixximo commented on Sep 28, 2026

hdiethelm commented on Sep 28, 2026

hdiethelm commented on Sep 28, 2026

BsAtHome commented on Sep 28, 2026

hdiethelm commented on Sep 28, 2026

hdiethelm commented on Sep 28, 2026

grandixximo commented on Oct 5, 2026

BsAtHome commented on Oct 5, 2026

hdiethelm commented on Oct 5, 2026

hdiethelm commented on Oct 5, 2026

hdiethelm commented on Oct 27, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions