gl-fwd.so segfaults on ppc64le when a GTK3 application preloads it

gl-fwd.so segfaults on ppc64le when a GTK3 application preloads it, and I could not reduce it below the full GTK3 environment

Spotted while adding ppc64le/loongarch64/riscv64 to the Anylinux-AppImages demo matrix (pkgforge-dev/Anylinux-AppImages#969). The gtk3 demo builds fine on all four new arches and runs on three of them; on ppc64le only it dies.

The observation

The demo AppImage on ppc64le exits ~1s after extraction with status 0 and no output, which quick-sharun --test reports as a failure because the process is not alive after 12s. The exit status is lost through xvfb-run and the uruntime wrapper. Bisecting the preload directory by preloading one library at a time against the AppImage's own shared/bin/gtk3-demo with the AppDir's lib/:

`LD_PRELOAD`	result
(none)	alive (124 = the 12s timeout)
`anylinux.so`	alive
`cross-libc-dlopen.so`	alive
`egl-fwd.so`	alive
`gles-fwd.so`	alive
`gl-fwd.so`	segfault (139)

It is not the application and not the display: the distro gtk3-demo package stays alive for the full window in the same container, and the AppImage fails identically with a working Xvfb DISPLAY=:99. gl-fwd.so is only deployed when USE_HOST_DRIVERS_EXPERIMENTAL=1, which the gtk3 demo sets.

Why it is only ppc64le

ppc64 (BE) passes because quick-sharun force-disables USE_HOST_DRIVERS_EXPERIMENTAL there (no cross-libc-dlopen support), so gl-fwd.so is never deployed.
riscv64 and loongarch64 deploy it too, and pass. So this is not "the shim is broken on the new arches", it is ppc64le-specific.

What I could not reproduce, and how hard I tried

I built the ppc64le trampolines and ran them under qemu-ppc64le against a real ppc64le glibc sysroot (Debian 2.41 and 2.44), using both the shipped v0.2.7 gl-fwd.so and a fresh zig build. Seven call shapes, all all passed:

the repo's own t_args/t_fret/t_absent vehicle from make gl-fwd-qemu-check
the full generated 3470-symbol table, linked with -lGL so DT_NEEDED binds the shim
dlopen + dlsym
unresolved slots reaching glfwd_resolve_asm → glfwd_resolve_one → glfwd_absent
13 live float arguments through the stfd/lfd save/restore
repeated calls through a patched slot
a resolved target living in a separate module

Theories I chased and eliminated:

std 3,0(1) / mr 3,0 / mflr 0 ordering in glfwd_resolve_asm is correct — r0 is consumed into r3 before mflr clobbers it.
Frame offsets: the 192-byte frame has no overlapping slots (stfd 13→160..168, std 11→168..176, LR→184..192, 176..184 unused).
ELFv2 global-entry prologue is not needed: both glfwd_resolve_asm and glfwd_absent are LOCAL HIDDEN, so the same-module TOC in r2 is valid.
__tls_get_addr_opt (a PowerPC-only symbol the ppc64le shim imports) is exported by both the bundled and the host libc, and preloading onto a trivial binary succeeds.
ld64.so.2 in DT_NEEDED is normal — x86_64 lists ld-linux-x86-64.so.2 and riscv64 lists ld-linux-riscv64-lp64d.so.1 the same way.

So the crash needs something the full GTK3 AppDir environment provides and my harness does not. My best guess is a GDK/GL path taken only by real GTK, but that is a guess and I would rather hand over the reproducer than a theory.

Reproducer

On a ppc64le container under qemu-user (this is what the AppImage CI does):

# from a checkout of Anylinux-AppImages at the PR above
sh ./useful-tools/demo/gtk3-demo-appimage.sh        # builds dist/gtk3-demo-ppc64le.AppImage
APP=$(ls dist/*.AppImage)
APPIMAGE_EXTRACT_AND_RUN=1 APPIMAGE_TARGET_DIR=$PWD/_d "$APP" >/dev/null 2>&1 || :
cd _d
xvfb-run -a -- env LD_LIBRARY_PATH=$PWD/lib:$PWD/lib/sharun-preload \
  LD_PRELOAD=$PWD/lib/sharun-preload/gl-fwd.so ./shared/bin/gtk3-demo; echo "exit=$?"
# exit=139, and no output at all

Preloading any of the other four libraries the same way gives 124 (still running at the timeout).

Also worth flagging: CI never runs the check that would have caught this

make gl-fwd-qemu-check exists, is documented in docs/building.md, and docs/report/10-measured-versus-assumed.md §10 cites it as the evidence for the cross-arch trampolines. But no workflow runs it: gates.yml builds ppc64le and runs scripts/verify-artifacts.sh, which is a build gate, not a run gate.

$ grep -rn "gl-fwd-qemu-check\|gl-fwd-asm-check" .github/ scripts/
(nothing)

That is how a ppc64le-only runtime defect can ship. Wiring it into the build matrix is a separate, small change, and I am happy to send it — but it would not by itself explain this report, since the check passes for the shipped binary here.

Environment

cross-libc-dlopen v0.2.7, shipped gl-fwd.so for ppc64le, 852,960 bytes, max symbol GLIBC_2.22, DT_NEEDED: libdl.so.2, libc.so.6, ld64.so.2
ArchPOWER ppc64le packages, mesa 1:26.2.3-1, gtk3 1:3.24.52-2, gtk3-demos 1:3.24.52-2
qemu-ppc64le 7.2.0 under a ghcr.io/pkgforge-dev/archlinux ppc64le container
Host: x86_64, kernel 7.2.11, binfmt_misc registered by anylinux-setup-action's register-binfmt.sh

Requested by Samueru via errand.
Conversation: https://discord.com/channels/1313385177703256064/1561764843965649021

Found a ppc64le bug in the resolver that is a candidate cause. I have not confirmed it is what crashes gtk3-demo, because I could not run the GTK3 AppImage.

Bug. glfwd_resolve_asm in src/gl-fwd.c (ELFv2) saves the argument registers at 0..56(sp). That is the frame header a C callee owns: back chain at 0, CR at 8, LR at 16, TOC at 24. glfwd_resolve_one is not a leaf, and its generated code does std 0,16(1) before stdu, so it overwrites the saved r5. The first call through any slot gets a corrupted third integer argument. Later calls go straight to the patched slot, so they are fine.

Why the existing check missed it. t_args(int, long, double, const char *) only uses r3, r4, f1 and r5 on the first call, and the pointer it checks is read back as garbage only if the clobbered value is dereferenced, which the test does not catch. (I did not work out why it still passed.)

Reproduction. ppc64le under qemu-ppc64le 7.2 with the Debian ppc64el sysroot. The target takes 8 long arguments (1..8) and returns 0 if all match. First call through the shim on current main: nonzero (third argument wrong). Second call: 0.

Fix. Save registers above a 112-byte header (48 bytes ELFv1 header plus a 64-byte parameter area, which also covers ELFv2). The frame goes from 192 to 304 bytes. I also added a first-call 8-argument case to make gl-fwd-qemu-check. On the old gl-fwd.c that target prints FAIL ... ppc64le, on the new one ok. aarch64 also passes. riscv64 and loongarch64 were not run (no cross-gcc here) and their code is untouched.

The fix is committed on a local branch and not pushed. Please rerun the gtk3 demo with a gl-fwd.so built from it. If it still segfaults after that, there is a second cause.

I have not touched the CI point (nothing runs gl-fwd-qemu-check).

Confirmed: your resolver bug is the cause, and the fix makes the gtk3-demo ppc64le AppImage start.

Your bug reproduces exactly as you described. ppc64le target taking 8 longs, first call through the shim on current main (3095c15):

first=3 second=0
FIRST CALL WRONG

The third argument is corrupted on the first call and correct on the second, which is the patched-slot signature you predicted. The mechanism is in the generated code for glfwd_resolve_one — after stdu 1, -128(1) it does std 0, 144(1), which is offset 16 of the caller's frame, the slot glfwd_resolve_asm saved r5 into:

glfwd_resolve_one:
	mflr 0
	std 31, -8(1)
	stdu 1, -128(1)
	std 0, 144(1)        # <- 16(r1) of the caller = saved r5

Your fix works. Moving the saved registers above a 112-byte header (frame 192 -> 304) gives:

first=0 second=0
all passed

And it fixes the gtk3-demo segfault. I built the gtk3-demo AppImage for ppc64le under qemu-ppc64le in the Anylinux-AppImages CI, then swapped only lib/sharun-preload/gl-fwd.so for a build of the patched gl-fwd.c:

`gl-fwd.so`	preloaded into the AppImage's `shared/bin/gtk3-demo`
shipped v0.2.7	EXITED 139 (SIGSEGV)
built from your fix	ALIVE at the 10s timeout

And the real path, ./AppRun with the fixed library in place:

AppRun -> ALIVE (ok)

So the answer to "if it still segfaults after that, there is a second cause" is: it does not, there is no second cause. One bug, and your fix closes it.

For the record, this also explains why I could not reproduce it earlier with the t_args-style vehicle: t_args(int, long, double, const char *) passes r3, r4, f1 and r5, and r5 is the clobbered slot, but the corrupted value is the const char * and the existing case only dereferences d[0]/d[1] — a garbage pointer there is a crash or not depending on what the clobbered value happens to be, not a clean assertion failure. Your 8-long case is the one that pins it deterministically. That is worth keeping.

Two notes on the AppImages side, no action needed here:

The workaround I had considered — unsetting USE_HOST_DRIVERS_EXPERIMENTAL for that demo so gl-fwd.so is never deployed — would have hidden this. I did not apply it.
Once a release with this fix ships, the ppc64le gtk3-demo job in update sharun to 3.7.1, deploy its dash and notice, and build demos on riscv64, loongarch64, ppc64le and ppc64 Anylinux-AppImages#969 should go green on its own. It is the only failing job in that matrix.

The CI point still stands: nothing runs gl-fwd-qemu-check, which is why this shipped. Your new first-call 8-argument case is exactly what that check needs to run.

I re-verified your patch against current main (3095c15) from scratch, and it holds. Since the branch you have is local, here is the exact change as a diff in case that is useful — one hunk, src/gl-fwd.c only, nothing else touched.

Reproduced on current main:

=== CURRENT main (buggy):
first=3 second=0
FIRST CALL WRONG
exit=1
=== WITH the patch:
first=0 second=0
all passed
exit=0

Built with zig cc -target powerpc64le-linux-gnu, run under qemu-ppc64le 7.2 against a Debian ppc64el glibc 2.41 sysroot. The target is your 8-long vehicle (1..8, returns 0 only if all match).

And the gtk3-demo AppImage, ppc64le, built in the Anylinux-AppImages CI with only lib/sharun-preload/gl-fwd.so swapped:

`gl-fwd.so`	preloaded into `shared/bin/gtk3-demo`
shipped v0.2.7	EXITED 139 (SIGSEGV)
built from your patch	ALIVE at the 10s timeout

./AppRun with the patched library also stays alive. So there is no second cause.

The patch, as verified

--- a/src/gl-fwd.c
+++ b/src/gl-fwd.c
@@ -554,60 +554,60 @@
         ".hidden glfwd_resolve_asm\n"
         ".type   glfwd_resolve_asm,@function\n"
         "glfwd_resolve_asm:\n"
-        "	stdu 1, -192(1)\n"
-        "	std  3, 0(1)\n"
+        "	stdu 1, -304(1)\n"
+        "	std  3, 112(1)\n"
         "	mr   3, 0\n"
         "	mflr 0\n"
-        "	std  0, 184(1)\n"
-        "	std  4, 8(1)\n"
-        "	std  5, 16(1)\n"
-        "	std  6, 24(1)\n"
-        "	std  7, 32(1)\n"
-        "	std  8, 40(1)\n"
-        "	std  9, 48(1)\n"
-        "	std  10, 56(1)\n"
-        "	stfd 1, 64(1)\n"
-        "	stfd 2, 72(1)\n"
-        "	stfd 3, 80(1)\n"
-        "	stfd 4, 88(1)\n"
-        "	stfd 5, 96(1)\n"
-        "	stfd 6, 104(1)\n"
-        "	stfd 7, 112(1)\n"
-        "	stfd 8, 120(1)\n"
-        "	stfd 9, 128(1)\n"
-        "	stfd 10, 136(1)\n"
-        "	stfd 11, 144(1)\n"
-        "	stfd 12, 152(1)\n"
-        "	stfd 13, 160(1)\n"
-        "	std  11, 168(1)\n"
+        "	std  0, 296(1)\n"
+        "	std  4, 120(1)\n"
+        "	std  5, 128(1)\n"
+        "	std  6, 136(1)\n"
+        "	std  7, 144(1)\n"
+        "	std  8, 152(1)\n"
+        "	std  9, 160(1)\n"
+        "	std  10, 168(1)\n"
+        "	stfd 1, 176(1)\n"
+        "	stfd 2, 184(1)\n"
+        "	stfd 3, 192(1)\n"
+        "	stfd 4, 200(1)\n"
+        "	stfd 5, 208(1)\n"
+        "	stfd 6, 216(1)\n"
+        "	stfd 7, 224(1)\n"
+        "	stfd 8, 232(1)\n"
+        "	stfd 9, 240(1)\n"
+        "	stfd 10, 248(1)\n"
+        "	stfd 11, 256(1)\n"
+        "	stfd 12, 264(1)\n"
+        "	stfd 13, 272(1)\n"
+        "	std  11, 280(1)\n"
         "	bl  glfwd_resolve_one\n"
         "	mr   12, 3\n"
         "	mtctr 3\n"
-        "	ld   11, 168(1)\n"
-        "	lfd  1, 64(1)\n"
-        "	lfd  2, 72(1)\n"
-        "	lfd  3, 80(1)\n"
-        "	lfd  4, 88(1)\n"
-        "	lfd  5, 96(1)\n"
-        "	lfd  6, 104(1)\n"
-        "	lfd  7, 112(1)\n"
-        "	lfd  8, 120(1)\n"
-        "	lfd  9, 128(1)\n"
-        "	lfd  10, 136(1)\n"
-        "	lfd  11, 144(1)\n"
-        "	lfd  12, 152(1)\n"
-        "	lfd  13, 160(1)\n"
-        "	ld   3, 0(1)\n"
-        "	ld   4, 8(1)\n"
-        "	ld   5, 16(1)\n"
-        "	ld   6, 24(1)\n"
-        "	ld   7, 32(1)\n"
-        "	ld   8, 40(1)\n"
-        "	ld   9, 48(1)\n"
-        "	ld   10, 56(1)\n"
-        "	ld   0, 184(1)\n"
+        "	ld   11, 280(1)\n"
+        "	lfd  1, 176(1)\n"
+        "	lfd  2, 184(1)\n"
+        "	lfd  3, 192(1)\n"
+        "	lfd  4, 200(1)\n"
+        "	lfd  5, 208(1)\n"
+        "	lfd  6, 216(1)\n"
+        "	lfd  7, 224(1)\n"
+        "	lfd  8, 232(1)\n"
+        "	lfd  9, 240(1)\n"
+        "	lfd  10, 248(1)\n"
+        "	lfd  11, 256(1)\n"
+        "	lfd  12, 264(1)\n"
+        "	lfd  13, 272(1)\n"
+        "	ld   3, 112(1)\n"
+        "	ld   4, 120(1)\n"
+        "	ld   5, 128(1)\n"
+        "	ld   6, 136(1)\n"
+        "	ld   7, 144(1)\n"
+        "	ld   8, 152(1)\n"
+        "	ld   9, 160(1)\n"
+        "	ld   10, 168(1)\n"
+        "	ld   0, 296(1)\n"
         "	mtlr 0\n"
-        "	addi 1, 1, 192\n"
+        "	addi 1, 1, 304\n"
         "	bctr\n"
         ".size glfwd_resolve_asm, .-glfwd_resolve_asm\n");
 /* The trampoline gets the table through r2 (its own TOC, valid at the local

That is the whole change: the frame moves from 192 to 304 bytes and every saved slot moves above a 112-byte header (48-byte ELFv1 header plus a 64-byte parameter area, which covers ELFv2 as well). LR moves from 184 to 296, so it still sits in the last 8 bytes of the frame.

I am not opening a PR for this because the fix is yours and you said it is on a local branch — it is your call how it lands. If you would rather I open one from a fork with your change plus the make gl-fwd-qemu-check case, say so and I will.

Two loose ends this leaves, both outside the code change:

Nothing runs gl-fwd-qemu-check. gates.yml builds ppc64le and runs scripts/verify-artifacts.sh, which never executes a trampoline. Your new first-call 8-argument case is exactly what that target needs to carry, and wiring the target into the build matrix is what stops this class of bug shipping again.
v0.2.7 is still the latest release, so every AppImage that preloads gl-fwd.so on ppc64le with host drivers still crashes. update sharun to 3.7.1, deploy its dash and notice, and build demos on riscv64, loongarch64, ppc64le and ppc64 Anylinux-AppImages#969 is green everywhere except that one job and will stay red until a release carries this.

update cross-libc-dlopen to v0.2.8

Correct two claims the review passes did not measure

Navigation Menu

Repository navigation

gl-fwd.so segfaults on ppc64le when a GTK3 application preloads it #59

The observation

Why it is only ppc64le

What I could not reproduce, and how hard I tried

Reproducer

Also worth flagging: CI never runs the check that would have caught this

Environment

The patch, as verified

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

gl-fwd.so segfaults on ppc64le when a GTK3 application preloads it #59

Description

The observation

Why it is only ppc64le

What I could not reproduce, and how hard I tried

Reproducer

Also worth flagging: CI never runs the check that would have caught this

Environment

Activity

talaria0101 commented on Oct 22, 2026

Nemo-010 commented on Oct 22, 2026

Nemo-010 commented on Oct 22, 2026

The patch, as verified

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions