Crashes/invalid output for a few x86_64 instructions

[qarmin]

I'm building an internal tool that runs a large x86_64 SIMD/CPU instruction test suite natively, to catch hardware/firmware bugs rather than app bugs.

I ran it under box64 to find if the results differ between native run and this emulator. I already ran the same idea against rustc_codegen_gcc and found a few issues there too.

It is possible that #4214 (I updated report there) - may fix some of the issues

The full suite of my testing tool is ~19k cases, with AVX-512/GFNI, but box64 looks that not implement them(or at least doesn't work on my PC), so only ~3900 cases I was able to check(AVX2/AES/SHA/RDRAND/ADX)

Test project - box64_repro.zip (just setup + just matrix needed)

Below is AI guessing about certain bugs

Tested against commit b5697f4 (b5697f410770bf71d688c44b2cbb533053dba2d7), three ways: box64 built directly on an x86_64 host (no ARM_DYNAREC/RV64_DYNAREC - falls back to the plain C interpreter), box64 cross-built for aarch64 (-DARM64=1, real ARM_DYNAREC JIT) run under qemu-aarch64, and box64 cross-built for riscv64 (-DRV64=1, real RV64_DYNAREC JIT) run under qemu-riscv64.

RDSEED (0F C7 /7, no mandatory prefix) is implemented as RDPID

This is the concrete, reproducible one. Per the SDM, NP 0F C7 /7 is RDSEED r32/r64 and F3 0F C7 /7 is RDPID r32 - two different instructions that share a ModRM.reg slot and are disambiguated only by the F3 prefix. box64's interpreter handles reg=7 unconditionally as RDPID, with no prefix check, so plain RDSEED decodes into this branch instead of ever reaching a real RDSEED implementation:

box64/src/emu/x64run0f.c

Lines 1845 to 1872 in b5697f4

	case 0xC7:
	CHECK_FLAGS(emu);
	nextop = F8;
	GETE8xw(0);
	if (MODREG)
	switch ((nextop >> 3) & 7) {
	case 6: /* RDRAND Ed */
	RESET_FLAGS(emu);
	CLEAR_FLAG(F_OF);
	CLEAR_FLAG(F_SF);
	CLEAR_FLAG(F_PF);
	CLEAR_FLAG(F_ZF);
	CLEAR_FLAG(F_AF);
	SET_FLAG(F_CF);
	if (rex.w)
	ED->q[0] = get_random64();
	else {
	ED->dword[0] = get_random32();
	if (MODREG)
	ED->dword[1] = 0;
	}
	break;
	case 7: /* RDPID Ed */
	ED->q[0] = helper_getcpu(emu);
	break;
	default:
	return 0;
	}

case 6: /* RDRAND Ed */
    RESET_FLAGS(emu);
    ...
    SET_FLAG(F_CF);
    if (rex.w)
        ED->q[0] = get_random64();
    else {
        ED->dword[0] = get_random32();
        ...
    }
    break;
case 7: /* RDPID Ed */
    ED->q[0] = helper_getcpu(emu);
    break;

RDRAND (reg=6, right next to it) is implemented correctly and does set F_CF. The RDPID-labeled reg=7 branch never touches any flag. The exact same mistake is duplicated in the ARM64 dynarec (so it's not just an interpreter fallback thing):

box64/src/dynarec/arm64/dynarec_arm64_0f.c

Lines 2684 to 2711 in b5697f4

	case 0xC7:
	// rep has no impact here
	nextop = F8;
	if(MODREG) switch((nextop>>3)&7) {
	case 6:
	INST_NAME("RDRAND Ed");
	SETFLAGS(X_ALL, SF_SET_NODF);
	SET_DFNONE();
	GETED(0);
	IFX(X_OF|X_SF|X_ZF|X_PF|X_AF) {
	MOV32w(x1, (1<<F_OF)|(1<<F_SF)|(1<<F_ZF)|(1<<F_PF)|(1<<F_AF));
	BICw(xFlags, xFlags, x1);
	}
	if(cpuext.rndr) {
	MRS_rndr(x1);
	IFX(X_CF) { CSETw(x3, cNE); }
	} else {
	CALL(rex.w?const_random64:const_random32, x1);
	IFX(X_CF) { MOV32w(x3, 1); }
	}
	IFX(X_CF) { BFIw(xFlags, x3, F_CF, 1); }
	MOVxw_REG(ed, x1);
	break;
	case 7:
	INST_NAME("RDPID Ed");
	GETED(0);
	CALL_(const_helper_getcpu, ed, x2);
	break;

case 7:
    INST_NAME("RDPID Ed");
    GETED(0);
    CALL_(const_helper_getcpu, ed, x2);
    break;

The RISC-V64 dynarec has no case for 0xC7 register-form at all (falls to DEFAULT, i.e. the interpreter), so it inherits the same bug through the interpreter fallback:

box64/src/dynarec/rv64/dynarec_rv64_0f.c

Lines 2817 to 2826 in b5697f4

	case 0xC7:
	// rep has no impact here
	nextop = F8;
	if (MODREG) {
	switch ((nextop >> 3) & 7) {
	default:
	DEFAULT;
	}
	} else {
	switch ((nextop >> 3) & 7) {

What this actually does to guest code: rustc's codegen for _rdseed32_step/_rdseed64_step is

rdseed %ecx
mov    $0x1,%eax
cmovae %ecx,%eax   ; on real hardware: CF=0 (failure) -> return the (zeroed) value in ecx
ret

Since box64 never sets F_CF for this opcode, whatever CF happened to be left over from earlier code decides whether cmovae fires, and when it does, eax gets loaded not with 0 (what real hardware would leave in the destination on failure) but with whatever helper_getcpu() put there - the calling CPU's core id. That's exactly what I saw: the "step" functions don't fail with a fixed wrong value, they return small integers that change run to run (2, 7, 9, 10, 11, 13, 14, 17, 18, 21, 22 all observed across repeated runs on a 24-core host) - consistent with core ids, not with any RDSEED-shaped bug. Repeated runs on real hardware return 1 every single time; every backend of box64 is flaky on this.

Fix direction: add a real F3-prefix check before dispatching to RDPID, and add an actual RDSEED implementation (mirroring the RDRAND branch, using a real RNG source and explicitly setting F_CF) for the no-prefix case. RDPID should then live behind its own prefix-gated path instead of unconditionally owning reg=7.

Six opcodes with no implementation at all (clean, deterministic SIGILL on every box64 backend)

All six pass natively and reliably SIGILL identically on the x86_64-host interpreter, the aarch64 dynarec and the riscv64 dynarec - no flakiness, no backend differences:

• RDRAND/RDSEED, 16-bit operand form (66 0F C7 /6, 66 0F C7 /7) - src/emu/x64run660f.c has no case 0xC7 at all, so the 16-bit forms never reach any handler (the 32-/64-bit forms at src/emu/x64run0f.c do, see above).
• XSAVEOPT/XSAVEOPT64 (0F AE /6) - the 0F AE group implements reg 0/1/2/3/4/5/7 (FXSAVE/FXRSTOR/LDMXCSR/STMXCSR/XSAVE/XRSTOR/CLFLUSH) but reg=6 falls straight to default: return 0:
  

  box64/src/emu/x64run0f.c

  Lines 1396 to 1444 in b5697f4

  	case 0: /* FXSAVE Ed */
  	_GETED(0);
  	#ifdef TEST_INTERPRETER
  	emu->sw.f.F87_TOP = emu->top&7;
  	#else
  	if(rex.is32bits)
  	fpu_fxsave32(emu, ED);
  	else
  	fpu_fxsave64(emu, ED);
  	#endif
  	break;
  	case 1: /* FXRSTOR Ed */
  	_GETED(0);
  	if(rex.is32bits)
  	fpu_fxrstor32(emu, ED);
  	else
  	fpu_fxrstor64(emu, ED);
  	break;
  	case 2: /* LDMXCSR Md */
  	GETED(0);
  	emu->mxcsr.x32 = ED->dword[0];
  	#ifndef TEST_INTERPRETER
  	if(BOX64ENV(sse_flushto0))
  	applyFlushTo0(emu);
  	#endif
  	break;
  	case 3: /* STMXCSR Md */
  	GETED(0);
  	ED->dword[0] = emu->mxcsr.x32;
  	break;
  	case 4: /* XSAVE Ed */
  	_GETED(0);
  	#ifdef TEST_INTERPRETER
  	emu->sw.f.F87_TOP = emu->top&7;
  	#else
  	fpu_xsave(emu, ED, rex.is32bits);
  	#endif
  	break;
  	case 5: /* XRSTOR Ed */
  	_GETED(0);
  	fpu_xrstor(emu, ED, rex.is32bits);
  	break;
  	case 7: /* CLFLUSH Ed */
  	_GETED(0);
  	__sync_synchronize();
  	break;
  	default:
  	return 0;
  	}

• XSAVEC/XSAVEC64 (0F C7 /4, memory operand) - the memory-operand half of the same 0xC7 case implements only case 1 (CMPXCHG8B/CMPXCHG16B); reg=4 falls to the same default: return 0:
  

  box64/src/emu/x64run0f.c

  Lines 1873 to 1908 in b5697f4

  	else {
  	switch ((nextop >> 3) & 7) {
  	case 1: /* CMPXCHG8B Eq / CMPXCHG16B Eq */
  	if (rex.w) {
  	#ifndef TEST_INTERPRETER
  	if (((uintptr_t)ED) & 0xf) {
  	EmitSignal(emu, X64_SIGSEGV, (void*)R_RIP, 0xbad0); // GPF
  	}
  	#endif
  	tmp64u = ED->q[0];
  	tmp64u2 = ED->q[1];
  	if (R_RAX == tmp64u && R_RDX == tmp64u2) {
  	SET_FLAG(F_ZF);
  	ED->q[0] = R_RBX;
  	ED->q[1] = R_RCX;
  	} else {
  	CLEAR_FLAG(F_ZF);
  	R_RAX = tmp64u;
  	R_RDX = tmp64u2;
  	}
  	} else {
  	tmp32u = ED->dword[0];
  	tmp32u2 = ED->dword[1];
  	if (R_EAX == tmp32u && R_EDX == tmp32u2) {
  	SET_FLAG(F_ZF);
  	ED->dword[0] = R_EBX;
  	ED->dword[1] = R_ECX;
  	} else {
  	CLEAR_FLAG(F_ZF);
  	R_RAX = tmp32u;
  	R_RDX = tmp32u2;
  	}
  	}
  	break;
  	default:
  	return 0;

Minor/weaker note, probably not worth acting on: XGETBV (0F 01 D0) also SIGILLs in my run, but that's because our test calls it with ECX=1, and box64 explicitly faults on any nonzero ECX:

box64/src/emu/x64run0f.c

Lines 122 to 131 in b5697f4

	case 0xD0:
	if(R_RCX) {
	#ifndef TEST_INTERPRETER
	EmitSignal(emu, X64_SIGILL, (void*)R_RIP, 0);
	#endif
	} else {
	R_RAX = 0b111; // x87 & SSE & AVX for now
	R_RDX = 0;
	}
	break;

Only XCR0 (ECX=0) is architecturally defined, so this is arguably box64 being more correct than the AMD chip I tested on, which tolerates ECX=1 and returns a value instead of #GPing. Including it for completeness, not as a real bug.

Open, not independently reproducible: SHA-NI SIGSEGVs on the RISC-V64 dynarec, but only inside a larger binary

Six of the seven SHA-NI intrinsics (_mm_sha1msg1_epu32, _mm_sha1msg2_epu32, _mm_sha1nexte_epu32, _mm_sha256msg1_epu32, _mm_sha256msg2_epu32, _mm_sha256rnds2_epu32 - _mm_sha1rnds4_epu32 is the one that's fine) reliably SIGSEGV (6/6 tries, real core dump, not SIGILL) under the riscv64 dynarec when called from inside my full test-suite binary, at the very first SHA case tried. I could not reproduce it in isolation: the exact same call, same inputs, in a minimal binary that does nothing else first, passes every time on every backend including riscv64. So this looks state-dependent (JIT translation-cache contents, register allocation, something built up by whatever ran before it) rather than an opcode/operand problem, and I don't have a clean enough repro to file with confidence yet - flagging it here in case it rings a bell, rather than as a confirmed bug.

Reproducing this

"just matrix"

One real run (rdseed32_step/rdseed64_step will show different numbers, or even pass, every time you re-run it - that's the point, see above; the SHA-NI rows below all show pass here for the reason explained above):

case              native          box64-x86       box64-aarch64   box64-riscv64
rdrand16_step     pass            SIGILL          SIGILL          SIGILL
rdseed16_step     pass            SIGILL          SIGILL          SIGILL
xgetbv            pass            SIGILL          SIGILL          SIGILL
xsaveopt          pass            SIGILL          SIGILL          SIGILL
xsavec            pass            SIGILL          SIGILL          SIGILL
xsaveopt64        pass            SIGILL          SIGILL          SIGILL
xsavec64          pass            SIGILL          SIGILL          SIGILL
rdseed32_step     pass            pass            pass            fail(2)
rdseed64_step     pass            fail(18)        fail(2)         fail(2)
sha1msg1_epu32    pass            pass            pass            pass
sha1msg2_epu32    pass            pass            pass            pass
sha1nexte_epu32   pass            pass            pass            pass
sha1rnds4_epu32   pass            pass            pass            pass
sha256msg1_epu32  pass            pass            pass            pass
sha256msg2_epu32  pass            pass            pass            pass
sha256rnds2_epu32 pass            pass            pass            pass

[ptitSeb]

Thanks for this, I will take a look.

But first comment: both XSAVEOPT & XSAVEC are not supported in box64. Thoses opcode are not implemented (also explains why XGETBV with ECX=1 fails), and the related bit in CPUID are not set. That was a design choice. Are those feature really required in the wild? because I haven't seen any program requiring those, I have always saw those has optionnal.