diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/guard-logs.tar.gz b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/guard-logs.tar.gz deleted file mode 100644 index 6f1c976b3..000000000 Binary files a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/guard-logs.tar.gz and /dev/null differ diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/guard/engine_Meta-Llama-3.1-8B-Instruct-4bit_r1_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/guard/engine_Meta-Llama-3.1-8B-Instruct-4bit_r1_base.log new file mode 100644 index 000000000..e1f62dcda --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/guard/engine_Meta-Llama-3.1-8B-Instruct-4bit_r1_base.log @@ -0,0 +1,28 @@ +2026-10-09T02:44:37+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:44:37+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:44:52+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:44:52+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-engine -m models/mlx/Meta-Llama-3.1-8B-Instruct-4bit --path both --decode-storage dense --prompt-tokens 256,2048 -n 128 --warmup-tokens 20 --csv /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/measure/engine/rounds.csv --label base-r1 +2026-10-09T02:44:52+0900 sample 1: clean gpu=[] +2026-10-09T02:44:54+0900 sample 2: clean gpu=[1130754:mlxcel-bench-en] +2026-10-09T02:44:55+0900 sample 3: clean gpu=[1130754:mlxcel-bench-en] +2026-10-09T02:44:57+0900 sample 4: clean gpu=[1130754:mlxcel-bench-en] +2026-10-09T02:44:58+0900 sample 5: clean gpu=[1130754:mlxcel-bench-en] +2026-10-09T02:45:00+0900 sample 6: clean gpu=[1130754:mlxcel-bench-en] +2026-10-09T02:45:01+0900 sample 7: clean gpu=[1130754:mlxcel-bench-en] +2026-10-09T02:45:03+0900 sample 8: clean gpu=[1130754:mlxcel-bench-en] +2026-10-09T02:45:04+0900 sample 9: clean gpu=[1130754:mlxcel-bench-en] +2026-10-09T02:45:06+0900 sample 10: clean gpu=[1130754:mlxcel-bench-en] +2026-10-09T02:45:07+0900 sample 11: clean gpu=[1130754:mlxcel-bench-en] +2026-10-09T02:45:09+0900 sample 12: clean gpu=[1130754:mlxcel-bench-en] +2026-10-09T02:45:10+0900 sample 13: clean gpu=[1130754:mlxcel-bench-en] +2026-10-09T02:45:12+0900 sample 14: clean gpu=[1130754:mlxcel-bench-en] +2026-10-09T02:45:13+0900 sample 15: clean gpu=[1130754:mlxcel-bench-en] +2026-10-09T02:45:15+0900 sample 16: clean gpu=[1130754:mlxcel-bench-en] +2026-10-09T02:45:16+0900 sample 17: clean gpu=[1130754:mlxcel-bench-en] +2026-10-09T02:45:18+0900 sample 18: clean gpu=[1130754:mlxcel-bench-en] +2026-10-09T02:45:19+0900 sample 19: clean gpu=[1130754:mlxcel-bench-en] +2026-10-09T02:45:21+0900 sample 20: clean gpu=[1130754:mlxcel-bench-en] +2026-10-09T02:45:22+0900 sample 21: clean gpu=[1130754:mlxcel-bench-en] +2026-10-09T02:45:24+0900 sample 22: clean gpu=[1130754:mlxcel-bench-en] +2026-10-09T02:45:25+0900 monitor: 22 samples +2026-10-09T02:45:25+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/guard/engine_Meta-Llama-3.1-8B-Instruct-4bit_r1_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/guard/engine_Meta-Llama-3.1-8B-Instruct-4bit_r1_new.log new file mode 100644 index 000000000..9924859b7 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/guard/engine_Meta-Llama-3.1-8B-Instruct-4bit_r1_new.log @@ -0,0 +1,25 @@ +2026-10-09T02:45:30+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:45:30+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:45:45+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:45:45+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-engine -m models/mlx/Meta-Llama-3.1-8B-Instruct-4bit --path both --decode-storage dense --prompt-tokens 256,2048 -n 128 --warmup-tokens 20 --csv /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/measure/engine/rounds.csv --label new-r1 +2026-10-09T02:45:45+0900 sample 1: clean gpu=[] +2026-10-09T02:45:47+0900 sample 2: clean gpu=[1161150:mlxcel-bench-en] +2026-10-09T02:45:48+0900 sample 3: clean gpu=[1161150:mlxcel-bench-en] +2026-10-09T02:45:50+0900 sample 4: clean gpu=[1161150:mlxcel-bench-en] +2026-10-09T02:45:51+0900 sample 5: clean gpu=[1161150:mlxcel-bench-en] +2026-10-09T02:45:53+0900 sample 6: clean gpu=[1161150:mlxcel-bench-en] +2026-10-09T02:45:54+0900 sample 7: clean gpu=[1161150:mlxcel-bench-en] +2026-10-09T02:45:55+0900 sample 8: clean gpu=[1161150:mlxcel-bench-en] +2026-10-09T02:45:57+0900 sample 9: clean gpu=[1161150:mlxcel-bench-en] +2026-10-09T02:45:58+0900 sample 10: clean gpu=[1161150:mlxcel-bench-en] +2026-10-09T02:46:00+0900 sample 11: clean gpu=[1161150:mlxcel-bench-en] +2026-10-09T02:46:01+0900 sample 12: clean gpu=[1161150:mlxcel-bench-en] +2026-10-09T02:46:03+0900 sample 13: clean gpu=[1161150:mlxcel-bench-en] +2026-10-09T02:46:04+0900 sample 14: clean gpu=[1161150:mlxcel-bench-en] +2026-10-09T02:46:06+0900 sample 15: clean gpu=[1161150:mlxcel-bench-en] +2026-10-09T02:46:07+0900 sample 16: clean gpu=[1161150:mlxcel-bench-en] +2026-10-09T02:46:09+0900 sample 17: clean gpu=[1161150:mlxcel-bench-en] +2026-10-09T02:46:10+0900 sample 18: clean gpu=[1161150:mlxcel-bench-en] +2026-10-09T02:46:12+0900 sample 19: clean gpu=[1161150:mlxcel-bench-en] +2026-10-09T02:46:13+0900 monitor: 19 samples +2026-10-09T02:46:13+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/guard/engine_Meta-Llama-3.1-8B-Instruct-4bit_r2_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/guard/engine_Meta-Llama-3.1-8B-Instruct-4bit_r2_base.log new file mode 100644 index 000000000..19bd89c8d --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/guard/engine_Meta-Llama-3.1-8B-Instruct-4bit_r2_base.log @@ -0,0 +1,26 @@ +2026-10-09T02:47:12+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:47:12+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:47:26+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:47:26+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-engine -m models/mlx/Meta-Llama-3.1-8B-Instruct-4bit --path both --decode-storage dense --prompt-tokens 256,2048 -n 128 --warmup-tokens 20 --csv /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/measure/engine/rounds.csv --label base-r2 +2026-10-09T02:47:27+0900 sample 1: clean gpu=[] +2026-10-09T02:47:28+0900 sample 2: clean gpu=[1217341:mlxcel-bench-en] +2026-10-09T02:47:30+0900 sample 3: clean gpu=[1217341:mlxcel-bench-en] +2026-10-09T02:47:31+0900 sample 4: clean gpu=[1217341:mlxcel-bench-en] +2026-10-09T02:47:33+0900 sample 5: clean gpu=[1217341:mlxcel-bench-en] +2026-10-09T02:47:34+0900 sample 6: clean gpu=[1217341:mlxcel-bench-en] +2026-10-09T02:47:36+0900 sample 7: clean gpu=[1217341:mlxcel-bench-en] +2026-10-09T02:47:37+0900 sample 8: clean gpu=[1217341:mlxcel-bench-en] +2026-10-09T02:47:39+0900 sample 9: clean gpu=[1217341:mlxcel-bench-en] +2026-10-09T02:47:40+0900 sample 10: clean gpu=[1217341:mlxcel-bench-en] +2026-10-09T02:47:42+0900 sample 11: clean gpu=[1217341:mlxcel-bench-en] +2026-10-09T02:47:43+0900 sample 12: clean gpu=[1217341:mlxcel-bench-en] +2026-10-09T02:47:45+0900 sample 13: clean gpu=[1217341:mlxcel-bench-en] +2026-10-09T02:47:46+0900 sample 14: clean gpu=[1217341:mlxcel-bench-en] +2026-10-09T02:47:48+0900 sample 15: clean gpu=[1217341:mlxcel-bench-en] +2026-10-09T02:47:49+0900 sample 16: clean gpu=[1217341:mlxcel-bench-en] +2026-10-09T02:47:51+0900 sample 17: clean gpu=[1217341:mlxcel-bench-en] +2026-10-09T02:47:52+0900 sample 18: clean gpu=[1217341:mlxcel-bench-en] +2026-10-09T02:47:54+0900 sample 19: clean gpu=[1217341:mlxcel-bench-en] +2026-10-09T02:47:55+0900 sample 20: clean gpu=[1217341:mlxcel-bench-en] +2026-10-09T02:47:56+0900 monitor: 20 samples +2026-10-09T02:47:56+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/guard/engine_Meta-Llama-3.1-8B-Instruct-4bit_r2_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/guard/engine_Meta-Llama-3.1-8B-Instruct-4bit_r2_new.log new file mode 100644 index 000000000..8f463584e --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/guard/engine_Meta-Llama-3.1-8B-Instruct-4bit_r2_new.log @@ -0,0 +1,25 @@ +2026-10-09T02:48:01+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:48:01+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:48:16+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:48:16+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-engine -m models/mlx/Meta-Llama-3.1-8B-Instruct-4bit --path both --decode-storage dense --prompt-tokens 256,2048 -n 128 --warmup-tokens 20 --csv /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/measure/engine/rounds.csv --label new-r2 +2026-10-09T02:48:16+0900 sample 1: clean gpu=[] +2026-10-09T02:48:18+0900 sample 2: clean gpu=[1245919:mlxcel-bench-en] +2026-10-09T02:48:19+0900 sample 3: clean gpu=[1245919:mlxcel-bench-en] +2026-10-09T02:48:21+0900 sample 4: clean gpu=[1245919:mlxcel-bench-en] +2026-10-09T02:48:22+0900 sample 5: clean gpu=[1245919:mlxcel-bench-en] +2026-10-09T02:48:24+0900 sample 6: clean gpu=[1245919:mlxcel-bench-en] +2026-10-09T02:48:25+0900 sample 7: clean gpu=[1245919:mlxcel-bench-en] +2026-10-09T02:48:27+0900 sample 8: clean gpu=[1245919:mlxcel-bench-en] +2026-10-09T02:48:28+0900 sample 9: clean gpu=[1245919:mlxcel-bench-en] +2026-10-09T02:48:30+0900 sample 10: clean gpu=[1245919:mlxcel-bench-en] +2026-10-09T02:48:31+0900 sample 11: clean gpu=[1245919:mlxcel-bench-en] +2026-10-09T02:48:33+0900 sample 12: clean gpu=[1245919:mlxcel-bench-en] +2026-10-09T02:48:34+0900 sample 13: clean gpu=[1245919:mlxcel-bench-en] +2026-10-09T02:48:36+0900 sample 14: clean gpu=[1245919:mlxcel-bench-en] +2026-10-09T02:48:37+0900 sample 15: clean gpu=[1245919:mlxcel-bench-en] +2026-10-09T02:48:39+0900 sample 16: clean gpu=[1245919:mlxcel-bench-en] +2026-10-09T02:48:40+0900 sample 17: clean gpu=[1245919:mlxcel-bench-en] +2026-10-09T02:48:42+0900 sample 18: clean gpu=[1245919:mlxcel-bench-en] +2026-10-09T02:48:43+0900 sample 19: clean gpu=[1245919:mlxcel-bench-en] +2026-10-09T02:48:44+0900 monitor: 19 samples +2026-10-09T02:48:44+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/guard/engine_Meta-Llama-3.1-8B-Instruct-4bit_r3_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/guard/engine_Meta-Llama-3.1-8B-Instruct-4bit_r3_base.log new file mode 100644 index 000000000..cd339c9d7 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/guard/engine_Meta-Llama-3.1-8B-Instruct-4bit_r3_base.log @@ -0,0 +1,26 @@ +2026-10-09T02:49:43+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:49:43+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:49:58+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:49:58+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-engine -m models/mlx/Meta-Llama-3.1-8B-Instruct-4bit --path both --decode-storage dense --prompt-tokens 256,2048 -n 128 --warmup-tokens 20 --csv /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/measure/engine/rounds.csv --label base-r3 +2026-10-09T02:49:59+0900 sample 1: clean gpu=[] +2026-10-09T02:50:00+0900 sample 2: clean gpu=[1302244:mlxcel-bench-en] +2026-10-09T02:50:02+0900 sample 3: clean gpu=[1302244:mlxcel-bench-en] +2026-10-09T02:50:03+0900 sample 4: clean gpu=[1302244:mlxcel-bench-en] +2026-10-09T02:50:05+0900 sample 5: clean gpu=[1302244:mlxcel-bench-en] +2026-10-09T02:50:06+0900 sample 6: clean gpu=[1302244:mlxcel-bench-en] +2026-10-09T02:50:07+0900 sample 7: clean gpu=[1302244:mlxcel-bench-en] +2026-10-09T02:50:09+0900 sample 8: clean gpu=[1302244:mlxcel-bench-en] +2026-10-09T02:50:11+0900 sample 9: clean gpu=[1302244:mlxcel-bench-en] +2026-10-09T02:50:12+0900 sample 10: clean gpu=[1302244:mlxcel-bench-en] +2026-10-09T02:50:14+0900 sample 11: clean gpu=[1302244:mlxcel-bench-en] +2026-10-09T02:50:15+0900 sample 12: clean gpu=[1302244:mlxcel-bench-en] +2026-10-09T02:50:17+0900 sample 13: clean gpu=[1302244:mlxcel-bench-en] +2026-10-09T02:50:18+0900 sample 14: clean gpu=[1302244:mlxcel-bench-en] +2026-10-09T02:50:20+0900 sample 15: clean gpu=[1302244:mlxcel-bench-en] +2026-10-09T02:50:21+0900 sample 16: clean gpu=[1302244:mlxcel-bench-en] +2026-10-09T02:50:23+0900 sample 17: clean gpu=[1302244:mlxcel-bench-en] +2026-10-09T02:50:24+0900 sample 18: clean gpu=[1302244:mlxcel-bench-en] +2026-10-09T02:50:26+0900 sample 19: clean gpu=[1302244:mlxcel-bench-en] +2026-10-09T02:50:27+0900 sample 20: clean gpu=[1302244:mlxcel-bench-en] +2026-10-09T02:50:28+0900 monitor: 20 samples +2026-10-09T02:50:28+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/guard/engine_Meta-Llama-3.1-8B-Instruct-4bit_r3_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/guard/engine_Meta-Llama-3.1-8B-Instruct-4bit_r3_new.log new file mode 100644 index 000000000..6bc98da11 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/guard/engine_Meta-Llama-3.1-8B-Instruct-4bit_r3_new.log @@ -0,0 +1,25 @@ +2026-10-09T02:50:33+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:50:33+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:50:48+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:50:48+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-engine -m models/mlx/Meta-Llama-3.1-8B-Instruct-4bit --path both --decode-storage dense --prompt-tokens 256,2048 -n 128 --warmup-tokens 20 --csv /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/measure/engine/rounds.csv --label new-r3 +2026-10-09T02:50:48+0900 sample 1: clean gpu=[] +2026-10-09T02:50:50+0900 sample 2: clean gpu=[1330719:mlxcel-bench-en] +2026-10-09T02:50:51+0900 sample 3: clean gpu=[1330719:mlxcel-bench-en] +2026-10-09T02:50:53+0900 sample 4: clean gpu=[1330719:mlxcel-bench-en] +2026-10-09T02:50:54+0900 sample 5: clean gpu=[1330719:mlxcel-bench-en] +2026-10-09T02:50:56+0900 sample 6: clean gpu=[1330719:mlxcel-bench-en] +2026-10-09T02:50:57+0900 sample 7: clean gpu=[1330719:mlxcel-bench-en] +2026-10-09T02:50:59+0900 sample 8: clean gpu=[1330719:mlxcel-bench-en] +2026-10-09T02:51:00+0900 sample 9: clean gpu=[1330719:mlxcel-bench-en] +2026-10-09T02:51:02+0900 sample 10: clean gpu=[1330719:mlxcel-bench-en] +2026-10-09T02:51:03+0900 sample 11: clean gpu=[1330719:mlxcel-bench-en] +2026-10-09T02:51:05+0900 sample 12: clean gpu=[1330719:mlxcel-bench-en] +2026-10-09T02:51:06+0900 sample 13: clean gpu=[1330719:mlxcel-bench-en] +2026-10-09T02:51:08+0900 sample 14: clean gpu=[1330719:mlxcel-bench-en] +2026-10-09T02:51:09+0900 sample 15: clean gpu=[1330719:mlxcel-bench-en] +2026-10-09T02:51:10+0900 sample 16: clean gpu=[1330719:mlxcel-bench-en] +2026-10-09T02:51:12+0900 sample 17: clean gpu=[1330719:mlxcel-bench-en] +2026-10-09T02:51:14+0900 sample 18: clean gpu=[1330719:mlxcel-bench-en] +2026-10-09T02:51:15+0900 sample 19: clean gpu=[1330719:mlxcel-bench-en] +2026-10-09T02:51:16+0900 monitor: 19 samples +2026-10-09T02:51:16+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/guard/engine_Qwen3-0.6B-4bit_r1_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/guard/engine_Qwen3-0.6B-4bit_r1_base.log new file mode 100644 index 000000000..9a74ed523 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/guard/engine_Qwen3-0.6B-4bit_r1_base.log @@ -0,0 +1,12 @@ +2026-10-09T02:43:42+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:43:42+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:43:56+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:43:56+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-engine -m models/mlx/Qwen3-0.6B-4bit --path both --decode-storage dense --prompt-tokens 256,2048 -n 128 --warmup-tokens 20 --csv /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/measure/engine/rounds.csv --label base-r1 +2026-10-09T02:43:57+0900 sample 1: clean gpu=[] +2026-10-09T02:43:58+0900 sample 2: clean gpu=[1101389:mlxcel-bench-en] +2026-10-09T02:44:00+0900 sample 3: clean gpu=[1101389:mlxcel-bench-en] +2026-10-09T02:44:01+0900 sample 4: clean gpu=[1101389:mlxcel-bench-en] +2026-10-09T02:44:03+0900 sample 5: clean gpu=[1101389:mlxcel-bench-en] +2026-10-09T02:44:04+0900 sample 6: clean gpu=[1101389:mlxcel-bench-en] +2026-10-09T02:44:05+0900 monitor: 6 samples +2026-10-09T02:44:05+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/guard/engine_Qwen3-0.6B-4bit_r1_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/guard/engine_Qwen3-0.6B-4bit_r1_new.log new file mode 100644 index 000000000..62d8bfdae --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/guard/engine_Qwen3-0.6B-4bit_r1_new.log @@ -0,0 +1,11 @@ +2026-10-09T02:44:10+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:44:10+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:44:25+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:44:25+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-engine -m models/mlx/Qwen3-0.6B-4bit --path both --decode-storage dense --prompt-tokens 256,2048 -n 128 --warmup-tokens 20 --csv /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/measure/engine/rounds.csv --label new-r1 +2026-10-09T02:44:25+0900 sample 1: clean gpu=[] +2026-10-09T02:44:27+0900 sample 2: clean gpu=[1116524:mlxcel-bench-en] +2026-10-09T02:44:28+0900 sample 3: clean gpu=[1116524:mlxcel-bench-en] +2026-10-09T02:44:30+0900 sample 4: clean gpu=[1116524:mlxcel-bench-en] +2026-10-09T02:44:31+0900 sample 5: clean gpu=[1116524:mlxcel-bench-en] +2026-10-09T02:44:32+0900 monitor: 5 samples +2026-10-09T02:44:32+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/guard/engine_Qwen3-0.6B-4bit_r2_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/guard/engine_Qwen3-0.6B-4bit_r2_base.log new file mode 100644 index 000000000..480427df9 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/guard/engine_Qwen3-0.6B-4bit_r2_base.log @@ -0,0 +1,11 @@ +2026-10-09T02:46:18+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:46:18+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:46:32+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:46:32+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-engine -m models/mlx/Qwen3-0.6B-4bit --path both --decode-storage dense --prompt-tokens 256,2048 -n 128 --warmup-tokens 20 --csv /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/measure/engine/rounds.csv --label base-r2 +2026-10-09T02:46:33+0900 sample 1: clean gpu=[] +2026-10-09T02:46:34+0900 sample 2: clean gpu=[1188757:mlxcel-bench-en] +2026-10-09T02:46:36+0900 sample 3: clean gpu=[1188757:mlxcel-bench-en] +2026-10-09T02:46:37+0900 sample 4: clean gpu=[1188757:mlxcel-bench-en] +2026-10-09T02:46:39+0900 sample 5: clean gpu=[1188757:mlxcel-bench-en] +2026-10-09T02:46:40+0900 monitor: 5 samples +2026-10-09T02:46:40+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/guard/engine_Qwen3-0.6B-4bit_r2_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/guard/engine_Qwen3-0.6B-4bit_r2_new.log new file mode 100644 index 000000000..15e3b009c --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/guard/engine_Qwen3-0.6B-4bit_r2_new.log @@ -0,0 +1,11 @@ +2026-10-09T02:46:45+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:46:45+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:46:59+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:46:59+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-engine -m models/mlx/Qwen3-0.6B-4bit --path both --decode-storage dense --prompt-tokens 256,2048 -n 128 --warmup-tokens 20 --csv /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/measure/engine/rounds.csv --label new-r2 +2026-10-09T02:47:00+0900 sample 1: clean gpu=[] +2026-10-09T02:47:01+0900 sample 2: clean gpu=[1203050:mlxcel-bench-en] +2026-10-09T02:47:03+0900 sample 3: clean gpu=[1203050:mlxcel-bench-en] +2026-10-09T02:47:04+0900 sample 4: clean gpu=[1203050:mlxcel-bench-en] +2026-10-09T02:47:06+0900 sample 5: clean gpu=[1203050:mlxcel-bench-en] +2026-10-09T02:47:07+0900 monitor: 5 samples +2026-10-09T02:47:07+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/guard/engine_Qwen3-0.6B-4bit_r3_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/guard/engine_Qwen3-0.6B-4bit_r3_base.log new file mode 100644 index 000000000..76e93a976 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/guard/engine_Qwen3-0.6B-4bit_r3_base.log @@ -0,0 +1,11 @@ +2026-10-09T02:48:49+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:48:49+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:49:04+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:49:04+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-engine -m models/mlx/Qwen3-0.6B-4bit --path both --decode-storage dense --prompt-tokens 256,2048 -n 128 --warmup-tokens 20 --csv /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/measure/engine/rounds.csv --label base-r3 +2026-10-09T02:49:04+0900 sample 1: clean gpu=[] +2026-10-09T02:49:06+0900 sample 2: clean gpu=[1273600:mlxcel-bench-en] +2026-10-09T02:49:07+0900 sample 3: clean gpu=[1273600:mlxcel-bench-en] +2026-10-09T02:49:09+0900 sample 4: clean gpu=[1273600:mlxcel-bench-en] +2026-10-09T02:49:10+0900 sample 5: clean gpu=[1273600:mlxcel-bench-en] +2026-10-09T02:49:11+0900 monitor: 5 samples +2026-10-09T02:49:11+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/guard/engine_Qwen3-0.6B-4bit_r3_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/guard/engine_Qwen3-0.6B-4bit_r3_new.log new file mode 100644 index 000000000..c78b3a5c8 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/guard/engine_Qwen3-0.6B-4bit_r3_new.log @@ -0,0 +1,11 @@ +2026-10-09T02:49:16+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:49:16+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:49:31+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:49:31+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-engine -m models/mlx/Qwen3-0.6B-4bit --path both --decode-storage dense --prompt-tokens 256,2048 -n 128 --warmup-tokens 20 --csv /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/measure/engine/rounds.csv --label new-r3 +2026-10-09T02:49:31+0900 sample 1: clean gpu=[] +2026-10-09T02:49:33+0900 sample 2: clean gpu=[1287900:mlxcel-bench-en] +2026-10-09T02:49:34+0900 sample 3: clean gpu=[1287900:mlxcel-bench-en] +2026-10-09T02:49:36+0900 sample 4: clean gpu=[1287900:mlxcel-bench-en] +2026-10-09T02:49:37+0900 sample 5: clean gpu=[1287900:mlxcel-bench-en] +2026-10-09T02:49:38+0900 monitor: 5 samples +2026-10-09T02:49:38+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/guard/gemma3_probe.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/guard/gemma3_probe.log new file mode 100644 index 000000000..3c3c94426 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/guard/gemma3_probe.log @@ -0,0 +1,9 @@ +2026-10-09T02:51:39+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:51:39+0900 waiting for 20s of idle GPU and no compiler +2026-10-09T02:52:08+0900 idle for 20s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:52:08+0900 attempt 1/3: start: bash /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/measure/gemma3_probe.sh +2026-10-09T02:52:08+0900 sample 1: clean gpu=[] +2026-10-09T02:52:10+0900 sample 2: clean gpu=[1378646:mlxcel] +2026-10-09T02:52:11+0900 sample 3: clean gpu=[1378646:mlxcel] +2026-10-09T02:52:12+0900 monitor: 3 samples +2026-10-09T02:52:12+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/guard/paged_pool_past_u32.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/guard/paged_pool_past_u32.log new file mode 100644 index 000000000..37baaff75 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/guard/paged_pool_past_u32.log @@ -0,0 +1,8 @@ +2026-10-09T02:51:21+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:51:21+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:51:36+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:51:36+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/test-fast/deps/mlxcel_core-480914a0f733fc77 paged_pool_past_u32_elements_matches_gather --ignored --test-threads=1 --nocapture +2026-10-09T02:51:36+0900 sample 1: clean gpu=[] +2026-10-09T02:51:38+0900 sample 2: clean gpu=[1358129:mlxcel_core-480] +2026-10-09T02:51:39+0900 monitor: 2 samples +2026-10-09T02:51:39+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r1_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r1_base-null.log new file mode 100644 index 000000000..5f2d4aac8 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r1_base-null.log @@ -0,0 +1,11 @@ +2026-10-09T01:47:04+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T01:47:04+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T01:47:18+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T01:47:18+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Meta-Llama-3.1-8B-Instruct-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T01:47:19+0900 sample 1: clean gpu=[] +2026-10-09T01:47:20+0900 sample 2: clean gpu=[3394666:mlxcel-bench-de] +2026-10-09T01:47:22+0900 sample 3: clean gpu=[3394666:mlxcel-bench-de] +2026-10-09T01:47:24+0900 sample 4: clean gpu=[3394666:mlxcel-bench-de] +2026-10-09T01:47:25+0900 sample 5: clean gpu=[3394666:mlxcel-bench-de] +2026-10-09T01:47:26+0900 monitor: 5 samples +2026-10-09T01:47:26+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r1_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r1_base.log new file mode 100644 index 000000000..97ee9e57d --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r1_base.log @@ -0,0 +1,13 @@ +2026-10-09T01:46:06+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T01:46:06+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T01:46:21+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T01:46:21+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Meta-Llama-3.1-8B-Instruct-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T01:46:21+0900 sample 1: clean gpu=[] +2026-10-09T01:46:23+0900 sample 2: clean gpu=[3363796:mlxcel-bench-de] +2026-10-09T01:46:24+0900 sample 3: clean gpu=[3363796:mlxcel-bench-de] +2026-10-09T01:46:26+0900 sample 4: clean gpu=[3363796:mlxcel-bench-de] +2026-10-09T01:46:27+0900 sample 5: clean gpu=[3363796:mlxcel-bench-de] +2026-10-09T01:46:29+0900 sample 6: clean gpu=[3363796:mlxcel-bench-de] +2026-10-09T01:46:30+0900 sample 7: clean gpu=[3363796:mlxcel-bench-de] +2026-10-09T01:46:31+0900 monitor: 7 samples +2026-10-09T01:46:31+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r1_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r1_new.log new file mode 100644 index 000000000..539d1606b --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r1_new.log @@ -0,0 +1,11 @@ +2026-10-09T01:46:36+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T01:46:36+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T01:46:51+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T01:46:51+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/Meta-Llama-3.1-8B-Instruct-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T01:46:52+0900 sample 1: clean gpu=[] +2026-10-09T01:46:53+0900 sample 2: clean gpu=[3380296:mlxcel-bench-de] +2026-10-09T01:46:55+0900 sample 3: clean gpu=[3380296:mlxcel-bench-de] +2026-10-09T01:46:56+0900 sample 4: clean gpu=[3380296:mlxcel-bench-de] +2026-10-09T01:46:58+0900 sample 5: clean gpu=[3380296:mlxcel-bench-de] +2026-10-09T01:46:59+0900 monitor: 5 samples +2026-10-09T01:46:59+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r2_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r2_base-null.log new file mode 100644 index 000000000..f82c9e230 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r2_base-null.log @@ -0,0 +1,11 @@ +2026-10-09T02:06:37+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:06:37+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:06:52+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:06:52+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Meta-Llama-3.1-8B-Instruct-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:06:53+0900 sample 1: clean gpu=[] +2026-10-09T02:06:54+0900 sample 2: clean gpu=[4075655:mlxcel-bench-de] +2026-10-09T02:06:56+0900 sample 3: clean gpu=[4075655:mlxcel-bench-de] +2026-10-09T02:06:57+0900 sample 4: clean gpu=[4075655:mlxcel-bench-de] +2026-10-09T02:06:59+0900 sample 5: clean gpu=[4075655:mlxcel-bench-de] +2026-10-09T02:07:00+0900 monitor: 5 samples +2026-10-09T02:07:00+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r2_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r2_base.log new file mode 100644 index 000000000..67cc4e352 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r2_base.log @@ -0,0 +1,11 @@ +2026-10-09T02:07:05+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:07:05+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:07:20+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:07:20+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Meta-Llama-3.1-8B-Instruct-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:07:20+0900 sample 1: clean gpu=[] +2026-10-09T02:07:21+0900 sample 2: clean gpu=[4089997:mlxcel-bench-de] +2026-10-09T02:07:23+0900 sample 3: clean gpu=[4089997:mlxcel-bench-de] +2026-10-09T02:07:25+0900 sample 4: clean gpu=[4089997:mlxcel-bench-de] +2026-10-09T02:07:26+0900 sample 5: clean gpu=[4089997:mlxcel-bench-de] +2026-10-09T02:07:27+0900 monitor: 5 samples +2026-10-09T02:07:27+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r2_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r2_new.log new file mode 100644 index 000000000..867349bea --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r2_new.log @@ -0,0 +1,13 @@ +2026-10-09T02:06:07+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:06:07+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:06:22+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:06:22+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/Meta-Llama-3.1-8B-Instruct-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:06:22+0900 sample 1: clean gpu=[] +2026-10-09T02:06:24+0900 sample 2: clean gpu=[4059413:mlxcel-bench-de] +2026-10-09T02:06:25+0900 sample 3: clean gpu=[4059413:mlxcel-bench-de] +2026-10-09T02:06:27+0900 sample 4: clean gpu=[4059413:mlxcel-bench-de] +2026-10-09T02:06:28+0900 sample 5: clean gpu=[4059413:mlxcel-bench-de] +2026-10-09T02:06:30+0900 sample 6: clean gpu=[4059413:mlxcel-bench-de] +2026-10-09T02:06:31+0900 sample 7: clean gpu=[4059413:mlxcel-bench-de] +2026-10-09T02:06:32+0900 monitor: 7 samples +2026-10-09T02:06:32+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r3_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r3_base-null.log new file mode 100644 index 000000000..e7d25323d --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r3_base-null.log @@ -0,0 +1,13 @@ +2026-10-09T02:26:00+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:26:00+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:26:15+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:26:15+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Meta-Llama-3.1-8B-Instruct-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:26:15+0900 sample 1: clean gpu=[] +2026-10-09T02:26:17+0900 sample 2: clean gpu=[515159:mlxcel-bench-de] +2026-10-09T02:26:18+0900 sample 3: clean gpu=[515159:mlxcel-bench-de] +2026-10-09T02:26:20+0900 sample 4: clean gpu=[515159:mlxcel-bench-de] +2026-10-09T02:26:21+0900 sample 5: clean gpu=[515159:mlxcel-bench-de] +2026-10-09T02:26:23+0900 sample 6: clean gpu=[515159:mlxcel-bench-de] +2026-10-09T02:26:24+0900 sample 7: clean gpu=[515159:mlxcel-bench-de] +2026-10-09T02:26:25+0900 monitor: 7 samples +2026-10-09T02:26:25+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r3_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r3_base.log new file mode 100644 index 000000000..f4da0a005 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r3_base.log @@ -0,0 +1,11 @@ +2026-10-09T02:26:30+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:26:30+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:26:45+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:26:45+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Meta-Llama-3.1-8B-Instruct-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:26:46+0900 sample 1: clean gpu=[] +2026-10-09T02:26:47+0900 sample 2: clean gpu=[531413:mlxcel-bench-de] +2026-10-09T02:26:49+0900 sample 3: clean gpu=[531413:mlxcel-bench-de] +2026-10-09T02:26:50+0900 sample 4: clean gpu=[531413:mlxcel-bench-de] +2026-10-09T02:26:52+0900 sample 5: clean gpu=[531413:mlxcel-bench-de] +2026-10-09T02:26:53+0900 monitor: 5 samples +2026-10-09T02:26:53+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r3_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r3_new.log new file mode 100644 index 000000000..33907e75e --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r3_new.log @@ -0,0 +1,11 @@ +2026-10-09T02:26:58+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:26:58+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:27:12+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:27:12+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/Meta-Llama-3.1-8B-Instruct-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:27:13+0900 sample 1: clean gpu=[] +2026-10-09T02:27:14+0900 sample 2: clean gpu=[545723:mlxcel-bench-de] +2026-10-09T02:27:16+0900 sample 3: clean gpu=[545723:mlxcel-bench-de] +2026-10-09T02:27:17+0900 sample 4: clean gpu=[545723:mlxcel-bench-de] +2026-10-09T02:27:19+0900 sample 5: clean gpu=[545723:mlxcel-bench-de] +2026-10-09T02:27:20+0900 monitor: 5 samples +2026-10-09T02:27:20+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r1_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r1_base-null.log new file mode 100644 index 000000000..c505ef67b --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r1_base-null.log @@ -0,0 +1,44 @@ +2026-10-09T02:02:31+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:02:31+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:02:46+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:02:46+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Mixtral-8x7B-Instruct-v0.1-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:02:46+0900 sample 1: clean gpu=[] +2026-10-09T02:02:48+0900 sample 2: clean gpu=[3944782:mlxcel-bench-de] +2026-10-09T02:02:49+0900 sample 3: clean gpu=[3944782:mlxcel-bench-de] +2026-10-09T02:02:51+0900 sample 4: clean gpu=[3944782:mlxcel-bench-de] +2026-10-09T02:02:52+0900 sample 5: clean gpu=[3944782:mlxcel-bench-de] +2026-10-09T02:02:54+0900 sample 6: clean gpu=[3944782:mlxcel-bench-de] +2026-10-09T02:02:55+0900 sample 7: clean gpu=[3944782:mlxcel-bench-de] +2026-10-09T02:02:57+0900 sample 8: clean gpu=[3944782:mlxcel-bench-de] +2026-10-09T02:02:58+0900 sample 9: clean gpu=[3944782:mlxcel-bench-de] +2026-10-09T02:02:59+0900 sample 10: clean gpu=[3944782:mlxcel-bench-de] +2026-10-09T02:03:01+0900 sample 11: clean gpu=[3944782:mlxcel-bench-de] +2026-10-09T02:03:02+0900 sample 12: clean gpu=[3944782:mlxcel-bench-de] +2026-10-09T02:03:04+0900 sample 13: clean gpu=[3944782:mlxcel-bench-de] +2026-10-09T02:03:05+0900 sample 14: clean gpu=[3944782:mlxcel-bench-de] +2026-10-09T02:03:07+0900 sample 15: clean gpu=[3944782:mlxcel-bench-de] +2026-10-09T02:03:08+0900 sample 16: clean gpu=[3944782:mlxcel-bench-de] +2026-10-09T02:03:10+0900 sample 17: clean gpu=[3944782:mlxcel-bench-de] +2026-10-09T02:03:11+0900 sample 18: clean gpu=[3944782:mlxcel-bench-de] +2026-10-09T02:03:13+0900 sample 19: clean gpu=[3944782:mlxcel-bench-de] +2026-10-09T02:03:14+0900 sample 20: clean gpu=[3944782:mlxcel-bench-de] +2026-10-09T02:03:16+0900 sample 21: clean gpu=[3944782:mlxcel-bench-de] +2026-10-09T02:03:17+0900 sample 22: clean gpu=[3944782:mlxcel-bench-de] +2026-10-09T02:03:19+0900 sample 23: clean gpu=[3944782:mlxcel-bench-de] +2026-10-09T02:03:21+0900 sample 24: clean gpu=[3944782:mlxcel-bench-de] +2026-10-09T02:03:22+0900 sample 25: clean gpu=[3944782:mlxcel-bench-de] +2026-10-09T02:03:24+0900 sample 26: clean gpu=[3944782:mlxcel-bench-de] +2026-10-09T02:03:26+0900 sample 27: clean gpu=[3944782:mlxcel-bench-de] +2026-10-09T02:03:28+0900 sample 28: clean gpu=[3944782:mlxcel-bench-de] +2026-10-09T02:03:29+0900 sample 29: clean gpu=[3944782:mlxcel-bench-de] +2026-10-09T02:03:31+0900 sample 30: clean gpu=[3944782:mlxcel-bench-de] +2026-10-09T02:03:32+0900 sample 31: clean gpu=[3944782:mlxcel-bench-de] +2026-10-09T02:03:34+0900 sample 32: clean gpu=[3944782:mlxcel-bench-de] +2026-10-09T02:03:35+0900 sample 33: clean gpu=[3944782:mlxcel-bench-de] +2026-10-09T02:03:37+0900 sample 34: clean gpu=[3944782:mlxcel-bench-de] +2026-10-09T02:03:38+0900 sample 35: clean gpu=[3944782:mlxcel-bench-de] +2026-10-09T02:03:40+0900 sample 36: clean gpu=[3944782:mlxcel-bench-de] +2026-10-09T02:03:41+0900 sample 37: clean gpu=[3944782:mlxcel-bench-de] +2026-10-09T02:03:43+0900 sample 38: clean gpu=[3944782:mlxcel-bench-de] +2026-10-09T02:03:44+0900 monitor: 38 samples +2026-10-09T02:03:44+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r1_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r1_base.log new file mode 100644 index 000000000..23e9423d9 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r1_base.log @@ -0,0 +1,45 @@ +2026-10-09T01:59:53+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T01:59:53+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:00:07+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:00:07+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Mixtral-8x7B-Instruct-v0.1-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:00:08+0900 sample 1: clean gpu=[] +2026-10-09T02:00:09+0900 sample 2: clean gpu=[3852117:mlxcel-bench-de] +2026-10-09T02:00:11+0900 sample 3: clean gpu=[3852117:mlxcel-bench-de] +2026-10-09T02:00:12+0900 sample 4: clean gpu=[3852117:mlxcel-bench-de] +2026-10-09T02:00:14+0900 sample 5: clean gpu=[3852117:mlxcel-bench-de] +2026-10-09T02:00:15+0900 sample 6: clean gpu=[3852117:mlxcel-bench-de] +2026-10-09T02:00:17+0900 sample 7: clean gpu=[3852117:mlxcel-bench-de] +2026-10-09T02:00:18+0900 sample 8: clean gpu=[3852117:mlxcel-bench-de] +2026-10-09T02:00:20+0900 sample 9: clean gpu=[3852117:mlxcel-bench-de] +2026-10-09T02:00:21+0900 sample 10: clean gpu=[3852117:mlxcel-bench-de] +2026-10-09T02:00:23+0900 sample 11: clean gpu=[3852117:mlxcel-bench-de] +2026-10-09T02:00:24+0900 sample 12: clean gpu=[3852117:mlxcel-bench-de] +2026-10-09T02:00:26+0900 sample 13: clean gpu=[3852117:mlxcel-bench-de] +2026-10-09T02:00:27+0900 sample 14: clean gpu=[3852117:mlxcel-bench-de] +2026-10-09T02:00:29+0900 sample 15: clean gpu=[3852117:mlxcel-bench-de] +2026-10-09T02:00:31+0900 sample 16: clean gpu=[3852117:mlxcel-bench-de] +2026-10-09T02:00:32+0900 sample 17: clean gpu=[3852117:mlxcel-bench-de] +2026-10-09T02:00:33+0900 sample 18: clean gpu=[3852117:mlxcel-bench-de] +2026-10-09T02:00:35+0900 sample 19: clean gpu=[3852117:mlxcel-bench-de] +2026-10-09T02:00:36+0900 sample 20: clean gpu=[3852117:mlxcel-bench-de] +2026-10-09T02:00:38+0900 sample 21: clean gpu=[3852117:mlxcel-bench-de] +2026-10-09T02:00:39+0900 sample 22: clean gpu=[3852117:mlxcel-bench-de] +2026-10-09T02:00:41+0900 sample 23: clean gpu=[3852117:mlxcel-bench-de] +2026-10-09T02:00:42+0900 sample 24: clean gpu=[3852117:mlxcel-bench-de] +2026-10-09T02:00:44+0900 sample 25: clean gpu=[3852117:mlxcel-bench-de] +2026-10-09T02:00:45+0900 sample 26: clean gpu=[3852117:mlxcel-bench-de] +2026-10-09T02:00:47+0900 sample 27: clean gpu=[3852117:mlxcel-bench-de] +2026-10-09T02:00:49+0900 sample 28: clean gpu=[3852117:mlxcel-bench-de] +2026-10-09T02:00:51+0900 sample 29: clean gpu=[3852117:mlxcel-bench-de] +2026-10-09T02:00:52+0900 sample 30: clean gpu=[3852117:mlxcel-bench-de] +2026-10-09T02:00:54+0900 sample 31: clean gpu=[3852117:mlxcel-bench-de] +2026-10-09T02:00:55+0900 sample 32: clean gpu=[3852117:mlxcel-bench-de] +2026-10-09T02:00:57+0900 sample 33: clean gpu=[3852117:mlxcel-bench-de] +2026-10-09T02:00:58+0900 sample 34: clean gpu=[3852117:mlxcel-bench-de] +2026-10-09T02:01:00+0900 sample 35: clean gpu=[3852117:mlxcel-bench-de] +2026-10-09T02:01:01+0900 sample 36: clean gpu=[3852117:mlxcel-bench-de] +2026-10-09T02:01:03+0900 sample 37: clean gpu=[3852117:mlxcel-bench-de] +2026-10-09T02:01:04+0900 sample 38: clean gpu=[3852117:mlxcel-bench-de] +2026-10-09T02:01:06+0900 sample 39: clean gpu=[3852117:mlxcel-bench-de] +2026-10-09T02:01:07+0900 monitor: 39 samples +2026-10-09T02:01:07+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r1_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r1_new.log new file mode 100644 index 000000000..02f99aa8b --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r1_new.log @@ -0,0 +1,45 @@ +2026-10-09T02:01:12+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:01:12+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:01:27+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:01:27+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/Mixtral-8x7B-Instruct-v0.1-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:01:27+0900 sample 1: clean gpu=[] +2026-10-09T02:01:29+0900 sample 2: clean gpu=[3899155:mlxcel-bench-de] +2026-10-09T02:01:30+0900 sample 3: clean gpu=[3899155:mlxcel-bench-de] +2026-10-09T02:01:32+0900 sample 4: clean gpu=[3899155:mlxcel-bench-de] +2026-10-09T02:01:33+0900 sample 5: clean gpu=[3899155:mlxcel-bench-de] +2026-10-09T02:01:35+0900 sample 6: clean gpu=[3899155:mlxcel-bench-de] +2026-10-09T02:01:36+0900 sample 7: clean gpu=[3899155:mlxcel-bench-de] +2026-10-09T02:01:38+0900 sample 8: clean gpu=[3899155:mlxcel-bench-de] +2026-10-09T02:01:39+0900 sample 9: clean gpu=[3899155:mlxcel-bench-de] +2026-10-09T02:01:41+0900 sample 10: clean gpu=[3899155:mlxcel-bench-de] +2026-10-09T02:01:42+0900 sample 11: clean gpu=[3899155:mlxcel-bench-de] +2026-10-09T02:01:44+0900 sample 12: clean gpu=[3899155:mlxcel-bench-de] +2026-10-09T02:01:45+0900 sample 13: clean gpu=[3899155:mlxcel-bench-de] +2026-10-09T02:01:47+0900 sample 14: clean gpu=[3899155:mlxcel-bench-de] +2026-10-09T02:01:48+0900 sample 15: clean gpu=[3899155:mlxcel-bench-de] +2026-10-09T02:01:50+0900 sample 16: clean gpu=[3899155:mlxcel-bench-de] +2026-10-09T02:01:51+0900 sample 17: clean gpu=[3899155:mlxcel-bench-de] +2026-10-09T02:01:53+0900 sample 18: clean gpu=[3899155:mlxcel-bench-de] +2026-10-09T02:01:54+0900 sample 19: clean gpu=[3899155:mlxcel-bench-de] +2026-10-09T02:01:56+0900 sample 20: clean gpu=[3899155:mlxcel-bench-de] +2026-10-09T02:01:57+0900 sample 21: clean gpu=[3899155:mlxcel-bench-de] +2026-10-09T02:01:58+0900 sample 22: clean gpu=[3899155:mlxcel-bench-de] +2026-10-09T02:02:00+0900 sample 23: clean gpu=[3899155:mlxcel-bench-de] +2026-10-09T02:02:01+0900 sample 24: clean gpu=[3899155:mlxcel-bench-de] +2026-10-09T02:02:03+0900 sample 25: clean gpu=[3899155:mlxcel-bench-de] +2026-10-09T02:02:05+0900 sample 26: clean gpu=[3899155:mlxcel-bench-de] +2026-10-09T02:02:07+0900 sample 27: clean gpu=[3899155:mlxcel-bench-de] +2026-10-09T02:02:08+0900 sample 28: clean gpu=[3899155:mlxcel-bench-de] +2026-10-09T02:02:10+0900 sample 29: clean gpu=[3899155:mlxcel-bench-de] +2026-10-09T02:02:12+0900 sample 30: clean gpu=[3899155:mlxcel-bench-de] +2026-10-09T02:02:13+0900 sample 31: clean gpu=[3899155:mlxcel-bench-de] +2026-10-09T02:02:15+0900 sample 32: clean gpu=[3899155:mlxcel-bench-de] +2026-10-09T02:02:16+0900 sample 33: clean gpu=[3899155:mlxcel-bench-de] +2026-10-09T02:02:18+0900 sample 34: clean gpu=[3899155:mlxcel-bench-de] +2026-10-09T02:02:19+0900 sample 35: clean gpu=[3899155:mlxcel-bench-de] +2026-10-09T02:02:21+0900 sample 36: clean gpu=[3899155:mlxcel-bench-de] +2026-10-09T02:02:22+0900 sample 37: clean gpu=[3899155:mlxcel-bench-de] +2026-10-09T02:02:24+0900 sample 38: clean gpu=[3899155:mlxcel-bench-de] +2026-10-09T02:02:25+0900 sample 39: clean gpu=[3899155:mlxcel-bench-de] +2026-10-09T02:02:26+0900 monitor: 39 samples +2026-10-09T02:02:26+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r2_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r2_base-null.log new file mode 100644 index 000000000..474ba7a9f --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r2_base-null.log @@ -0,0 +1,46 @@ +2026-10-09T02:21:04+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:21:04+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:21:19+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:21:19+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Mixtral-8x7B-Instruct-v0.1-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:21:19+0900 sample 1: clean gpu=[] +2026-10-09T02:21:21+0900 sample 2: clean gpu=[355365:mlxcel-bench-de] +2026-10-09T02:21:22+0900 sample 3: clean gpu=[355365:mlxcel-bench-de] +2026-10-09T02:21:24+0900 sample 4: clean gpu=[355365:mlxcel-bench-de] +2026-10-09T02:21:25+0900 sample 5: clean gpu=[355365:mlxcel-bench-de] +2026-10-09T02:21:27+0900 sample 6: clean gpu=[355365:mlxcel-bench-de] +2026-10-09T02:21:28+0900 sample 7: clean gpu=[355365:mlxcel-bench-de] +2026-10-09T02:21:30+0900 sample 8: clean gpu=[355365:mlxcel-bench-de] +2026-10-09T02:21:31+0900 sample 9: clean gpu=[355365:mlxcel-bench-de] +2026-10-09T02:21:33+0900 sample 10: clean gpu=[355365:mlxcel-bench-de] +2026-10-09T02:21:34+0900 sample 11: clean gpu=[355365:mlxcel-bench-de] +2026-10-09T02:21:36+0900 sample 12: clean gpu=[355365:mlxcel-bench-de] +2026-10-09T02:21:37+0900 sample 13: clean gpu=[355365:mlxcel-bench-de] +2026-10-09T02:21:39+0900 sample 14: clean gpu=[355365:mlxcel-bench-de] +2026-10-09T02:21:40+0900 sample 15: clean gpu=[355365:mlxcel-bench-de] +2026-10-09T02:21:42+0900 sample 16: clean gpu=[355365:mlxcel-bench-de] +2026-10-09T02:21:43+0900 sample 17: clean gpu=[355365:mlxcel-bench-de] +2026-10-09T02:21:45+0900 sample 18: clean gpu=[355365:mlxcel-bench-de] +2026-10-09T02:21:46+0900 sample 19: clean gpu=[355365:mlxcel-bench-de] +2026-10-09T02:21:48+0900 sample 20: clean gpu=[355365:mlxcel-bench-de] +2026-10-09T02:21:49+0900 sample 21: clean gpu=[355365:mlxcel-bench-de] +2026-10-09T02:21:50+0900 sample 22: clean gpu=[355365:mlxcel-bench-de] +2026-10-09T02:21:52+0900 sample 23: clean gpu=[355365:mlxcel-bench-de] +2026-10-09T02:21:53+0900 sample 24: clean gpu=[355365:mlxcel-bench-de] +2026-10-09T02:21:55+0900 sample 25: clean gpu=[355365:mlxcel-bench-de] +2026-10-09T02:21:57+0900 sample 26: clean gpu=[355365:mlxcel-bench-de] +2026-10-09T02:21:58+0900 sample 27: clean gpu=[355365:mlxcel-bench-de] +2026-10-09T02:22:00+0900 sample 28: clean gpu=[355365:mlxcel-bench-de] +2026-10-09T02:22:02+0900 sample 29: clean gpu=[355365:mlxcel-bench-de] +2026-10-09T02:22:04+0900 sample 30: clean gpu=[355365:mlxcel-bench-de] +2026-10-09T02:22:05+0900 sample 31: clean gpu=[355365:mlxcel-bench-de] +2026-10-09T02:22:07+0900 sample 32: clean gpu=[355365:mlxcel-bench-de] +2026-10-09T02:22:08+0900 sample 33: clean gpu=[355365:mlxcel-bench-de] +2026-10-09T02:22:10+0900 sample 34: clean gpu=[355365:mlxcel-bench-de] +2026-10-09T02:22:11+0900 sample 35: clean gpu=[355365:mlxcel-bench-de] +2026-10-09T02:22:13+0900 sample 36: clean gpu=[355365:mlxcel-bench-de] +2026-10-09T02:22:14+0900 sample 37: clean gpu=[355365:mlxcel-bench-de] +2026-10-09T02:22:16+0900 sample 38: clean gpu=[355365:mlxcel-bench-de] +2026-10-09T02:22:17+0900 sample 39: clean gpu=[355365:mlxcel-bench-de] +2026-10-09T02:22:19+0900 sample 40: clean gpu=[355365:mlxcel-bench-de] +2026-10-09T02:22:20+0900 monitor: 40 samples +2026-10-09T02:22:20+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r2_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r2_base.log new file mode 100644 index 000000000..f0096413b --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r2_base.log @@ -0,0 +1,44 @@ +2026-10-09T02:22:25+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:22:25+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:22:39+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:22:39+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Mixtral-8x7B-Instruct-v0.1-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:22:40+0900 sample 1: clean gpu=[] +2026-10-09T02:22:41+0900 sample 2: clean gpu=[401749:mlxcel-bench-de] +2026-10-09T02:22:43+0900 sample 3: clean gpu=[401749:mlxcel-bench-de] +2026-10-09T02:22:44+0900 sample 4: clean gpu=[401749:mlxcel-bench-de] +2026-10-09T02:22:46+0900 sample 5: clean gpu=[401749:mlxcel-bench-de] +2026-10-09T02:22:47+0900 sample 6: clean gpu=[401749:mlxcel-bench-de] +2026-10-09T02:22:49+0900 sample 7: clean gpu=[401749:mlxcel-bench-de] +2026-10-09T02:22:50+0900 sample 8: clean gpu=[401749:mlxcel-bench-de] +2026-10-09T02:22:52+0900 sample 9: clean gpu=[401749:mlxcel-bench-de] +2026-10-09T02:22:53+0900 sample 10: clean gpu=[401749:mlxcel-bench-de] +2026-10-09T02:22:55+0900 sample 11: clean gpu=[401749:mlxcel-bench-de] +2026-10-09T02:22:56+0900 sample 12: clean gpu=[401749:mlxcel-bench-de] +2026-10-09T02:22:58+0900 sample 13: clean gpu=[401749:mlxcel-bench-de] +2026-10-09T02:23:00+0900 sample 14: clean gpu=[401749:mlxcel-bench-de] +2026-10-09T02:23:01+0900 sample 15: clean gpu=[401749:mlxcel-bench-de] +2026-10-09T02:23:02+0900 sample 16: clean gpu=[401749:mlxcel-bench-de] +2026-10-09T02:23:04+0900 sample 17: clean gpu=[401749:mlxcel-bench-de] +2026-10-09T02:23:05+0900 sample 18: clean gpu=[401749:mlxcel-bench-de] +2026-10-09T02:23:07+0900 sample 19: clean gpu=[401749:mlxcel-bench-de] +2026-10-09T02:23:08+0900 sample 20: clean gpu=[401749:mlxcel-bench-de] +2026-10-09T02:23:10+0900 sample 21: clean gpu=[401749:mlxcel-bench-de] +2026-10-09T02:23:11+0900 sample 22: clean gpu=[401749:mlxcel-bench-de] +2026-10-09T02:23:13+0900 sample 23: clean gpu=[401749:mlxcel-bench-de] +2026-10-09T02:23:14+0900 sample 24: clean gpu=[401749:mlxcel-bench-de] +2026-10-09T02:23:16+0900 sample 25: clean gpu=[401749:mlxcel-bench-de] +2026-10-09T02:23:18+0900 sample 26: clean gpu=[401749:mlxcel-bench-de] +2026-10-09T02:23:20+0900 sample 27: clean gpu=[401749:mlxcel-bench-de] +2026-10-09T02:23:22+0900 sample 28: clean gpu=[401749:mlxcel-bench-de] +2026-10-09T02:23:23+0900 sample 29: clean gpu=[401749:mlxcel-bench-de] +2026-10-09T02:23:25+0900 sample 30: clean gpu=[401749:mlxcel-bench-de] +2026-10-09T02:23:26+0900 sample 31: clean gpu=[401749:mlxcel-bench-de] +2026-10-09T02:23:28+0900 sample 32: clean gpu=[401749:mlxcel-bench-de] +2026-10-09T02:23:29+0900 sample 33: clean gpu=[401749:mlxcel-bench-de] +2026-10-09T02:23:31+0900 sample 34: clean gpu=[401749:mlxcel-bench-de] +2026-10-09T02:23:32+0900 sample 35: clean gpu=[401749:mlxcel-bench-de] +2026-10-09T02:23:34+0900 sample 36: clean gpu=[401749:mlxcel-bench-de] +2026-10-09T02:23:36+0900 sample 37: clean gpu=[401749:mlxcel-bench-de] +2026-10-09T02:23:37+0900 sample 38: clean gpu=[401749:mlxcel-bench-de] +2026-10-09T02:23:38+0900 monitor: 38 samples +2026-10-09T02:23:38+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r2_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r2_new.log new file mode 100644 index 000000000..b696cc2bb --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r2_new.log @@ -0,0 +1,45 @@ +2026-10-09T02:19:45+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:19:45+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:20:00+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:20:00+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/Mixtral-8x7B-Instruct-v0.1-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:20:00+0900 sample 1: clean gpu=[] +2026-10-09T02:20:02+0900 sample 2: clean gpu=[309913:mlxcel-bench-de] +2026-10-09T02:20:03+0900 sample 3: clean gpu=[309913:mlxcel-bench-de] +2026-10-09T02:20:05+0900 sample 4: clean gpu=[309913:mlxcel-bench-de] +2026-10-09T02:20:06+0900 sample 5: clean gpu=[309913:mlxcel-bench-de] +2026-10-09T02:20:08+0900 sample 6: clean gpu=[309913:mlxcel-bench-de] +2026-10-09T02:20:09+0900 sample 7: clean gpu=[309913:mlxcel-bench-de] +2026-10-09T02:20:11+0900 sample 8: clean gpu=[309913:mlxcel-bench-de] +2026-10-09T02:20:12+0900 sample 9: clean gpu=[309913:mlxcel-bench-de] +2026-10-09T02:20:14+0900 sample 10: clean gpu=[309913:mlxcel-bench-de] +2026-10-09T02:20:15+0900 sample 11: clean gpu=[309913:mlxcel-bench-de] +2026-10-09T02:20:17+0900 sample 12: clean gpu=[309913:mlxcel-bench-de] +2026-10-09T02:20:18+0900 sample 13: clean gpu=[309913:mlxcel-bench-de] +2026-10-09T02:20:20+0900 sample 14: clean gpu=[309913:mlxcel-bench-de] +2026-10-09T02:20:22+0900 sample 15: clean gpu=[309913:mlxcel-bench-de] +2026-10-09T02:20:23+0900 sample 16: clean gpu=[309913:mlxcel-bench-de] +2026-10-09T02:20:25+0900 sample 17: clean gpu=[309913:mlxcel-bench-de] +2026-10-09T02:20:26+0900 sample 18: clean gpu=[309913:mlxcel-bench-de] +2026-10-09T02:20:27+0900 sample 19: clean gpu=[309913:mlxcel-bench-de] +2026-10-09T02:20:29+0900 sample 20: clean gpu=[309913:mlxcel-bench-de] +2026-10-09T02:20:30+0900 sample 21: clean gpu=[309913:mlxcel-bench-de] +2026-10-09T02:20:32+0900 sample 22: clean gpu=[309913:mlxcel-bench-de] +2026-10-09T02:20:33+0900 sample 23: clean gpu=[309913:mlxcel-bench-de] +2026-10-09T02:20:35+0900 sample 24: clean gpu=[309913:mlxcel-bench-de] +2026-10-09T02:20:36+0900 sample 25: clean gpu=[309913:mlxcel-bench-de] +2026-10-09T02:20:38+0900 sample 26: clean gpu=[309913:mlxcel-bench-de] +2026-10-09T02:20:40+0900 sample 27: clean gpu=[309913:mlxcel-bench-de] +2026-10-09T02:20:41+0900 sample 28: clean gpu=[309913:mlxcel-bench-de] +2026-10-09T02:20:43+0900 sample 29: clean gpu=[309913:mlxcel-bench-de] +2026-10-09T02:20:45+0900 sample 30: clean gpu=[309913:mlxcel-bench-de] +2026-10-09T02:20:46+0900 sample 31: clean gpu=[309913:mlxcel-bench-de] +2026-10-09T02:20:48+0900 sample 32: clean gpu=[309913:mlxcel-bench-de] +2026-10-09T02:20:49+0900 sample 33: clean gpu=[309913:mlxcel-bench-de] +2026-10-09T02:20:51+0900 sample 34: clean gpu=[309913:mlxcel-bench-de] +2026-10-09T02:20:52+0900 sample 35: clean gpu=[309913:mlxcel-bench-de] +2026-10-09T02:20:54+0900 sample 36: clean gpu=[309913:mlxcel-bench-de] +2026-10-09T02:20:55+0900 sample 37: clean gpu=[309913:mlxcel-bench-de] +2026-10-09T02:20:57+0900 sample 38: clean gpu=[309913:mlxcel-bench-de] +2026-10-09T02:20:58+0900 sample 39: clean gpu=[309913:mlxcel-bench-de] +2026-10-09T02:20:59+0900 monitor: 39 samples +2026-10-09T02:20:59+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r3_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r3_base-null.log new file mode 100644 index 000000000..602d4aab4 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r3_base-null.log @@ -0,0 +1,45 @@ +2026-10-09T02:39:31+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:39:31+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:39:45+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:39:45+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Mixtral-8x7B-Instruct-v0.1-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:39:46+0900 sample 1: clean gpu=[] +2026-10-09T02:39:47+0900 sample 2: clean gpu=[953720:mlxcel-bench-de] +2026-10-09T02:39:49+0900 sample 3: clean gpu=[953720:mlxcel-bench-de] +2026-10-09T02:39:50+0900 sample 4: clean gpu=[953720:mlxcel-bench-de] +2026-10-09T02:39:52+0900 sample 5: clean gpu=[953720:mlxcel-bench-de] +2026-10-09T02:39:53+0900 sample 6: clean gpu=[953720:mlxcel-bench-de] +2026-10-09T02:39:55+0900 sample 7: clean gpu=[953720:mlxcel-bench-de] +2026-10-09T02:39:56+0900 sample 8: clean gpu=[953720:mlxcel-bench-de] +2026-10-09T02:39:58+0900 sample 9: clean gpu=[953720:mlxcel-bench-de] +2026-10-09T02:39:59+0900 sample 10: clean gpu=[953720:mlxcel-bench-de] +2026-10-09T02:40:01+0900 sample 11: clean gpu=[953720:mlxcel-bench-de] +2026-10-09T02:40:02+0900 sample 12: clean gpu=[953720:mlxcel-bench-de] +2026-10-09T02:40:04+0900 sample 13: clean gpu=[953720:mlxcel-bench-de] +2026-10-09T02:40:05+0900 sample 14: clean gpu=[953720:mlxcel-bench-de] +2026-10-09T02:40:07+0900 sample 15: clean gpu=[953720:mlxcel-bench-de] +2026-10-09T02:40:08+0900 sample 16: clean gpu=[953720:mlxcel-bench-de] +2026-10-09T02:40:10+0900 sample 17: clean gpu=[953720:mlxcel-bench-de] +2026-10-09T02:40:11+0900 sample 18: clean gpu=[953720:mlxcel-bench-de] +2026-10-09T02:40:12+0900 sample 19: clean gpu=[953720:mlxcel-bench-de] +2026-10-09T02:40:14+0900 sample 20: clean gpu=[953720:mlxcel-bench-de] +2026-10-09T02:40:15+0900 sample 21: clean gpu=[953720:mlxcel-bench-de] +2026-10-09T02:40:17+0900 sample 22: clean gpu=[953720:mlxcel-bench-de] +2026-10-09T02:40:18+0900 sample 23: clean gpu=[953720:mlxcel-bench-de] +2026-10-09T02:40:20+0900 sample 24: clean gpu=[953720:mlxcel-bench-de] +2026-10-09T02:40:21+0900 sample 25: clean gpu=[953720:mlxcel-bench-de] +2026-10-09T02:40:23+0900 sample 26: clean gpu=[953720:mlxcel-bench-de] +2026-10-09T02:40:25+0900 sample 27: clean gpu=[953720:mlxcel-bench-de] +2026-10-09T02:40:26+0900 sample 28: clean gpu=[953720:mlxcel-bench-de] +2026-10-09T02:40:28+0900 sample 29: clean gpu=[953720:mlxcel-bench-de] +2026-10-09T02:40:30+0900 sample 30: clean gpu=[953720:mlxcel-bench-de] +2026-10-09T02:40:31+0900 sample 31: clean gpu=[953720:mlxcel-bench-de] +2026-10-09T02:40:33+0900 sample 32: clean gpu=[953720:mlxcel-bench-de] +2026-10-09T02:40:34+0900 sample 33: clean gpu=[953720:mlxcel-bench-de] +2026-10-09T02:40:36+0900 sample 34: clean gpu=[953720:mlxcel-bench-de] +2026-10-09T02:40:37+0900 sample 35: clean gpu=[953720:mlxcel-bench-de] +2026-10-09T02:40:39+0900 sample 36: clean gpu=[953720:mlxcel-bench-de] +2026-10-09T02:40:41+0900 sample 37: clean gpu=[953720:mlxcel-bench-de] +2026-10-09T02:40:42+0900 sample 38: clean gpu=[953720:mlxcel-bench-de] +2026-10-09T02:40:44+0900 sample 39: clean gpu=[953720:mlxcel-bench-de] +2026-10-09T02:40:45+0900 monitor: 39 samples +2026-10-09T02:40:45+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r3_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r3_base.log new file mode 100644 index 000000000..5089cd132 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r3_base.log @@ -0,0 +1,54 @@ +2026-10-09T02:40:50+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:40:50+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:41:04+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:41:04+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Mixtral-8x7B-Instruct-v0.1-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:41:05+0900 sample 1: clean gpu=[] +2026-10-09T02:41:06+0900 sample 2: clean gpu=[1000365:mlxcel-bench-de] +2026-10-09T02:41:08+0900 sample 3: clean gpu=[1000365:mlxcel-bench-de] +2026-10-09T02:41:09+0900 sample 4: clean gpu=[1000365:mlxcel-bench-de] +2026-10-09T02:41:11+0900 sample 5: clean gpu=[1000365:mlxcel-bench-de] +2026-10-09T02:41:13+0900 sample 6: clean gpu=[1000365:mlxcel-bench-de] +2026-10-09T02:41:14+0900 sample 7: clean gpu=[1000365:mlxcel-bench-de] +2026-10-09T02:41:16+0900 sample 8: clean gpu=[1000365:mlxcel-bench-de] +2026-10-09T02:41:17+0900 sample 9: clean gpu=[1000365:mlxcel-bench-de] +2026-10-09T02:41:19+0900 sample 10: clean gpu=[1000365:mlxcel-bench-de] +2026-10-09T02:41:20+0900 sample 11: clean gpu=[1000365:mlxcel-bench-de] +2026-10-09T02:41:22+0900 sample 12: clean gpu=[1000365:mlxcel-bench-de] +2026-10-09T02:41:23+0900 sample 13: clean gpu=[1000365:mlxcel-bench-de] +2026-10-09T02:41:25+0900 sample 14: clean gpu=[1000365:mlxcel-bench-de] +2026-10-09T02:41:26+0900 sample 15: clean gpu=[1000365:mlxcel-bench-de] +2026-10-09T02:41:28+0900 sample 16: clean gpu=[1000365:mlxcel-bench-de] +2026-10-09T02:41:29+0900 sample 17: clean gpu=[1000365:mlxcel-bench-de] +2026-10-09T02:41:31+0900 sample 18: clean gpu=[1000365:mlxcel-bench-de] +2026-10-09T02:41:32+0900 sample 19: clean gpu=[1000365:mlxcel-bench-de] +2026-10-09T02:41:34+0900 sample 20: clean gpu=[1000365:mlxcel-bench-de] +2026-10-09T02:41:35+0900 sample 21: clean gpu=[1000365:mlxcel-bench-de] +2026-10-09T02:41:37+0900 sample 22: clean gpu=[1000365:mlxcel-bench-de] +2026-10-09T02:41:38+0900 sample 23: clean gpu=[1000365:mlxcel-bench-de] +2026-10-09T02:41:40+0900 sample 24: clean gpu=[1000365:mlxcel-bench-de] +2026-10-09T02:41:41+0900 sample 25: clean gpu=[1000365:mlxcel-bench-de] +2026-10-09T02:41:43+0900 sample 26: clean gpu=[1000365:mlxcel-bench-de] +2026-10-09T02:41:45+0900 sample 27: clean gpu=[1000365:mlxcel-bench-de] +2026-10-09T02:41:46+0900 sample 28: clean gpu=[1000365:mlxcel-bench-de] +2026-10-09T02:41:47+0900 sample 29: clean gpu=[1000365:mlxcel-bench-de] +2026-10-09T02:41:49+0900 sample 30: clean gpu=[1000365:mlxcel-bench-de] +2026-10-09T02:41:50+0900 sample 31: clean gpu=[1000365:mlxcel-bench-de] +2026-10-09T02:41:52+0900 sample 32: clean gpu=[1000365:mlxcel-bench-de] +2026-10-09T02:41:53+0900 sample 33: clean gpu=[1000365:mlxcel-bench-de] +2026-10-09T02:41:55+0900 sample 34: clean gpu=[1000365:mlxcel-bench-de] +2026-10-09T02:41:56+0900 sample 35: clean gpu=[1000365:mlxcel-bench-de] +2026-10-09T02:41:58+0900 sample 36: clean gpu=[1000365:mlxcel-bench-de] +2026-10-09T02:42:00+0900 sample 37: clean gpu=[1000365:mlxcel-bench-de] +2026-10-09T02:42:01+0900 sample 38: clean gpu=[1000365:mlxcel-bench-de] +2026-10-09T02:42:03+0900 sample 39: clean gpu=[1000365:mlxcel-bench-de] +2026-10-09T02:42:04+0900 sample 40: clean gpu=[1000365:mlxcel-bench-de] +2026-10-09T02:42:06+0900 sample 41: clean gpu=[1000365:mlxcel-bench-de] +2026-10-09T02:42:08+0900 sample 42: clean gpu=[1000365:mlxcel-bench-de] +2026-10-09T02:42:09+0900 sample 43: clean gpu=[1000365:mlxcel-bench-de] +2026-10-09T02:42:11+0900 sample 44: clean gpu=[1000365:mlxcel-bench-de] +2026-10-09T02:42:12+0900 sample 45: clean gpu=[1000365:mlxcel-bench-de] +2026-10-09T02:42:14+0900 sample 46: clean gpu=[1000365:mlxcel-bench-de] +2026-10-09T02:42:15+0900 sample 47: clean gpu=[1000365:mlxcel-bench-de] +2026-10-09T02:42:17+0900 sample 48: clean gpu=[1000365:mlxcel-bench-de] +2026-10-09T02:42:18+0900 monitor: 48 samples +2026-10-09T02:42:18+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r3_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r3_new.log new file mode 100644 index 000000000..cbe1b6f6d --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r3_new.log @@ -0,0 +1,44 @@ +2026-10-09T02:42:23+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:42:23+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:42:38+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:42:38+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/Mixtral-8x7B-Instruct-v0.1-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:42:38+0900 sample 1: clean gpu=[] +2026-10-09T02:42:40+0900 sample 2: clean gpu=[1055761:mlxcel-bench-de] +2026-10-09T02:42:41+0900 sample 3: clean gpu=[1055761:mlxcel-bench-de] +2026-10-09T02:42:43+0900 sample 4: clean gpu=[1055761:mlxcel-bench-de] +2026-10-09T02:42:44+0900 sample 5: clean gpu=[1055761:mlxcel-bench-de] +2026-10-09T02:42:46+0900 sample 6: clean gpu=[1055761:mlxcel-bench-de] +2026-10-09T02:42:47+0900 sample 7: clean gpu=[1055761:mlxcel-bench-de] +2026-10-09T02:42:49+0900 sample 8: clean gpu=[1055761:mlxcel-bench-de] +2026-10-09T02:42:50+0900 sample 9: clean gpu=[1055761:mlxcel-bench-de] +2026-10-09T02:42:52+0900 sample 10: clean gpu=[1055761:mlxcel-bench-de] +2026-10-09T02:42:53+0900 sample 11: clean gpu=[1055761:mlxcel-bench-de] +2026-10-09T02:42:55+0900 sample 12: clean gpu=[1055761:mlxcel-bench-de] +2026-10-09T02:42:56+0900 sample 13: clean gpu=[1055761:mlxcel-bench-de] +2026-10-09T02:42:58+0900 sample 14: clean gpu=[1055761:mlxcel-bench-de] +2026-10-09T02:42:59+0900 sample 15: clean gpu=[1055761:mlxcel-bench-de] +2026-10-09T02:43:01+0900 sample 16: clean gpu=[1055761:mlxcel-bench-de] +2026-10-09T02:43:03+0900 sample 17: clean gpu=[1055761:mlxcel-bench-de] +2026-10-09T02:43:04+0900 sample 18: clean gpu=[1055761:mlxcel-bench-de] +2026-10-09T02:43:05+0900 sample 19: clean gpu=[1055761:mlxcel-bench-de] +2026-10-09T02:43:07+0900 sample 20: clean gpu=[1055761:mlxcel-bench-de] +2026-10-09T02:43:08+0900 sample 21: clean gpu=[1055761:mlxcel-bench-de] +2026-10-09T02:43:10+0900 sample 22: clean gpu=[1055761:mlxcel-bench-de] +2026-10-09T02:43:11+0900 sample 23: clean gpu=[1055761:mlxcel-bench-de] +2026-10-09T02:43:13+0900 sample 24: clean gpu=[1055761:mlxcel-bench-de] +2026-10-09T02:43:15+0900 sample 25: clean gpu=[1055761:mlxcel-bench-de] +2026-10-09T02:43:17+0900 sample 26: clean gpu=[1055761:mlxcel-bench-de] +2026-10-09T02:43:18+0900 sample 27: clean gpu=[1055761:mlxcel-bench-de] +2026-10-09T02:43:20+0900 sample 28: clean gpu=[1055761:mlxcel-bench-de] +2026-10-09T02:43:22+0900 sample 29: clean gpu=[1055761:mlxcel-bench-de] +2026-10-09T02:43:24+0900 sample 30: clean gpu=[1055761:mlxcel-bench-de] +2026-10-09T02:43:25+0900 sample 31: clean gpu=[1055761:mlxcel-bench-de] +2026-10-09T02:43:27+0900 sample 32: clean gpu=[1055761:mlxcel-bench-de] +2026-10-09T02:43:28+0900 sample 33: clean gpu=[1055761:mlxcel-bench-de] +2026-10-09T02:43:30+0900 sample 34: clean gpu=[1055761:mlxcel-bench-de] +2026-10-09T02:43:31+0900 sample 35: clean gpu=[1055761:mlxcel-bench-de] +2026-10-09T02:43:33+0900 sample 36: clean gpu=[1055761:mlxcel-bench-de] +2026-10-09T02:43:34+0900 sample 37: clean gpu=[1055761:mlxcel-bench-de] +2026-10-09T02:43:36+0900 sample 38: clean gpu=[1055761:mlxcel-bench-de] +2026-10-09T02:43:37+0900 monitor: 38 samples +2026-10-09T02:43:37+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r1_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r1_base-null.log new file mode 100644 index 000000000..98074ebd5 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r1_base-null.log @@ -0,0 +1,16 @@ +2026-10-09T01:56:56+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T01:56:56+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T01:57:11+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T01:57:11+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T01:57:11+0900 sample 1: clean gpu=[] +2026-10-09T01:57:13+0900 sample 2: clean gpu=[3752289:mlxcel-bench-de] +2026-10-09T01:57:15+0900 sample 3: clean gpu=[3752289:mlxcel-bench-de] +2026-10-09T01:57:16+0900 sample 4: clean gpu=[3752289:mlxcel-bench-de] +2026-10-09T01:57:18+0900 sample 5: clean gpu=[3752289:mlxcel-bench-de] +2026-10-09T01:57:19+0900 sample 6: clean gpu=[3752289:mlxcel-bench-de] +2026-10-09T01:57:21+0900 sample 7: clean gpu=[3752289:mlxcel-bench-de] +2026-10-09T01:57:22+0900 sample 8: clean gpu=[3752289:mlxcel-bench-de] +2026-10-09T01:57:24+0900 sample 9: clean gpu=[3752289:mlxcel-bench-de] +2026-10-09T01:57:25+0900 sample 10: clean gpu=[3752289:mlxcel-bench-de] +2026-10-09T01:57:26+0900 monitor: 10 samples +2026-10-09T01:57:26+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r1_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r1_base.log new file mode 100644 index 000000000..aadd9d79e --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r1_base.log @@ -0,0 +1,17 @@ +2026-10-09T01:55:45+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T01:55:45+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T01:56:00+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T01:56:00+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T01:56:00+0900 sample 1: clean gpu=[] +2026-10-09T01:56:02+0900 sample 2: clean gpu=[3713056:mlxcel-bench-de] +2026-10-09T01:56:03+0900 sample 3: clean gpu=[3713056:mlxcel-bench-de] +2026-10-09T01:56:05+0900 sample 4: clean gpu=[3713056:mlxcel-bench-de] +2026-10-09T01:56:06+0900 sample 5: clean gpu=[3713056:mlxcel-bench-de] +2026-10-09T01:56:08+0900 sample 6: clean gpu=[3713056:mlxcel-bench-de] +2026-10-09T01:56:09+0900 sample 7: clean gpu=[3713056:mlxcel-bench-de] +2026-10-09T01:56:11+0900 sample 8: clean gpu=[3713056:mlxcel-bench-de] +2026-10-09T01:56:12+0900 sample 9: clean gpu=[3713056:mlxcel-bench-de] +2026-10-09T01:56:14+0900 sample 10: clean gpu=[3713056:mlxcel-bench-de] +2026-10-09T01:56:15+0900 sample 11: clean gpu=[3713056:mlxcel-bench-de] +2026-10-09T01:56:16+0900 monitor: 11 samples +2026-10-09T01:56:16+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r1_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r1_new.log new file mode 100644 index 000000000..918416306 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r1_new.log @@ -0,0 +1,16 @@ +2026-10-09T01:56:21+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T01:56:21+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T01:56:36+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T01:56:36+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T01:56:36+0900 sample 1: clean gpu=[] +2026-10-09T01:56:38+0900 sample 2: clean gpu=[3733061:mlxcel-bench-de] +2026-10-09T01:56:39+0900 sample 3: clean gpu=[3733061:mlxcel-bench-de] +2026-10-09T01:56:41+0900 sample 4: clean gpu=[3733061:mlxcel-bench-de] +2026-10-09T01:56:43+0900 sample 5: clean gpu=[3733061:mlxcel-bench-de] +2026-10-09T01:56:44+0900 sample 6: clean gpu=[3733061:mlxcel-bench-de] +2026-10-09T01:56:46+0900 sample 7: clean gpu=[3733061:mlxcel-bench-de] +2026-10-09T01:56:47+0900 sample 8: clean gpu=[3733061:mlxcel-bench-de] +2026-10-09T01:56:49+0900 sample 9: clean gpu=[3733061:mlxcel-bench-de] +2026-10-09T01:56:50+0900 sample 10: clean gpu=[3733061:mlxcel-bench-de] +2026-10-09T01:56:51+0900 monitor: 10 samples +2026-10-09T01:56:51+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r2_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r2_base-null.log new file mode 100644 index 000000000..1833fb3e1 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r2_base-null.log @@ -0,0 +1,16 @@ +2026-10-09T02:16:12+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:16:12+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:16:27+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:16:27+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:16:27+0900 sample 1: clean gpu=[] +2026-10-09T02:16:29+0900 sample 2: clean gpu=[193318:mlxcel-bench-de] +2026-10-09T02:16:30+0900 sample 3: clean gpu=[193318:mlxcel-bench-de] +2026-10-09T02:16:32+0900 sample 4: clean gpu=[193318:mlxcel-bench-de] +2026-10-09T02:16:33+0900 sample 5: clean gpu=[193318:mlxcel-bench-de] +2026-10-09T02:16:35+0900 sample 6: clean gpu=[193318:mlxcel-bench-de] +2026-10-09T02:16:36+0900 sample 7: clean gpu=[193318:mlxcel-bench-de] +2026-10-09T02:16:38+0900 sample 8: clean gpu=[193318:mlxcel-bench-de] +2026-10-09T02:16:39+0900 sample 9: clean gpu=[193318:mlxcel-bench-de] +2026-10-09T02:16:41+0900 sample 10: clean gpu=[193318:mlxcel-bench-de] +2026-10-09T02:16:42+0900 monitor: 10 samples +2026-10-09T02:16:42+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r2_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r2_base.log new file mode 100644 index 000000000..321a1d51a --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r2_base.log @@ -0,0 +1,16 @@ +2026-10-09T02:16:47+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:16:47+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:17:01+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:17:01+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:17:02+0900 sample 1: clean gpu=[] +2026-10-09T02:17:03+0900 sample 2: clean gpu=[211756:mlxcel-bench-de] +2026-10-09T02:17:05+0900 sample 3: clean gpu=[211756:mlxcel-bench-de] +2026-10-09T02:17:06+0900 sample 4: clean gpu=[211756:mlxcel-bench-de] +2026-10-09T02:17:08+0900 sample 5: clean gpu=[211756:mlxcel-bench-de] +2026-10-09T02:17:09+0900 sample 6: clean gpu=[211756:mlxcel-bench-de] +2026-10-09T02:17:11+0900 sample 7: clean gpu=[211756:mlxcel-bench-de] +2026-10-09T02:17:12+0900 sample 8: clean gpu=[211756:mlxcel-bench-de] +2026-10-09T02:17:14+0900 sample 9: clean gpu=[211756:mlxcel-bench-de] +2026-10-09T02:17:15+0900 sample 10: clean gpu=[211756:mlxcel-bench-de] +2026-10-09T02:17:16+0900 monitor: 10 samples +2026-10-09T02:17:16+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r2_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r2_new.log new file mode 100644 index 000000000..75cf8e9f6 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r2_new.log @@ -0,0 +1,17 @@ +2026-10-09T02:15:36+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:15:36+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:15:51+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:15:51+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:15:51+0900 sample 1: clean gpu=[] +2026-10-09T02:15:53+0900 sample 2: clean gpu=[174046:mlxcel-bench-de] +2026-10-09T02:15:54+0900 sample 3: clean gpu=[174046:mlxcel-bench-de] +2026-10-09T02:15:56+0900 sample 4: clean gpu=[174046:mlxcel-bench-de] +2026-10-09T02:15:57+0900 sample 5: clean gpu=[174046:mlxcel-bench-de] +2026-10-09T02:15:59+0900 sample 6: clean gpu=[174046:mlxcel-bench-de] +2026-10-09T02:16:00+0900 sample 7: clean gpu=[174046:mlxcel-bench-de] +2026-10-09T02:16:02+0900 sample 8: clean gpu=[174046:mlxcel-bench-de] +2026-10-09T02:16:03+0900 sample 9: clean gpu=[174046:mlxcel-bench-de] +2026-10-09T02:16:05+0900 sample 10: clean gpu=[174046:mlxcel-bench-de] +2026-10-09T02:16:06+0900 sample 11: clean gpu=[174046:mlxcel-bench-de] +2026-10-09T02:16:07+0900 monitor: 11 samples +2026-10-09T02:16:07+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r3_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r3_base-null.log new file mode 100644 index 000000000..8726437a5 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r3_base-null.log @@ -0,0 +1,17 @@ +2026-10-09T02:35:21+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:35:21+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:35:36+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:35:36+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:35:36+0900 sample 1: clean gpu=[] +2026-10-09T02:35:38+0900 sample 2: clean gpu=[816050:mlxcel-bench-de] +2026-10-09T02:35:39+0900 sample 3: clean gpu=[816050:mlxcel-bench-de] +2026-10-09T02:35:41+0900 sample 4: clean gpu=[816050:mlxcel-bench-de] +2026-10-09T02:35:42+0900 sample 5: clean gpu=[816050:mlxcel-bench-de] +2026-10-09T02:35:44+0900 sample 6: clean gpu=[816050:mlxcel-bench-de] +2026-10-09T02:35:45+0900 sample 7: clean gpu=[816050:mlxcel-bench-de] +2026-10-09T02:35:47+0900 sample 8: clean gpu=[816050:mlxcel-bench-de] +2026-10-09T02:35:48+0900 sample 9: clean gpu=[816050:mlxcel-bench-de] +2026-10-09T02:35:50+0900 sample 10: clean gpu=[816050:mlxcel-bench-de] +2026-10-09T02:35:51+0900 sample 11: clean gpu=[816050:mlxcel-bench-de] +2026-10-09T02:35:52+0900 monitor: 11 samples +2026-10-09T02:35:52+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r3_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r3_base.log new file mode 100644 index 000000000..75c9e792a --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r3_base.log @@ -0,0 +1,16 @@ +2026-10-09T02:35:57+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:35:57+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:36:12+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:36:12+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:36:12+0900 sample 1: clean gpu=[] +2026-10-09T02:36:14+0900 sample 2: clean gpu=[835434:mlxcel-bench-de] +2026-10-09T02:36:15+0900 sample 3: clean gpu=[835434:mlxcel-bench-de] +2026-10-09T02:36:17+0900 sample 4: clean gpu=[835434:mlxcel-bench-de] +2026-10-09T02:36:18+0900 sample 5: clean gpu=[835434:mlxcel-bench-de] +2026-10-09T02:36:20+0900 sample 6: clean gpu=[835434:mlxcel-bench-de] +2026-10-09T02:36:21+0900 sample 7: clean gpu=[835434:mlxcel-bench-de] +2026-10-09T02:36:23+0900 sample 8: clean gpu=[835434:mlxcel-bench-de] +2026-10-09T02:36:24+0900 sample 9: clean gpu=[835434:mlxcel-bench-de] +2026-10-09T02:36:26+0900 sample 10: clean gpu=[835434:mlxcel-bench-de] +2026-10-09T02:36:27+0900 monitor: 10 samples +2026-10-09T02:36:27+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r3_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r3_new.log new file mode 100644 index 000000000..d217ae2e4 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r3_new.log @@ -0,0 +1,16 @@ +2026-10-09T02:36:32+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:36:32+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:36:46+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:36:46+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:36:47+0900 sample 1: clean gpu=[] +2026-10-09T02:36:48+0900 sample 2: clean gpu=[853905:mlxcel-bench-de] +2026-10-09T02:36:50+0900 sample 3: clean gpu=[853905:mlxcel-bench-de] +2026-10-09T02:36:51+0900 sample 4: clean gpu=[853905:mlxcel-bench-de] +2026-10-09T02:36:53+0900 sample 5: clean gpu=[853905:mlxcel-bench-de] +2026-10-09T02:36:54+0900 sample 6: clean gpu=[853905:mlxcel-bench-de] +2026-10-09T02:36:56+0900 sample 7: clean gpu=[853905:mlxcel-bench-de] +2026-10-09T02:36:57+0900 sample 8: clean gpu=[853905:mlxcel-bench-de] +2026-10-09T02:36:59+0900 sample 9: clean gpu=[853905:mlxcel-bench-de] +2026-10-09T02:37:00+0900 sample 10: clean gpu=[853905:mlxcel-bench-de] +2026-10-09T02:37:01+0900 monitor: 10 samples +2026-10-09T02:37:01+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp2048_r1_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp2048_r1_base-null.log new file mode 100644 index 000000000..1741b9b09 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp2048_r1_base-null.log @@ -0,0 +1,11 @@ +2026-10-08T23:28:27+0900 waiting for guard lock /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T00:20:29+0900 caught SIGTERM: stopping command and monitor +2026-10-09T01:19:10+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T01:19:10+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T01:45:57+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T01:45:57+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-0.6B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +2026-10-09T01:45:57+0900 sample 1: clean gpu=[] +2026-10-09T01:45:59+0900 sample 2: clean gpu=[3351173:mlxcel-bench-de] +2026-10-09T01:46:00+0900 sample 3: clean gpu=[3351173:mlxcel-bench-de] +2026-10-09T01:46:01+0900 monitor: 3 samples +2026-10-09T01:46:01+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp2048_r1_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp2048_r1_base.log new file mode 100644 index 000000000..21eedf38a --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp2048_r1_base.log @@ -0,0 +1,12 @@ +2026-10-08T21:33:51+0900 waiting for guard lock /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-08T22:19:00+0900 acquired guard lock /tmp/mlxcel-rocm-gpu-guard.lock after 2709s +2026-10-08T22:19:00+0900 waiting for 60s of idle GPU and no compiler +2026-10-08T22:19:18+0900 idle streak reset after 12s: gpu=[] compilers=[1173803:cargo] +2026-10-08T22:23:33+0900 idle streak reset after 1s: gpu=[] compilers=[1334471:cargo] +2026-10-08T23:05:01+0900 idle for 60s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-08T23:05:01+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-0.6B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +2026-10-08T23:05:01+0900 sample 1: clean gpu=[] +2026-10-08T23:05:02+0900 sample 2: clean gpu=[2908527:mlxcel-bench-de] +2026-10-08T23:05:04+0900 sample 3: clean gpu=[2908527:mlxcel-bench-de] +2026-10-08T23:05:05+0900 monitor: 3 samples +2026-10-08T23:05:05+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp2048_r1_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp2048_r1_new.log new file mode 100644 index 000000000..2f164e6b9 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp2048_r1_new.log @@ -0,0 +1,12 @@ +2026-10-08T23:05:15+0900 waiting for guard lock /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-08T23:13:56+0900 acquired guard lock /tmp/mlxcel-rocm-gpu-guard.lock after 521s +2026-10-08T23:13:56+0900 waiting for 60s of idle GPU and no compiler +2026-10-08T23:14:31+0900 idle streak reset after 23s: gpu=[3262038:rocminfo] compilers=[3262000:cargo] +2026-10-08T23:23:02+0900 idle streak reset after 14s: gpu=[] compilers=[3571564:cargo,3571605:rustc,3571606:rustc,3571607:rustc,3571608:rustc,3571609:rustc] +2026-10-08T23:28:13+0900 idle for 60s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-08T23:28:13+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-0.6B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +2026-10-08T23:28:13+0900 sample 1: clean gpu=[] +2026-10-08T23:28:15+0900 sample 2: clean gpu=[3751605:mlxcel-bench-de] +2026-10-08T23:28:16+0900 sample 3: clean gpu=[3751605:mlxcel-bench-de] +2026-10-08T23:28:17+0900 monitor: 3 samples +2026-10-08T23:28:17+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp2048_r2_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp2048_r2_base-null.log new file mode 100644 index 000000000..6060db29e --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp2048_r2_base-null.log @@ -0,0 +1,9 @@ +2026-10-09T02:05:19+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:05:19+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:05:34+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:05:34+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-0.6B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +2026-10-09T02:05:34+0900 sample 1: clean gpu=[] +2026-10-09T02:05:36+0900 sample 2: clean gpu=[4034722:mlxcel-bench-de] +2026-10-09T02:05:37+0900 sample 3: clean gpu=[4034722:mlxcel-bench-de] +2026-10-09T02:05:38+0900 monitor: 3 samples +2026-10-09T02:05:38+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp2048_r2_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp2048_r2_base.log new file mode 100644 index 000000000..694241de9 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp2048_r2_base.log @@ -0,0 +1,9 @@ +2026-10-09T02:05:43+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:05:43+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:05:58+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:05:58+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-0.6B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +2026-10-09T02:05:58+0900 sample 1: clean gpu=[] +2026-10-09T02:06:00+0900 sample 2: clean gpu=[4047073:mlxcel-bench-de] +2026-10-09T02:06:01+0900 sample 3: clean gpu=[4047073:mlxcel-bench-de] +2026-10-09T02:06:02+0900 monitor: 3 samples +2026-10-09T02:06:02+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp2048_r2_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp2048_r2_new.log new file mode 100644 index 000000000..bd3302986 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp2048_r2_new.log @@ -0,0 +1,8 @@ +2026-10-09T02:04:57+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:04:57+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:05:11+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:05:11+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-0.6B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +2026-10-09T02:05:12+0900 sample 1: clean gpu=[] +2026-10-09T02:05:13+0900 sample 2: clean gpu=[4023270:mlxcel-bench-de] +2026-10-09T02:05:14+0900 monitor: 2 samples +2026-10-09T02:05:14+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp2048_r3_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp2048_r3_base-null.log new file mode 100644 index 000000000..5b350e952 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp2048_r3_base-null.log @@ -0,0 +1,8 @@ +2026-10-09T02:24:51+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:24:51+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:25:05+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:25:05+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-0.6B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +2026-10-09T02:25:06+0900 sample 1: clean gpu=[] +2026-10-09T02:25:07+0900 sample 2: clean gpu=[479954:mlxcel-bench-de] +2026-10-09T02:25:08+0900 monitor: 2 samples +2026-10-09T02:25:08+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp2048_r3_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp2048_r3_base.log new file mode 100644 index 000000000..8a3bbd7a2 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp2048_r3_base.log @@ -0,0 +1,9 @@ +2026-10-09T02:25:13+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:25:13+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:25:28+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:25:28+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-0.6B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +2026-10-09T02:25:28+0900 sample 1: clean gpu=[] +2026-10-09T02:25:30+0900 sample 2: clean gpu=[491350:mlxcel-bench-de] +2026-10-09T02:25:31+0900 sample 3: clean gpu=[491350:mlxcel-bench-de] +2026-10-09T02:25:32+0900 monitor: 3 samples +2026-10-09T02:25:32+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp2048_r3_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp2048_r3_new.log new file mode 100644 index 000000000..e15670b14 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp2048_r3_new.log @@ -0,0 +1,8 @@ +2026-10-09T02:25:37+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:25:37+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:25:52+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:25:52+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-0.6B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +2026-10-09T02:25:52+0900 sample 1: clean gpu=[] +2026-10-09T02:25:54+0900 sample 2: clean gpu=[503716:mlxcel-bench-de] +2026-10-09T02:25:55+0900 monitor: 2 samples +2026-10-09T02:25:55+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp512_r1_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp512_r1_base-null.log new file mode 100644 index 000000000..dfbcd6f75 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp512_r1_base-null.log @@ -0,0 +1,9 @@ +2026-10-08T21:22:36+0900 waiting for guard lock /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-08T21:32:09+0900 acquired guard lock /tmp/mlxcel-rocm-gpu-guard.lock after 573s +2026-10-08T21:32:09+0900 waiting for 60s of idle GPU and no compiler +2026-10-08T21:33:38+0900 idle for 60s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-08T21:33:38+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-0.6B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-08T21:33:39+0900 sample 1: clean gpu=[] +2026-10-08T21:33:40+0900 sample 2: clean gpu=[3633468:mlxcel-bench-de] +2026-10-08T21:33:41+0900 monitor: 2 samples +2026-10-08T21:33:41+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp512_r1_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp512_r1_base.log new file mode 100644 index 000000000..0f61f6c29 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp512_r1_base.log @@ -0,0 +1,13 @@ +2026-10-08T19:17:44+0900 waiting for guard lock /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-08T19:41:27+0900 acquired guard lock /tmp/mlxcel-rocm-gpu-guard.lock after 1423s +2026-10-08T19:41:27+0900 waiting for 60s of idle GPU and no compiler +2026-10-08T19:42:09+0900 idle streak reset after 28s: gpu=[] compilers=[3313432:cargo] +2026-10-08T20:39:08+0900 idle for 60s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-08T20:39:08+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-0.6B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-08T20:39:08+0900 sample 1: clean gpu=[] +2026-10-08T20:39:10+0900 sample 2: clean gpu=[] +2026-10-08T20:39:11+0900 sample 3: clean gpu=[1515758:mlxcel-bench-de] +2026-10-08T20:39:13+0900 sample 4: clean gpu=[1515758:mlxcel-bench-de] +2026-10-08T20:39:14+0900 sample 5: clean gpu=[1515758:mlxcel-bench-de] +2026-10-08T20:39:15+0900 monitor: 5 samples +2026-10-08T20:39:15+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp512_r1_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp512_r1_new.log new file mode 100644 index 000000000..72e8e1dd3 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp512_r1_new.log @@ -0,0 +1,9 @@ +2026-10-08T20:39:25+0900 waiting for guard lock /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-08T21:20:56+0900 acquired guard lock /tmp/mlxcel-rocm-gpu-guard.lock after 2491s +2026-10-08T21:20:56+0900 waiting for 60s of idle GPU and no compiler +2026-10-08T21:22:23+0900 idle for 60s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-08T21:22:23+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-0.6B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-08T21:22:23+0900 sample 1: clean gpu=[] +2026-10-08T21:22:25+0900 sample 2: clean gpu=[3213872:mlxcel-bench-de] +2026-10-08T21:22:26+0900 monitor: 2 samples +2026-10-08T21:22:26+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp512_r2_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp512_r2_base-null.log new file mode 100644 index 000000000..d99261dae --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp512_r2_base-null.log @@ -0,0 +1,8 @@ +2026-10-09T02:04:12+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:04:12+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:04:26+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:04:26+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-0.6B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:04:26+0900 sample 1: clean gpu=[] +2026-10-09T02:04:28+0900 sample 2: clean gpu=[4000296:mlxcel-bench-de] +2026-10-09T02:04:29+0900 monitor: 2 samples +2026-10-09T02:04:29+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp512_r2_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp512_r2_base.log new file mode 100644 index 000000000..55ff939a4 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp512_r2_base.log @@ -0,0 +1,8 @@ +2026-10-09T02:04:34+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:04:34+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:04:49+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:04:49+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-0.6B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:04:49+0900 sample 1: clean gpu=[] +2026-10-09T02:04:51+0900 sample 2: clean gpu=[4011756:mlxcel-bench-de] +2026-10-09T02:04:52+0900 monitor: 2 samples +2026-10-09T02:04:52+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp512_r2_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp512_r2_new.log new file mode 100644 index 000000000..2dace6b91 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp512_r2_new.log @@ -0,0 +1,8 @@ +2026-10-09T02:03:49+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:03:49+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:04:04+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:04:04+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-0.6B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:04:04+0900 sample 1: clean gpu=[] +2026-10-09T02:04:05+0900 sample 2: clean gpu=[3989149:mlxcel-bench-de] +2026-10-09T02:04:06+0900 monitor: 2 samples +2026-10-09T02:04:06+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp512_r3_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp512_r3_base-null.log new file mode 100644 index 000000000..b54f6047c --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp512_r3_base-null.log @@ -0,0 +1,8 @@ +2026-10-09T02:23:43+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:23:43+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:23:58+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:23:58+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-0.6B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:23:58+0900 sample 1: clean gpu=[] +2026-10-09T02:24:00+0900 sample 2: clean gpu=[446099:mlxcel-bench-de] +2026-10-09T02:24:01+0900 monitor: 2 samples +2026-10-09T02:24:01+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp512_r3_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp512_r3_base.log new file mode 100644 index 000000000..87434b7eb --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp512_r3_base.log @@ -0,0 +1,8 @@ +2026-10-09T02:24:06+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:24:06+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:24:20+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:24:20+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-0.6B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:24:20+0900 sample 1: clean gpu=[] +2026-10-09T02:24:22+0900 sample 2: clean gpu=[457164:mlxcel-bench-de] +2026-10-09T02:24:23+0900 monitor: 2 samples +2026-10-09T02:24:23+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp512_r3_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp512_r3_new.log new file mode 100644 index 000000000..721abb3c9 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-0.6B-4bit_pp512_r3_new.log @@ -0,0 +1,8 @@ +2026-10-09T02:24:28+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:24:28+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:24:43+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:24:43+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-0.6B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:24:43+0900 sample 1: clean gpu=[] +2026-10-09T02:24:45+0900 sample 2: clean gpu=[468540:mlxcel-bench-de] +2026-10-09T02:24:46+0900 monitor: 2 samples +2026-10-09T02:24:46+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp2048_r1_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp2048_r1_base-null.log new file mode 100644 index 000000000..161f3078a --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp2048_r1_base-null.log @@ -0,0 +1,23 @@ +2026-10-09T01:53:41+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T01:53:41+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T01:53:56+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T01:53:56+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +2026-10-09T01:53:56+0900 sample 1: clean gpu=[] +2026-10-09T01:53:58+0900 sample 2: clean gpu=[3646745:mlxcel-bench-de] +2026-10-09T01:53:59+0900 sample 3: clean gpu=[3646745:mlxcel-bench-de] +2026-10-09T01:54:01+0900 sample 4: clean gpu=[3646745:mlxcel-bench-de] +2026-10-09T01:54:02+0900 sample 5: clean gpu=[3646745:mlxcel-bench-de] +2026-10-09T01:54:04+0900 sample 6: clean gpu=[3646745:mlxcel-bench-de] +2026-10-09T01:54:05+0900 sample 7: clean gpu=[3646745:mlxcel-bench-de] +2026-10-09T01:54:07+0900 sample 8: clean gpu=[3646745:mlxcel-bench-de] +2026-10-09T01:54:08+0900 sample 9: clean gpu=[3646745:mlxcel-bench-de] +2026-10-09T01:54:09+0900 sample 10: clean gpu=[3646745:mlxcel-bench-de] +2026-10-09T01:54:11+0900 sample 11: clean gpu=[3646745:mlxcel-bench-de] +2026-10-09T01:54:12+0900 sample 12: clean gpu=[3646745:mlxcel-bench-de] +2026-10-09T01:54:14+0900 sample 13: clean gpu=[3646745:mlxcel-bench-de] +2026-10-09T01:54:15+0900 sample 14: clean gpu=[3646745:mlxcel-bench-de] +2026-10-09T01:54:17+0900 sample 15: clean gpu=[3646745:mlxcel-bench-de] +2026-10-09T01:54:18+0900 sample 16: clean gpu=[3646745:mlxcel-bench-de] +2026-10-09T01:54:20+0900 sample 17: clean gpu=[3646745:mlxcel-bench-de] +2026-10-09T01:54:21+0900 monitor: 17 samples +2026-10-09T01:54:21+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp2048_r1_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp2048_r1_base.log new file mode 100644 index 000000000..901482112 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp2048_r1_base.log @@ -0,0 +1,23 @@ +2026-10-09T01:52:12+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T01:52:12+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T01:52:26+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T01:52:26+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +2026-10-09T01:52:27+0900 sample 1: clean gpu=[] +2026-10-09T01:52:28+0900 sample 2: clean gpu=[3596927:mlxcel-bench-de] +2026-10-09T01:52:30+0900 sample 3: clean gpu=[3596927:mlxcel-bench-de] +2026-10-09T01:52:31+0900 sample 4: clean gpu=[3596927:mlxcel-bench-de] +2026-10-09T01:52:33+0900 sample 5: clean gpu=[3596927:mlxcel-bench-de] +2026-10-09T01:52:34+0900 sample 6: clean gpu=[3596927:mlxcel-bench-de] +2026-10-09T01:52:36+0900 sample 7: clean gpu=[3596927:mlxcel-bench-de] +2026-10-09T01:52:37+0900 sample 8: clean gpu=[3596927:mlxcel-bench-de] +2026-10-09T01:52:39+0900 sample 9: clean gpu=[3596927:mlxcel-bench-de] +2026-10-09T01:52:40+0900 sample 10: clean gpu=[3596927:mlxcel-bench-de] +2026-10-09T01:52:42+0900 sample 11: clean gpu=[3596927:mlxcel-bench-de] +2026-10-09T01:52:43+0900 sample 12: clean gpu=[3596927:mlxcel-bench-de] +2026-10-09T01:52:45+0900 sample 13: clean gpu=[3596927:mlxcel-bench-de] +2026-10-09T01:52:46+0900 sample 14: clean gpu=[3596927:mlxcel-bench-de] +2026-10-09T01:52:48+0900 sample 15: clean gpu=[3596927:mlxcel-bench-de] +2026-10-09T01:52:49+0900 sample 16: clean gpu=[3596927:mlxcel-bench-de] +2026-10-09T01:52:51+0900 sample 17: clean gpu=[3596927:mlxcel-bench-de] +2026-10-09T01:52:52+0900 monitor: 17 samples +2026-10-09T01:52:52+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp2048_r1_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp2048_r1_new.log new file mode 100644 index 000000000..c68c48163 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp2048_r1_new.log @@ -0,0 +1,23 @@ +2026-10-09T01:52:57+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T01:52:57+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T01:53:11+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T01:53:11+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +2026-10-09T01:53:12+0900 sample 1: clean gpu=[] +2026-10-09T01:53:13+0900 sample 2: clean gpu=[3621883:mlxcel-bench-de] +2026-10-09T01:53:15+0900 sample 3: clean gpu=[3621883:mlxcel-bench-de] +2026-10-09T01:53:16+0900 sample 4: clean gpu=[3621883:mlxcel-bench-de] +2026-10-09T01:53:17+0900 sample 5: clean gpu=[3621883:mlxcel-bench-de] +2026-10-09T01:53:19+0900 sample 6: clean gpu=[3621883:mlxcel-bench-de] +2026-10-09T01:53:20+0900 sample 7: clean gpu=[3621883:mlxcel-bench-de] +2026-10-09T01:53:22+0900 sample 8: clean gpu=[3621883:mlxcel-bench-de] +2026-10-09T01:53:23+0900 sample 9: clean gpu=[3621883:mlxcel-bench-de] +2026-10-09T01:53:25+0900 sample 10: clean gpu=[3621883:mlxcel-bench-de] +2026-10-09T01:53:26+0900 sample 11: clean gpu=[3621883:mlxcel-bench-de] +2026-10-09T01:53:28+0900 sample 12: clean gpu=[3621883:mlxcel-bench-de] +2026-10-09T01:53:29+0900 sample 13: clean gpu=[3621883:mlxcel-bench-de] +2026-10-09T01:53:31+0900 sample 14: clean gpu=[3621883:mlxcel-bench-de] +2026-10-09T01:53:32+0900 sample 15: clean gpu=[3621883:mlxcel-bench-de] +2026-10-09T01:53:34+0900 sample 16: clean gpu=[3621883:mlxcel-bench-de] +2026-10-09T01:53:35+0900 sample 17: clean gpu=[3621883:mlxcel-bench-de] +2026-10-09T01:53:36+0900 monitor: 17 samples +2026-10-09T01:53:36+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp2048_r2_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp2048_r2_base-null.log new file mode 100644 index 000000000..a80cc0eca --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp2048_r2_base-null.log @@ -0,0 +1,22 @@ +2026-10-09T02:12:50+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:12:50+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:13:04+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:13:04+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +2026-10-09T02:13:05+0900 sample 1: clean gpu=[] +2026-10-09T02:13:06+0900 sample 2: clean gpu=[82768:mlxcel-bench-de] +2026-10-09T02:13:08+0900 sample 3: clean gpu=[82768:mlxcel-bench-de] +2026-10-09T02:13:09+0900 sample 4: clean gpu=[82768:mlxcel-bench-de] +2026-10-09T02:13:11+0900 sample 5: clean gpu=[82768:mlxcel-bench-de] +2026-10-09T02:13:12+0900 sample 6: clean gpu=[82768:mlxcel-bench-de] +2026-10-09T02:13:14+0900 sample 7: clean gpu=[82768:mlxcel-bench-de] +2026-10-09T02:13:15+0900 sample 8: clean gpu=[82768:mlxcel-bench-de] +2026-10-09T02:13:17+0900 sample 9: clean gpu=[82768:mlxcel-bench-de] +2026-10-09T02:13:18+0900 sample 10: clean gpu=[82768:mlxcel-bench-de] +2026-10-09T02:13:20+0900 sample 11: clean gpu=[82768:mlxcel-bench-de] +2026-10-09T02:13:21+0900 sample 12: clean gpu=[82768:mlxcel-bench-de] +2026-10-09T02:13:23+0900 sample 13: clean gpu=[82768:mlxcel-bench-de] +2026-10-09T02:13:24+0900 sample 14: clean gpu=[82768:mlxcel-bench-de] +2026-10-09T02:13:26+0900 sample 15: clean gpu=[82768:mlxcel-bench-de] +2026-10-09T02:13:27+0900 sample 16: clean gpu=[82768:mlxcel-bench-de] +2026-10-09T02:13:28+0900 monitor: 16 samples +2026-10-09T02:13:28+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp2048_r2_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp2048_r2_base.log new file mode 100644 index 000000000..e4455fc97 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp2048_r2_base.log @@ -0,0 +1,23 @@ +2026-10-09T02:13:33+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:13:33+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:13:48+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:13:48+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +2026-10-09T02:13:48+0900 sample 1: clean gpu=[] +2026-10-09T02:13:50+0900 sample 2: clean gpu=[107711:mlxcel-bench-de] +2026-10-09T02:13:51+0900 sample 3: clean gpu=[107711:mlxcel-bench-de] +2026-10-09T02:13:53+0900 sample 4: clean gpu=[107711:mlxcel-bench-de] +2026-10-09T02:13:54+0900 sample 5: clean gpu=[107711:mlxcel-bench-de] +2026-10-09T02:13:56+0900 sample 6: clean gpu=[107711:mlxcel-bench-de] +2026-10-09T02:13:57+0900 sample 7: clean gpu=[107711:mlxcel-bench-de] +2026-10-09T02:13:59+0900 sample 8: clean gpu=[107711:mlxcel-bench-de] +2026-10-09T02:14:00+0900 sample 9: clean gpu=[107711:mlxcel-bench-de] +2026-10-09T02:14:02+0900 sample 10: clean gpu=[107711:mlxcel-bench-de] +2026-10-09T02:14:03+0900 sample 11: clean gpu=[107711:mlxcel-bench-de] +2026-10-09T02:14:05+0900 sample 12: clean gpu=[107711:mlxcel-bench-de] +2026-10-09T02:14:06+0900 sample 13: clean gpu=[107711:mlxcel-bench-de] +2026-10-09T02:14:08+0900 sample 14: clean gpu=[107711:mlxcel-bench-de] +2026-10-09T02:14:09+0900 sample 15: clean gpu=[107711:mlxcel-bench-de] +2026-10-09T02:14:11+0900 sample 16: clean gpu=[107711:mlxcel-bench-de] +2026-10-09T02:14:12+0900 sample 17: clean gpu=[107711:mlxcel-bench-de] +2026-10-09T02:14:13+0900 monitor: 17 samples +2026-10-09T02:14:13+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp2048_r2_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp2048_r2_new.log new file mode 100644 index 000000000..2897db43d --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp2048_r2_new.log @@ -0,0 +1,30 @@ +2026-10-09T02:11:54+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:11:54+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:12:09+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:12:09+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +2026-10-09T02:12:09+0900 sample 1: clean gpu=[] +2026-10-09T02:12:10+0900 sample 2: clean gpu=[50124:mlxcel-bench-de] +2026-10-09T02:12:12+0900 sample 3: clean gpu=[50124:mlxcel-bench-de] +2026-10-09T02:12:13+0900 sample 4: clean gpu=[50124:mlxcel-bench-de] +2026-10-09T02:12:15+0900 sample 5: clean gpu=[50124:mlxcel-bench-de] +2026-10-09T02:12:16+0900 sample 6: clean gpu=[50124:mlxcel-bench-de] +2026-10-09T02:12:18+0900 sample 7: clean gpu=[50124:mlxcel-bench-de] +2026-10-09T02:12:20+0900 sample 8: clean gpu=[50124:mlxcel-bench-de] +2026-10-09T02:12:21+0900 sample 9: clean gpu=[50124:mlxcel-bench-de] +2026-10-09T02:12:23+0900 sample 10: clean gpu=[50124:mlxcel-bench-de] +2026-10-09T02:12:24+0900 sample 11: clean gpu=[50124:mlxcel-bench-de] +2026-10-09T02:12:26+0900 sample 12: clean gpu=[50124:mlxcel-bench-de] +2026-10-09T02:12:27+0900 sample 13: clean gpu=[50124:mlxcel-bench-de] +2026-10-09T02:12:29+0900 sample 14: clean gpu=[50124:mlxcel-bench-de] +2026-10-09T02:12:30+0900 sample 15: clean gpu=[50124:mlxcel-bench-de] +2026-10-09T02:12:32+0900 sample 16: clean gpu=[50124:mlxcel-bench-de] +2026-10-09T02:12:33+0900 sample 17: clean gpu=[50124:mlxcel-bench-de] +2026-10-09T02:12:34+0900 sample 18: clean gpu=[50124:mlxcel-bench-de] +2026-10-09T02:12:36+0900 sample 19: clean gpu=[50124:mlxcel-bench-de] +2026-10-09T02:12:37+0900 sample 20: clean gpu=[50124:mlxcel-bench-de] +2026-10-09T02:12:39+0900 sample 21: clean gpu=[50124:mlxcel-bench-de] +2026-10-09T02:12:40+0900 sample 22: clean gpu=[50124:mlxcel-bench-de] +2026-10-09T02:12:42+0900 sample 23: clean gpu=[50124:mlxcel-bench-de] +2026-10-09T02:12:43+0900 sample 24: clean gpu=[50124:mlxcel-bench-de] +2026-10-09T02:12:44+0900 monitor: 24 samples +2026-10-09T02:12:44+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp2048_r3_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp2048_r3_base-null.log new file mode 100644 index 000000000..de93de17b --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp2048_r3_base-null.log @@ -0,0 +1,23 @@ +2026-10-09T02:31:48+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:31:48+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:32:03+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:32:03+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +2026-10-09T02:32:03+0900 sample 1: clean gpu=[] +2026-10-09T02:32:05+0900 sample 2: clean gpu=[700676:mlxcel-bench-de] +2026-10-09T02:32:06+0900 sample 3: clean gpu=[700676:mlxcel-bench-de] +2026-10-09T02:32:08+0900 sample 4: clean gpu=[700676:mlxcel-bench-de] +2026-10-09T02:32:09+0900 sample 5: clean gpu=[700676:mlxcel-bench-de] +2026-10-09T02:32:11+0900 sample 6: clean gpu=[700676:mlxcel-bench-de] +2026-10-09T02:32:12+0900 sample 7: clean gpu=[700676:mlxcel-bench-de] +2026-10-09T02:32:14+0900 sample 8: clean gpu=[700676:mlxcel-bench-de] +2026-10-09T02:32:15+0900 sample 9: clean gpu=[700676:mlxcel-bench-de] +2026-10-09T02:32:17+0900 sample 10: clean gpu=[700676:mlxcel-bench-de] +2026-10-09T02:32:18+0900 sample 11: clean gpu=[700676:mlxcel-bench-de] +2026-10-09T02:32:19+0900 sample 12: clean gpu=[700676:mlxcel-bench-de] +2026-10-09T02:32:21+0900 sample 13: clean gpu=[700676:mlxcel-bench-de] +2026-10-09T02:32:22+0900 sample 14: clean gpu=[700676:mlxcel-bench-de] +2026-10-09T02:32:24+0900 sample 15: clean gpu=[700676:mlxcel-bench-de] +2026-10-09T02:32:25+0900 sample 16: clean gpu=[700676:mlxcel-bench-de] +2026-10-09T02:32:27+0900 sample 17: clean gpu=[700676:mlxcel-bench-de] +2026-10-09T02:32:28+0900 monitor: 17 samples +2026-10-09T02:32:28+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp2048_r3_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp2048_r3_base.log new file mode 100644 index 000000000..fa79ced51 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp2048_r3_base.log @@ -0,0 +1,23 @@ +2026-10-09T02:32:33+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:32:33+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:32:47+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:32:47+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +2026-10-09T02:32:48+0900 sample 1: clean gpu=[] +2026-10-09T02:32:49+0900 sample 2: clean gpu=[725677:mlxcel-bench-de] +2026-10-09T02:32:51+0900 sample 3: clean gpu=[725677:mlxcel-bench-de] +2026-10-09T02:32:52+0900 sample 4: clean gpu=[725677:mlxcel-bench-de] +2026-10-09T02:32:54+0900 sample 5: clean gpu=[725677:mlxcel-bench-de] +2026-10-09T02:32:55+0900 sample 6: clean gpu=[725677:mlxcel-bench-de] +2026-10-09T02:32:57+0900 sample 7: clean gpu=[725677:mlxcel-bench-de] +2026-10-09T02:32:58+0900 sample 8: clean gpu=[725677:mlxcel-bench-de] +2026-10-09T02:33:00+0900 sample 9: clean gpu=[725677:mlxcel-bench-de] +2026-10-09T02:33:01+0900 sample 10: clean gpu=[725677:mlxcel-bench-de] +2026-10-09T02:33:03+0900 sample 11: clean gpu=[725677:mlxcel-bench-de] +2026-10-09T02:33:04+0900 sample 12: clean gpu=[725677:mlxcel-bench-de] +2026-10-09T02:33:06+0900 sample 13: clean gpu=[725677:mlxcel-bench-de] +2026-10-09T02:33:07+0900 sample 14: clean gpu=[725677:mlxcel-bench-de] +2026-10-09T02:33:09+0900 sample 15: clean gpu=[725677:mlxcel-bench-de] +2026-10-09T02:33:10+0900 sample 16: clean gpu=[725677:mlxcel-bench-de] +2026-10-09T02:33:12+0900 sample 17: clean gpu=[725677:mlxcel-bench-de] +2026-10-09T02:33:13+0900 monitor: 17 samples +2026-10-09T02:33:13+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp2048_r3_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp2048_r3_new.log new file mode 100644 index 000000000..2913a0b35 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp2048_r3_new.log @@ -0,0 +1,23 @@ +2026-10-09T02:33:18+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:33:18+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:33:32+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:33:32+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +2026-10-09T02:33:33+0900 sample 1: clean gpu=[] +2026-10-09T02:33:34+0900 sample 2: clean gpu=[750642:mlxcel-bench-de] +2026-10-09T02:33:35+0900 sample 3: clean gpu=[750642:mlxcel-bench-de] +2026-10-09T02:33:37+0900 sample 4: clean gpu=[750642:mlxcel-bench-de] +2026-10-09T02:33:38+0900 sample 5: clean gpu=[750642:mlxcel-bench-de] +2026-10-09T02:33:40+0900 sample 6: clean gpu=[750642:mlxcel-bench-de] +2026-10-09T02:33:41+0900 sample 7: clean gpu=[750642:mlxcel-bench-de] +2026-10-09T02:33:43+0900 sample 8: clean gpu=[750642:mlxcel-bench-de] +2026-10-09T02:33:44+0900 sample 9: clean gpu=[750642:mlxcel-bench-de] +2026-10-09T02:33:46+0900 sample 10: clean gpu=[750642:mlxcel-bench-de] +2026-10-09T02:33:47+0900 sample 11: clean gpu=[750642:mlxcel-bench-de] +2026-10-09T02:33:49+0900 sample 12: clean gpu=[750642:mlxcel-bench-de] +2026-10-09T02:33:50+0900 sample 13: clean gpu=[750642:mlxcel-bench-de] +2026-10-09T02:33:52+0900 sample 14: clean gpu=[750642:mlxcel-bench-de] +2026-10-09T02:33:53+0900 sample 15: clean gpu=[750642:mlxcel-bench-de] +2026-10-09T02:33:55+0900 sample 16: clean gpu=[750642:mlxcel-bench-de] +2026-10-09T02:33:56+0900 sample 17: clean gpu=[750642:mlxcel-bench-de] +2026-10-09T02:33:57+0900 monitor: 17 samples +2026-10-09T02:33:57+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp512_r1_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp512_r1_base-null.log new file mode 100644 index 000000000..c1c13baf8 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp512_r1_base-null.log @@ -0,0 +1,16 @@ +2026-10-09T01:51:37+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T01:51:37+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T01:51:52+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T01:51:52+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T01:51:52+0900 sample 1: clean gpu=[] +2026-10-09T01:51:54+0900 sample 2: clean gpu=[3577931:mlxcel-bench-de] +2026-10-09T01:51:55+0900 sample 3: clean gpu=[3577931:mlxcel-bench-de] +2026-10-09T01:51:57+0900 sample 4: clean gpu=[3577931:mlxcel-bench-de] +2026-10-09T01:51:59+0900 sample 5: clean gpu=[3577931:mlxcel-bench-de] +2026-10-09T01:52:00+0900 sample 6: clean gpu=[3577931:mlxcel-bench-de] +2026-10-09T01:52:01+0900 sample 7: clean gpu=[3577931:mlxcel-bench-de] +2026-10-09T01:52:03+0900 sample 8: clean gpu=[3577931:mlxcel-bench-de] +2026-10-09T01:52:04+0900 sample 9: clean gpu=[3577931:mlxcel-bench-de] +2026-10-09T01:52:06+0900 sample 10: clean gpu=[3577931:mlxcel-bench-de] +2026-10-09T01:52:07+0900 monitor: 10 samples +2026-10-09T01:52:07+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp512_r1_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp512_r1_base.log new file mode 100644 index 000000000..b393faedb --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp512_r1_base.log @@ -0,0 +1,27 @@ +2026-10-09T01:50:11+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T01:50:11+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T01:50:26+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T01:50:26+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T01:50:26+0900 sample 1: clean gpu=[] +2026-10-09T01:50:28+0900 sample 2: clean gpu=[3529109:mlxcel-bench-de] +2026-10-09T01:50:29+0900 sample 3: clean gpu=[3529109:mlxcel-bench-de] +2026-10-09T01:50:31+0900 sample 4: clean gpu=[3529109:mlxcel-bench-de] +2026-10-09T01:50:32+0900 sample 5: clean gpu=[3529109:mlxcel-bench-de] +2026-10-09T01:50:34+0900 sample 6: clean gpu=[3529109:mlxcel-bench-de] +2026-10-09T01:50:35+0900 sample 7: clean gpu=[3529109:mlxcel-bench-de] +2026-10-09T01:50:37+0900 sample 8: clean gpu=[3529109:mlxcel-bench-de] +2026-10-09T01:50:38+0900 sample 9: clean gpu=[3529109:mlxcel-bench-de] +2026-10-09T01:50:40+0900 sample 10: clean gpu=[3529109:mlxcel-bench-de] +2026-10-09T01:50:41+0900 sample 11: clean gpu=[3529109:mlxcel-bench-de] +2026-10-09T01:50:43+0900 sample 12: clean gpu=[3529109:mlxcel-bench-de] +2026-10-09T01:50:45+0900 sample 13: clean gpu=[3529109:mlxcel-bench-de] +2026-10-09T01:50:46+0900 sample 14: clean gpu=[3529109:mlxcel-bench-de] +2026-10-09T01:50:48+0900 sample 15: clean gpu=[3529109:mlxcel-bench-de] +2026-10-09T01:50:49+0900 sample 16: clean gpu=[3529109:mlxcel-bench-de] +2026-10-09T01:50:51+0900 sample 17: clean gpu=[3529109:mlxcel-bench-de] +2026-10-09T01:50:52+0900 sample 18: clean gpu=[3529109:mlxcel-bench-de] +2026-10-09T01:50:54+0900 sample 19: clean gpu=[3529109:mlxcel-bench-de] +2026-10-09T01:50:55+0900 sample 20: clean gpu=[3529109:mlxcel-bench-de] +2026-10-09T01:50:57+0900 sample 21: clean gpu=[3529109:mlxcel-bench-de] +2026-10-09T01:50:58+0900 monitor: 21 samples +2026-10-09T01:50:58+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp512_r1_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp512_r1_new.log new file mode 100644 index 000000000..f7d499617 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp512_r1_new.log @@ -0,0 +1,16 @@ +2026-10-09T01:51:03+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T01:51:03+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T01:51:17+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T01:51:17+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T01:51:18+0900 sample 1: clean gpu=[] +2026-10-09T01:51:19+0900 sample 2: clean gpu=[3558843:mlxcel-bench-de] +2026-10-09T01:51:21+0900 sample 3: clean gpu=[3558843:mlxcel-bench-de] +2026-10-09T01:51:22+0900 sample 4: clean gpu=[3558843:mlxcel-bench-de] +2026-10-09T01:51:24+0900 sample 5: clean gpu=[3558843:mlxcel-bench-de] +2026-10-09T01:51:25+0900 sample 6: clean gpu=[3558843:mlxcel-bench-de] +2026-10-09T01:51:27+0900 sample 7: clean gpu=[3558843:mlxcel-bench-de] +2026-10-09T01:51:28+0900 sample 8: clean gpu=[3558843:mlxcel-bench-de] +2026-10-09T01:51:30+0900 sample 9: clean gpu=[3558843:mlxcel-bench-de] +2026-10-09T01:51:31+0900 sample 10: clean gpu=[3558843:mlxcel-bench-de] +2026-10-09T01:51:32+0900 monitor: 10 samples +2026-10-09T01:51:32+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp512_r2_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp512_r2_base-null.log new file mode 100644 index 000000000..59dfefe37 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp512_r2_base-null.log @@ -0,0 +1,16 @@ +2026-10-09T02:10:44+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:10:44+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:10:59+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:10:59+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:11:00+0900 sample 1: clean gpu=[] +2026-10-09T02:11:01+0900 sample 2: clean gpu=[11664:mlxcel-bench-de] +2026-10-09T02:11:03+0900 sample 3: clean gpu=[11664:mlxcel-bench-de] +2026-10-09T02:11:04+0900 sample 4: clean gpu=[11664:mlxcel-bench-de] +2026-10-09T02:11:06+0900 sample 5: clean gpu=[11664:mlxcel-bench-de] +2026-10-09T02:11:07+0900 sample 6: clean gpu=[11664:mlxcel-bench-de] +2026-10-09T02:11:09+0900 sample 7: clean gpu=[11664:mlxcel-bench-de] +2026-10-09T02:11:10+0900 sample 8: clean gpu=[11664:mlxcel-bench-de] +2026-10-09T02:11:12+0900 sample 9: clean gpu=[11664:mlxcel-bench-de] +2026-10-09T02:11:13+0900 sample 10: clean gpu=[11664:mlxcel-bench-de] +2026-10-09T02:11:14+0900 monitor: 10 samples +2026-10-09T02:11:14+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp512_r2_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp512_r2_base.log new file mode 100644 index 000000000..895d4c74e --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp512_r2_base.log @@ -0,0 +1,16 @@ +2026-10-09T02:11:19+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:11:19+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:11:34+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:11:34+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:11:34+0900 sample 1: clean gpu=[] +2026-10-09T02:11:36+0900 sample 2: clean gpu=[30912:mlxcel-bench-de] +2026-10-09T02:11:37+0900 sample 3: clean gpu=[30912:mlxcel-bench-de] +2026-10-09T02:11:39+0900 sample 4: clean gpu=[30912:mlxcel-bench-de] +2026-10-09T02:11:40+0900 sample 5: clean gpu=[30912:mlxcel-bench-de] +2026-10-09T02:11:42+0900 sample 6: clean gpu=[30912:mlxcel-bench-de] +2026-10-09T02:11:43+0900 sample 7: clean gpu=[30912:mlxcel-bench-de] +2026-10-09T02:11:45+0900 sample 8: clean gpu=[30912:mlxcel-bench-de] +2026-10-09T02:11:46+0900 sample 9: clean gpu=[30912:mlxcel-bench-de] +2026-10-09T02:11:48+0900 sample 10: clean gpu=[30912:mlxcel-bench-de] +2026-10-09T02:11:49+0900 monitor: 10 samples +2026-10-09T02:11:49+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp512_r2_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp512_r2_new.log new file mode 100644 index 000000000..f9b69743e --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp512_r2_new.log @@ -0,0 +1,16 @@ +2026-10-09T02:10:10+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:10:10+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:10:25+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:10:25+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:10:25+0900 sample 1: clean gpu=[] +2026-10-09T02:10:27+0900 sample 2: clean gpu=[4186267:mlxcel-bench-de] +2026-10-09T02:10:28+0900 sample 3: clean gpu=[4186267:mlxcel-bench-de] +2026-10-09T02:10:29+0900 sample 4: clean gpu=[4186267:mlxcel-bench-de] +2026-10-09T02:10:31+0900 sample 5: clean gpu=[4186267:mlxcel-bench-de] +2026-10-09T02:10:32+0900 sample 6: clean gpu=[4186267:mlxcel-bench-de] +2026-10-09T02:10:34+0900 sample 7: clean gpu=[4186267:mlxcel-bench-de] +2026-10-09T02:10:35+0900 sample 8: clean gpu=[4186267:mlxcel-bench-de] +2026-10-09T02:10:37+0900 sample 9: clean gpu=[4186267:mlxcel-bench-de] +2026-10-09T02:10:38+0900 sample 10: clean gpu=[4186267:mlxcel-bench-de] +2026-10-09T02:10:39+0900 monitor: 10 samples +2026-10-09T02:10:39+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp512_r3_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp512_r3_base-null.log new file mode 100644 index 000000000..dacbb1135 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp512_r3_base-null.log @@ -0,0 +1,17 @@ +2026-10-09T02:30:03+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:30:03+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:30:18+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:30:18+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:30:19+0900 sample 1: clean gpu=[] +2026-10-09T02:30:20+0900 sample 2: clean gpu=[643275:mlxcel-bench-de] +2026-10-09T02:30:21+0900 sample 3: clean gpu=[643275:mlxcel-bench-de] +2026-10-09T02:30:23+0900 sample 4: clean gpu=[643275:mlxcel-bench-de] +2026-10-09T02:30:24+0900 sample 5: clean gpu=[643275:mlxcel-bench-de] +2026-10-09T02:30:26+0900 sample 6: clean gpu=[643275:mlxcel-bench-de] +2026-10-09T02:30:27+0900 sample 7: clean gpu=[643275:mlxcel-bench-de] +2026-10-09T02:30:29+0900 sample 8: clean gpu=[643275:mlxcel-bench-de] +2026-10-09T02:30:30+0900 sample 9: clean gpu=[643275:mlxcel-bench-de] +2026-10-09T02:30:32+0900 sample 10: clean gpu=[643275:mlxcel-bench-de] +2026-10-09T02:30:33+0900 sample 11: clean gpu=[643275:mlxcel-bench-de] +2026-10-09T02:30:34+0900 monitor: 11 samples +2026-10-09T02:30:34+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp512_r3_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp512_r3_base.log new file mode 100644 index 000000000..21b084363 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp512_r3_base.log @@ -0,0 +1,16 @@ +2026-10-09T02:30:39+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:30:39+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:30:54+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:30:54+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:30:54+0900 sample 1: clean gpu=[] +2026-10-09T02:30:56+0900 sample 2: clean gpu=[663423:mlxcel-bench-de] +2026-10-09T02:30:57+0900 sample 3: clean gpu=[663423:mlxcel-bench-de] +2026-10-09T02:30:59+0900 sample 4: clean gpu=[663423:mlxcel-bench-de] +2026-10-09T02:31:00+0900 sample 5: clean gpu=[663423:mlxcel-bench-de] +2026-10-09T02:31:02+0900 sample 6: clean gpu=[663423:mlxcel-bench-de] +2026-10-09T02:31:03+0900 sample 7: clean gpu=[663423:mlxcel-bench-de] +2026-10-09T02:31:05+0900 sample 8: clean gpu=[663423:mlxcel-bench-de] +2026-10-09T02:31:06+0900 sample 9: clean gpu=[663423:mlxcel-bench-de] +2026-10-09T02:31:08+0900 sample 10: clean gpu=[663423:mlxcel-bench-de] +2026-10-09T02:31:09+0900 monitor: 10 samples +2026-10-09T02:31:09+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp512_r3_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp512_r3_new.log new file mode 100644 index 000000000..b51392ba2 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/Qwen3-30B-A3B-4bit_pp512_r3_new.log @@ -0,0 +1,16 @@ +2026-10-09T02:31:14+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:31:14+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:31:28+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:31:28+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:31:29+0900 sample 1: clean gpu=[] +2026-10-09T02:31:30+0900 sample 2: clean gpu=[682097:mlxcel-bench-de] +2026-10-09T02:31:32+0900 sample 3: clean gpu=[682097:mlxcel-bench-de] +2026-10-09T02:31:33+0900 sample 4: clean gpu=[682097:mlxcel-bench-de] +2026-10-09T02:31:35+0900 sample 5: clean gpu=[682097:mlxcel-bench-de] +2026-10-09T02:31:36+0900 sample 6: clean gpu=[682097:mlxcel-bench-de] +2026-10-09T02:31:38+0900 sample 7: clean gpu=[682097:mlxcel-bench-de] +2026-10-09T02:31:39+0900 sample 8: clean gpu=[682097:mlxcel-bench-de] +2026-10-09T02:31:41+0900 sample 9: clean gpu=[682097:mlxcel-bench-de] +2026-10-09T02:31:42+0900 sample 10: clean gpu=[682097:mlxcel-bench-de] +2026-10-09T02:31:43+0900 monitor: 10 samples +2026-10-09T02:31:43+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp2048_r1_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp2048_r1_base-null.log new file mode 100644 index 000000000..e4bde4d97 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp2048_r1_base-null.log @@ -0,0 +1,11 @@ +2026-10-09T01:49:44+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T01:49:44+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T01:49:58+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T01:49:58+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/gemma-3-4b-it-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +2026-10-09T01:49:59+0900 sample 1: clean gpu=[] +2026-10-09T01:50:00+0900 sample 2: clean gpu=[3514781:mlxcel-bench-de] +2026-10-09T01:50:02+0900 sample 3: clean gpu=[3514781:mlxcel-bench-de] +2026-10-09T01:50:03+0900 sample 4: clean gpu=[3514781:mlxcel-bench-de] +2026-10-09T01:50:05+0900 sample 5: clean gpu=[3514781:mlxcel-bench-de] +2026-10-09T01:50:06+0900 monitor: 5 samples +2026-10-09T01:50:06+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp2048_r1_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp2048_r1_base.log new file mode 100644 index 000000000..6a91bef7a --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp2048_r1_base.log @@ -0,0 +1,11 @@ +2026-10-09T01:48:50+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T01:48:50+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T01:49:04+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T01:49:04+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/gemma-3-4b-it-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +2026-10-09T01:49:05+0900 sample 1: clean gpu=[] +2026-10-09T01:49:06+0900 sample 2: clean gpu=[3486042:mlxcel-bench-de] +2026-10-09T01:49:08+0900 sample 3: clean gpu=[3486042:mlxcel-bench-de] +2026-10-09T01:49:09+0900 sample 4: clean gpu=[3486042:mlxcel-bench-de] +2026-10-09T01:49:11+0900 sample 5: clean gpu=[3486042:mlxcel-bench-de] +2026-10-09T01:49:12+0900 monitor: 5 samples +2026-10-09T01:49:12+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp2048_r1_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp2048_r1_new.log new file mode 100644 index 000000000..adfb25015 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp2048_r1_new.log @@ -0,0 +1,11 @@ +2026-10-09T01:49:17+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T01:49:17+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T01:49:31+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T01:49:31+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/gemma-3-4b-it-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +2026-10-09T01:49:32+0900 sample 1: clean gpu=[] +2026-10-09T01:49:33+0900 sample 2: clean gpu=[3500474:mlxcel-bench-de] +2026-10-09T01:49:35+0900 sample 3: clean gpu=[3500474:mlxcel-bench-de] +2026-10-09T01:49:36+0900 sample 4: clean gpu=[3500474:mlxcel-bench-de] +2026-10-09T01:49:38+0900 sample 5: clean gpu=[3500474:mlxcel-bench-de] +2026-10-09T01:49:39+0900 monitor: 5 samples +2026-10-09T01:49:39+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp2048_r2_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp2048_r2_base-null.log new file mode 100644 index 000000000..37c2f6162 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp2048_r2_base-null.log @@ -0,0 +1,11 @@ +2026-10-09T02:09:16+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:09:16+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:09:31+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:09:31+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/gemma-3-4b-it-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +2026-10-09T02:09:31+0900 sample 1: clean gpu=[] +2026-10-09T02:09:33+0900 sample 2: clean gpu=[4158223:mlxcel-bench-de] +2026-10-09T02:09:34+0900 sample 3: clean gpu=[4158223:mlxcel-bench-de] +2026-10-09T02:09:36+0900 sample 4: clean gpu=[4158223:mlxcel-bench-de] +2026-10-09T02:09:37+0900 sample 5: clean gpu=[4158223:mlxcel-bench-de] +2026-10-09T02:09:38+0900 monitor: 5 samples +2026-10-09T02:09:38+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp2048_r2_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp2048_r2_base.log new file mode 100644 index 000000000..520ff5375 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp2048_r2_base.log @@ -0,0 +1,11 @@ +2026-10-09T02:09:43+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:09:43+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:09:58+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:09:58+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/gemma-3-4b-it-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +2026-10-09T02:09:58+0900 sample 1: clean gpu=[] +2026-10-09T02:10:00+0900 sample 2: clean gpu=[4172265:mlxcel-bench-de] +2026-10-09T02:10:01+0900 sample 3: clean gpu=[4172265:mlxcel-bench-de] +2026-10-09T02:10:02+0900 sample 4: clean gpu=[4172265:mlxcel-bench-de] +2026-10-09T02:10:04+0900 sample 5: clean gpu=[4172265:mlxcel-bench-de] +2026-10-09T02:10:05+0900 monitor: 5 samples +2026-10-09T02:10:05+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp2048_r2_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp2048_r2_new.log new file mode 100644 index 000000000..8d0535c4b --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp2048_r2_new.log @@ -0,0 +1,11 @@ +2026-10-09T02:08:49+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:08:49+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:09:04+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:09:04+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/gemma-3-4b-it-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +2026-10-09T02:09:04+0900 sample 1: clean gpu=[] +2026-10-09T02:09:05+0900 sample 2: clean gpu=[4144067:mlxcel-bench-de] +2026-10-09T02:09:07+0900 sample 3: clean gpu=[4144067:mlxcel-bench-de] +2026-10-09T02:09:08+0900 sample 4: clean gpu=[4144067:mlxcel-bench-de] +2026-10-09T02:09:10+0900 sample 5: clean gpu=[4144067:mlxcel-bench-de] +2026-10-09T02:09:11+0900 monitor: 5 samples +2026-10-09T02:09:11+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp2048_r3_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp2048_r3_base-null.log new file mode 100644 index 000000000..1aa849f12 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp2048_r3_base-null.log @@ -0,0 +1,11 @@ +2026-10-09T02:28:42+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:28:42+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:28:57+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:28:57+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/gemma-3-4b-it-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +2026-10-09T02:28:57+0900 sample 1: clean gpu=[] +2026-10-09T02:28:59+0900 sample 2: clean gpu=[600016:mlxcel-bench-de] +2026-10-09T02:29:00+0900 sample 3: clean gpu=[600016:mlxcel-bench-de] +2026-10-09T02:29:02+0900 sample 4: clean gpu=[600016:mlxcel-bench-de] +2026-10-09T02:29:03+0900 sample 5: clean gpu=[600016:mlxcel-bench-de] +2026-10-09T02:29:04+0900 monitor: 5 samples +2026-10-09T02:29:04+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp2048_r3_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp2048_r3_base.log new file mode 100644 index 000000000..6a6fbb3a2 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp2048_r3_base.log @@ -0,0 +1,11 @@ +2026-10-09T02:29:09+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:29:09+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:29:24+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:29:24+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/gemma-3-4b-it-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +2026-10-09T02:29:24+0900 sample 1: clean gpu=[] +2026-10-09T02:29:26+0900 sample 2: clean gpu=[614459:mlxcel-bench-de] +2026-10-09T02:29:27+0900 sample 3: clean gpu=[614459:mlxcel-bench-de] +2026-10-09T02:29:29+0900 sample 4: clean gpu=[614459:mlxcel-bench-de] +2026-10-09T02:29:30+0900 sample 5: clean gpu=[614459:mlxcel-bench-de] +2026-10-09T02:29:31+0900 monitor: 5 samples +2026-10-09T02:29:31+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp2048_r3_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp2048_r3_new.log new file mode 100644 index 000000000..6e64b9684 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp2048_r3_new.log @@ -0,0 +1,11 @@ +2026-10-09T02:29:36+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:29:36+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:29:51+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:29:51+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/gemma-3-4b-it-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +2026-10-09T02:29:51+0900 sample 1: clean gpu=[] +2026-10-09T02:29:53+0900 sample 2: clean gpu=[628844:mlxcel-bench-de] +2026-10-09T02:29:54+0900 sample 3: clean gpu=[628844:mlxcel-bench-de] +2026-10-09T02:29:56+0900 sample 4: clean gpu=[628844:mlxcel-bench-de] +2026-10-09T02:29:57+0900 sample 5: clean gpu=[628844:mlxcel-bench-de] +2026-10-09T02:29:58+0900 monitor: 5 samples +2026-10-09T02:29:58+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp512_r1_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp512_r1_base-null.log new file mode 100644 index 000000000..0c4b5c694 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp512_r1_base-null.log @@ -0,0 +1,10 @@ +2026-10-09T01:48:24+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T01:48:24+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T01:48:39+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T01:48:39+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/gemma-3-4b-it-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T01:48:39+0900 sample 1: clean gpu=[] +2026-10-09T01:48:41+0900 sample 2: clean gpu=[3468221:mlxcel-bench-de] +2026-10-09T01:48:42+0900 sample 3: clean gpu=[3468221:mlxcel-bench-de] +2026-10-09T01:48:44+0900 sample 4: clean gpu=[3468221:mlxcel-bench-de] +2026-10-09T01:48:45+0900 monitor: 4 samples +2026-10-09T01:48:45+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp512_r1_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp512_r1_base.log new file mode 100644 index 000000000..4c60b7edb --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp512_r1_base.log @@ -0,0 +1,11 @@ +2026-10-09T01:47:31+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T01:47:31+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T01:47:46+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T01:47:46+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/gemma-3-4b-it-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T01:47:46+0900 sample 1: clean gpu=[] +2026-10-09T01:47:48+0900 sample 2: clean gpu=[3408879:mlxcel-bench-de] +2026-10-09T01:47:49+0900 sample 3: clean gpu=[3408879:mlxcel-bench-de] +2026-10-09T01:47:51+0900 sample 4: clean gpu=[3408879:mlxcel-bench-de] +2026-10-09T01:47:52+0900 sample 5: clean gpu=[3408879:mlxcel-bench-de] +2026-10-09T01:47:53+0900 monitor: 5 samples +2026-10-09T01:47:53+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp512_r1_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp512_r1_new.log new file mode 100644 index 000000000..ee5b08fc8 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp512_r1_new.log @@ -0,0 +1,10 @@ +2026-10-09T01:47:58+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T01:47:58+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T01:48:13+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T01:48:13+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/gemma-3-4b-it-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T01:48:13+0900 sample 1: clean gpu=[] +2026-10-09T01:48:15+0900 sample 2: clean gpu=[3437165:mlxcel-bench-de] +2026-10-09T01:48:16+0900 sample 3: clean gpu=[3437165:mlxcel-bench-de] +2026-10-09T01:48:18+0900 sample 4: clean gpu=[3437165:mlxcel-bench-de] +2026-10-09T01:48:19+0900 monitor: 4 samples +2026-10-09T01:48:19+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp512_r2_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp512_r2_base-null.log new file mode 100644 index 000000000..0a064da03 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp512_r2_base-null.log @@ -0,0 +1,10 @@ +2026-10-09T02:07:58+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:07:58+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:08:12+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:08:12+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/gemma-3-4b-it-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:08:13+0900 sample 1: clean gpu=[] +2026-10-09T02:08:14+0900 sample 2: clean gpu=[4117635:mlxcel-bench-de] +2026-10-09T02:08:16+0900 sample 3: clean gpu=[4117635:mlxcel-bench-de] +2026-10-09T02:08:17+0900 sample 4: clean gpu=[4117635:mlxcel-bench-de] +2026-10-09T02:08:18+0900 monitor: 4 samples +2026-10-09T02:08:18+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp512_r2_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp512_r2_base.log new file mode 100644 index 000000000..da0dfe82c --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp512_r2_base.log @@ -0,0 +1,10 @@ +2026-10-09T02:08:23+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:08:23+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:08:38+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:08:38+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/gemma-3-4b-it-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:08:38+0900 sample 1: clean gpu=[] +2026-10-09T02:08:40+0900 sample 2: clean gpu=[4130867:mlxcel-bench-de] +2026-10-09T02:08:41+0900 sample 3: clean gpu=[4130867:mlxcel-bench-de] +2026-10-09T02:08:43+0900 sample 4: clean gpu=[4130867:mlxcel-bench-de] +2026-10-09T02:08:44+0900 monitor: 4 samples +2026-10-09T02:08:44+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp512_r2_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp512_r2_new.log new file mode 100644 index 000000000..a030fa689 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp512_r2_new.log @@ -0,0 +1,10 @@ +2026-10-09T02:07:32+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:07:32+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:07:47+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:07:47+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/gemma-3-4b-it-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:07:47+0900 sample 1: clean gpu=[] +2026-10-09T02:07:49+0900 sample 2: clean gpu=[4104326:mlxcel-bench-de] +2026-10-09T02:07:50+0900 sample 3: clean gpu=[4104326:mlxcel-bench-de] +2026-10-09T02:07:52+0900 sample 4: clean gpu=[4104326:mlxcel-bench-de] +2026-10-09T02:07:53+0900 monitor: 4 samples +2026-10-09T02:07:53+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp512_r3_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp512_r3_base-null.log new file mode 100644 index 000000000..8de1c1f88 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp512_r3_base-null.log @@ -0,0 +1,10 @@ +2026-10-09T02:27:25+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:27:25+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:27:40+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:27:40+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/gemma-3-4b-it-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:27:40+0900 sample 1: clean gpu=[] +2026-10-09T02:27:42+0900 sample 2: clean gpu=[559971:mlxcel-bench-de] +2026-10-09T02:27:43+0900 sample 3: clean gpu=[559971:mlxcel-bench-de] +2026-10-09T02:27:45+0900 sample 4: clean gpu=[559971:mlxcel-bench-de] +2026-10-09T02:27:46+0900 monitor: 4 samples +2026-10-09T02:27:46+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp512_r3_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp512_r3_base.log new file mode 100644 index 000000000..f227e46b1 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp512_r3_base.log @@ -0,0 +1,10 @@ +2026-10-09T02:27:51+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:27:51+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:28:05+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:28:05+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/gemma-3-4b-it-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:28:06+0900 sample 1: clean gpu=[] +2026-10-09T02:28:07+0900 sample 2: clean gpu=[573242:mlxcel-bench-de] +2026-10-09T02:28:09+0900 sample 3: clean gpu=[573242:mlxcel-bench-de] +2026-10-09T02:28:10+0900 sample 4: clean gpu=[573242:mlxcel-bench-de] +2026-10-09T02:28:11+0900 monitor: 4 samples +2026-10-09T02:28:11+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp512_r3_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp512_r3_new.log new file mode 100644 index 000000000..998d1089b --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gemma-3-4b-it-4bit_pp512_r3_new.log @@ -0,0 +1,10 @@ +2026-10-09T02:28:16+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:28:16+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:28:31+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:28:31+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/gemma-3-4b-it-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:28:31+0900 sample 1: clean gpu=[] +2026-10-09T02:28:33+0900 sample 2: clean gpu=[586534:mlxcel-bench-de] +2026-10-09T02:28:34+0900 sample 3: clean gpu=[586534:mlxcel-bench-de] +2026-10-09T02:28:36+0900 sample 4: clean gpu=[586534:mlxcel-bench-de] +2026-10-09T02:28:37+0900 monitor: 4 samples +2026-10-09T02:28:37+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gpt-oss-20b-MXFP4-Q4_pp512_r1_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gpt-oss-20b-MXFP4-Q4_pp512_r1_base-null.log new file mode 100644 index 000000000..0648dfb72 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gpt-oss-20b-MXFP4-Q4_pp512_r1_base-null.log @@ -0,0 +1,20 @@ +2026-10-09T01:59:12+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T01:59:12+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T01:59:26+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T01:59:26+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/gpt-oss-20b-MXFP4-Q4 -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T01:59:27+0900 sample 1: clean gpu=[] +2026-10-09T01:59:28+0900 sample 2: clean gpu=[3829107:mlxcel-bench-de] +2026-10-09T01:59:30+0900 sample 3: clean gpu=[3829107:mlxcel-bench-de] +2026-10-09T01:59:31+0900 sample 4: clean gpu=[3829107:mlxcel-bench-de] +2026-10-09T01:59:33+0900 sample 5: clean gpu=[3829107:mlxcel-bench-de] +2026-10-09T01:59:35+0900 sample 6: clean gpu=[3829107:mlxcel-bench-de] +2026-10-09T01:59:36+0900 sample 7: clean gpu=[3829107:mlxcel-bench-de] +2026-10-09T01:59:38+0900 sample 8: clean gpu=[3829107:mlxcel-bench-de] +2026-10-09T01:59:39+0900 sample 9: clean gpu=[3829107:mlxcel-bench-de] +2026-10-09T01:59:41+0900 sample 10: clean gpu=[3829107:mlxcel-bench-de] +2026-10-09T01:59:42+0900 sample 11: clean gpu=[3829107:mlxcel-bench-de] +2026-10-09T01:59:44+0900 sample 12: clean gpu=[3829107:mlxcel-bench-de] +2026-10-09T01:59:45+0900 sample 13: clean gpu=[3829107:mlxcel-bench-de] +2026-10-09T01:59:47+0900 sample 14: clean gpu=[3829107:mlxcel-bench-de] +2026-10-09T01:59:48+0900 monitor: 14 samples +2026-10-09T01:59:48+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gpt-oss-20b-MXFP4-Q4_pp512_r1_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gpt-oss-20b-MXFP4-Q4_pp512_r1_base.log new file mode 100644 index 000000000..f8fc978d1 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gpt-oss-20b-MXFP4-Q4_pp512_r1_base.log @@ -0,0 +1,30 @@ +2026-10-09T01:57:31+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T01:57:31+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T01:57:46+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T01:57:46+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/gpt-oss-20b-MXFP4-Q4 -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T01:57:46+0900 sample 1: clean gpu=[] +2026-10-09T01:57:48+0900 sample 2: clean gpu=[3771509:mlxcel-bench-de] +2026-10-09T01:57:49+0900 sample 3: clean gpu=[3771509:mlxcel-bench-de] +2026-10-09T01:57:51+0900 sample 4: clean gpu=[3771509:mlxcel-bench-de] +2026-10-09T01:57:52+0900 sample 5: clean gpu=[3771509:mlxcel-bench-de] +2026-10-09T01:57:54+0900 sample 6: clean gpu=[3771509:mlxcel-bench-de] +2026-10-09T01:57:56+0900 sample 7: clean gpu=[3771509:mlxcel-bench-de] +2026-10-09T01:57:57+0900 sample 8: clean gpu=[3771509:mlxcel-bench-de] +2026-10-09T01:57:59+0900 sample 9: clean gpu=[3771509:mlxcel-bench-de] +2026-10-09T01:58:00+0900 sample 10: clean gpu=[3771509:mlxcel-bench-de] +2026-10-09T01:58:02+0900 sample 11: clean gpu=[3771509:mlxcel-bench-de] +2026-10-09T01:58:03+0900 sample 12: clean gpu=[3771509:mlxcel-bench-de] +2026-10-09T01:58:05+0900 sample 13: clean gpu=[3771509:mlxcel-bench-de] +2026-10-09T01:58:06+0900 sample 14: clean gpu=[3771509:mlxcel-bench-de] +2026-10-09T01:58:08+0900 sample 15: clean gpu=[3771509:mlxcel-bench-de] +2026-10-09T01:58:09+0900 sample 16: clean gpu=[3771509:mlxcel-bench-de] +2026-10-09T01:58:11+0900 sample 17: clean gpu=[3771509:mlxcel-bench-de] +2026-10-09T01:58:12+0900 sample 18: clean gpu=[3771509:mlxcel-bench-de] +2026-10-09T01:58:14+0900 sample 19: clean gpu=[3771509:mlxcel-bench-de] +2026-10-09T01:58:15+0900 sample 20: clean gpu=[3771509:mlxcel-bench-de] +2026-10-09T01:58:17+0900 sample 21: clean gpu=[3771509:mlxcel-bench-de] +2026-10-09T01:58:18+0900 sample 22: clean gpu=[3771509:mlxcel-bench-de] +2026-10-09T01:58:20+0900 sample 23: clean gpu=[3771509:mlxcel-bench-de] +2026-10-09T01:58:21+0900 sample 24: clean gpu=[3771509:mlxcel-bench-de] +2026-10-09T01:58:22+0900 monitor: 24 samples +2026-10-09T01:58:22+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gpt-oss-20b-MXFP4-Q4_pp512_r1_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gpt-oss-20b-MXFP4-Q4_pp512_r1_new.log new file mode 100644 index 000000000..d2f0b65aa --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gpt-oss-20b-MXFP4-Q4_pp512_r1_new.log @@ -0,0 +1,22 @@ +2026-10-09T01:58:27+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T01:58:27+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T01:58:42+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T01:58:42+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/gpt-oss-20b-MXFP4-Q4 -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T01:58:43+0900 sample 1: clean gpu=[] +2026-10-09T01:58:44+0900 sample 2: clean gpu=[3804148:mlxcel-bench-de] +2026-10-09T01:58:46+0900 sample 3: clean gpu=[3804148:mlxcel-bench-de] +2026-10-09T01:58:47+0900 sample 4: clean gpu=[3804148:mlxcel-bench-de] +2026-10-09T01:58:49+0900 sample 5: clean gpu=[3804148:mlxcel-bench-de] +2026-10-09T01:58:50+0900 sample 6: clean gpu=[3804148:mlxcel-bench-de] +2026-10-09T01:58:52+0900 sample 7: clean gpu=[3804148:mlxcel-bench-de] +2026-10-09T01:58:53+0900 sample 8: clean gpu=[3804148:mlxcel-bench-de] +2026-10-09T01:58:55+0900 sample 9: clean gpu=[3804148:mlxcel-bench-de] +2026-10-09T01:58:56+0900 sample 10: clean gpu=[3804148:mlxcel-bench-de] +2026-10-09T01:58:58+0900 sample 11: clean gpu=[3804148:mlxcel-bench-de] +2026-10-09T01:58:59+0900 sample 12: clean gpu=[3804148:mlxcel-bench-de] +2026-10-09T01:59:01+0900 sample 13: clean gpu=[3804148:mlxcel-bench-de] +2026-10-09T01:59:03+0900 sample 14: clean gpu=[3804148:mlxcel-bench-de] +2026-10-09T01:59:04+0900 sample 15: clean gpu=[3804148:mlxcel-bench-de] +2026-10-09T01:59:06+0900 sample 16: clean gpu=[3804148:mlxcel-bench-de] +2026-10-09T01:59:07+0900 monitor: 16 samples +2026-10-09T01:59:07+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gpt-oss-20b-MXFP4-Q4_pp512_r2_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gpt-oss-20b-MXFP4-Q4_pp512_r2_base-null.log new file mode 100644 index 000000000..2d5b23539 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gpt-oss-20b-MXFP4-Q4_pp512_r2_base-null.log @@ -0,0 +1,22 @@ +2026-10-09T02:18:19+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:18:19+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:18:33+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:18:33+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/gpt-oss-20b-MXFP4-Q4 -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:18:34+0900 sample 1: clean gpu=[] +2026-10-09T02:18:35+0900 sample 2: clean gpu=[262691:mlxcel-bench-de] +2026-10-09T02:18:37+0900 sample 3: clean gpu=[262691:mlxcel-bench-de] +2026-10-09T02:18:38+0900 sample 4: clean gpu=[262691:mlxcel-bench-de] +2026-10-09T02:18:40+0900 sample 5: clean gpu=[262691:mlxcel-bench-de] +2026-10-09T02:18:41+0900 sample 6: clean gpu=[262691:mlxcel-bench-de] +2026-10-09T02:18:43+0900 sample 7: clean gpu=[262691:mlxcel-bench-de] +2026-10-09T02:18:44+0900 sample 8: clean gpu=[262691:mlxcel-bench-de] +2026-10-09T02:18:46+0900 sample 9: clean gpu=[262691:mlxcel-bench-de] +2026-10-09T02:18:47+0900 sample 10: clean gpu=[262691:mlxcel-bench-de] +2026-10-09T02:18:49+0900 sample 11: clean gpu=[262691:mlxcel-bench-de] +2026-10-09T02:18:50+0900 sample 12: clean gpu=[262691:mlxcel-bench-de] +2026-10-09T02:18:52+0900 sample 13: clean gpu=[262691:mlxcel-bench-de] +2026-10-09T02:18:54+0900 sample 14: clean gpu=[262691:mlxcel-bench-de] +2026-10-09T02:18:55+0900 sample 15: clean gpu=[262691:mlxcel-bench-de] +2026-10-09T02:18:57+0900 sample 16: clean gpu=[262691:mlxcel-bench-de] +2026-10-09T02:18:58+0900 monitor: 16 samples +2026-10-09T02:18:58+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gpt-oss-20b-MXFP4-Q4_pp512_r2_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gpt-oss-20b-MXFP4-Q4_pp512_r2_base.log new file mode 100644 index 000000000..f2e7bb2d9 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gpt-oss-20b-MXFP4-Q4_pp512_r2_base.log @@ -0,0 +1,21 @@ +2026-10-09T02:19:03+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:19:03+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:19:17+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:19:17+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/gpt-oss-20b-MXFP4-Q4 -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:19:18+0900 sample 1: clean gpu=[] +2026-10-09T02:19:19+0900 sample 2: clean gpu=[286819:mlxcel-bench-de] +2026-10-09T02:19:21+0900 sample 3: clean gpu=[286819:mlxcel-bench-de] +2026-10-09T02:19:22+0900 sample 4: clean gpu=[286819:mlxcel-bench-de] +2026-10-09T02:19:24+0900 sample 5: clean gpu=[286819:mlxcel-bench-de] +2026-10-09T02:19:25+0900 sample 6: clean gpu=[286819:mlxcel-bench-de] +2026-10-09T02:19:27+0900 sample 7: clean gpu=[286819:mlxcel-bench-de] +2026-10-09T02:19:28+0900 sample 8: clean gpu=[286819:mlxcel-bench-de] +2026-10-09T02:19:30+0900 sample 9: clean gpu=[286819:mlxcel-bench-de] +2026-10-09T02:19:31+0900 sample 10: clean gpu=[286819:mlxcel-bench-de] +2026-10-09T02:19:33+0900 sample 11: clean gpu=[286819:mlxcel-bench-de] +2026-10-09T02:19:34+0900 sample 12: clean gpu=[286819:mlxcel-bench-de] +2026-10-09T02:19:36+0900 sample 13: clean gpu=[286819:mlxcel-bench-de] +2026-10-09T02:19:37+0900 sample 14: clean gpu=[286819:mlxcel-bench-de] +2026-10-09T02:19:39+0900 sample 15: clean gpu=[286819:mlxcel-bench-de] +2026-10-09T02:19:40+0900 monitor: 15 samples +2026-10-09T02:19:40+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gpt-oss-20b-MXFP4-Q4_pp512_r2_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gpt-oss-20b-MXFP4-Q4_pp512_r2_new.log new file mode 100644 index 000000000..31a731b87 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gpt-oss-20b-MXFP4-Q4_pp512_r2_new.log @@ -0,0 +1,31 @@ +2026-10-09T02:17:21+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:17:21+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:17:36+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:17:36+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/gpt-oss-20b-MXFP4-Q4 -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:17:36+0900 sample 1: clean gpu=[] +2026-10-09T02:17:38+0900 sample 2: clean gpu=[230262:mlxcel-bench-de] +2026-10-09T02:17:39+0900 sample 3: clean gpu=[230262:mlxcel-bench-de] +2026-10-09T02:17:41+0900 sample 4: clean gpu=[230262:mlxcel-bench-de] +2026-10-09T02:17:42+0900 sample 5: clean gpu=[230262:mlxcel-bench-de] +2026-10-09T02:17:44+0900 sample 6: clean gpu=[230262:mlxcel-bench-de] +2026-10-09T02:17:45+0900 sample 7: clean gpu=[230262:mlxcel-bench-de] +2026-10-09T02:17:47+0900 sample 8: clean gpu=[230262:mlxcel-bench-de] +2026-10-09T02:17:48+0900 sample 9: clean gpu=[230262:mlxcel-bench-de] +2026-10-09T02:17:50+0900 sample 10: clean gpu=[230262:mlxcel-bench-de] +2026-10-09T02:17:51+0900 sample 11: clean gpu=[230262:mlxcel-bench-de] +2026-10-09T02:17:53+0900 sample 12: clean gpu=[230262:mlxcel-bench-de] +2026-10-09T02:17:54+0900 sample 13: clean gpu=[230262:mlxcel-bench-de] +2026-10-09T02:17:56+0900 sample 14: clean gpu=[230262:mlxcel-bench-de] +2026-10-09T02:17:57+0900 sample 15: clean gpu=[230262:mlxcel-bench-de] +2026-10-09T02:17:59+0900 sample 16: clean gpu=[230262:mlxcel-bench-de] +2026-10-09T02:18:00+0900 sample 17: clean gpu=[230262:mlxcel-bench-de] +2026-10-09T02:18:02+0900 sample 18: clean gpu=[230262:mlxcel-bench-de] +2026-10-09T02:18:04+0900 sample 19: clean gpu=[230262:mlxcel-bench-de] +2026-10-09T02:18:05+0900 sample 20: clean gpu=[230262:mlxcel-bench-de] +2026-10-09T02:18:07+0900 sample 21: clean gpu=[230262:mlxcel-bench-de] +2026-10-09T02:18:08+0900 sample 22: clean gpu=[230262:mlxcel-bench-de] +2026-10-09T02:18:10+0900 sample 23: clean gpu=[230262:mlxcel-bench-de] +2026-10-09T02:18:11+0900 sample 24: clean gpu=[230262:mlxcel-bench-de] +2026-10-09T02:18:13+0900 sample 25: clean gpu=[230262:mlxcel-bench-de] +2026-10-09T02:18:14+0900 monitor: 25 samples +2026-10-09T02:18:14+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gpt-oss-20b-MXFP4-Q4_pp512_r3_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gpt-oss-20b-MXFP4-Q4_pp512_r3_base-null.log new file mode 100644 index 000000000..1270b6386 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gpt-oss-20b-MXFP4-Q4_pp512_r3_base-null.log @@ -0,0 +1,31 @@ +2026-10-09T02:37:06+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:37:06+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:37:21+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:37:21+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/gpt-oss-20b-MXFP4-Q4 -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:37:21+0900 sample 1: clean gpu=[] +2026-10-09T02:37:23+0900 sample 2: clean gpu=[872333:mlxcel-bench-de] +2026-10-09T02:37:24+0900 sample 3: clean gpu=[872333:mlxcel-bench-de] +2026-10-09T02:37:26+0900 sample 4: clean gpu=[872333:mlxcel-bench-de] +2026-10-09T02:37:27+0900 sample 5: clean gpu=[872333:mlxcel-bench-de] +2026-10-09T02:37:29+0900 sample 6: clean gpu=[872333:mlxcel-bench-de] +2026-10-09T02:37:31+0900 sample 7: clean gpu=[872333:mlxcel-bench-de] +2026-10-09T02:37:32+0900 sample 8: clean gpu=[872333:mlxcel-bench-de] +2026-10-09T02:37:34+0900 sample 9: clean gpu=[872333:mlxcel-bench-de] +2026-10-09T02:37:35+0900 sample 10: clean gpu=[872333:mlxcel-bench-de] +2026-10-09T02:37:37+0900 sample 11: clean gpu=[872333:mlxcel-bench-de] +2026-10-09T02:37:38+0900 sample 12: clean gpu=[872333:mlxcel-bench-de] +2026-10-09T02:37:40+0900 sample 13: clean gpu=[872333:mlxcel-bench-de] +2026-10-09T02:37:41+0900 sample 14: clean gpu=[872333:mlxcel-bench-de] +2026-10-09T02:37:43+0900 sample 15: clean gpu=[872333:mlxcel-bench-de] +2026-10-09T02:37:44+0900 sample 16: clean gpu=[872333:mlxcel-bench-de] +2026-10-09T02:37:46+0900 sample 17: clean gpu=[872333:mlxcel-bench-de] +2026-10-09T02:37:47+0900 sample 18: clean gpu=[872333:mlxcel-bench-de] +2026-10-09T02:37:49+0900 sample 19: clean gpu=[872333:mlxcel-bench-de] +2026-10-09T02:37:50+0900 sample 20: clean gpu=[872333:mlxcel-bench-de] +2026-10-09T02:37:51+0900 sample 21: clean gpu=[872333:mlxcel-bench-de] +2026-10-09T02:37:53+0900 sample 22: clean gpu=[872333:mlxcel-bench-de] +2026-10-09T02:37:54+0900 sample 23: clean gpu=[872333:mlxcel-bench-de] +2026-10-09T02:37:56+0900 sample 24: clean gpu=[872333:mlxcel-bench-de] +2026-10-09T02:37:57+0900 sample 25: clean gpu=[872333:mlxcel-bench-de] +2026-10-09T02:37:58+0900 monitor: 25 samples +2026-10-09T02:37:58+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gpt-oss-20b-MXFP4-Q4_pp512_r3_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gpt-oss-20b-MXFP4-Q4_pp512_r3_base.log new file mode 100644 index 000000000..28d486c30 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gpt-oss-20b-MXFP4-Q4_pp512_r3_base.log @@ -0,0 +1,23 @@ +2026-10-09T02:38:03+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:38:03+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:38:18+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:38:18+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/gpt-oss-20b-MXFP4-Q4 -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:38:18+0900 sample 1: clean gpu=[] +2026-10-09T02:38:20+0900 sample 2: clean gpu=[904504:mlxcel-bench-de] +2026-10-09T02:38:21+0900 sample 3: clean gpu=[904504:mlxcel-bench-de] +2026-10-09T02:38:23+0900 sample 4: clean gpu=[904504:mlxcel-bench-de] +2026-10-09T02:38:24+0900 sample 5: clean gpu=[904504:mlxcel-bench-de] +2026-10-09T02:38:26+0900 sample 6: clean gpu=[904504:mlxcel-bench-de] +2026-10-09T02:38:27+0900 sample 7: clean gpu=[904504:mlxcel-bench-de] +2026-10-09T02:38:29+0900 sample 8: clean gpu=[904504:mlxcel-bench-de] +2026-10-09T02:38:30+0900 sample 9: clean gpu=[904504:mlxcel-bench-de] +2026-10-09T02:38:32+0900 sample 10: clean gpu=[904504:mlxcel-bench-de] +2026-10-09T02:38:33+0900 sample 11: clean gpu=[904504:mlxcel-bench-de] +2026-10-09T02:38:35+0900 sample 12: clean gpu=[904504:mlxcel-bench-de] +2026-10-09T02:38:36+0900 sample 13: clean gpu=[904504:mlxcel-bench-de] +2026-10-09T02:38:38+0900 sample 14: clean gpu=[904504:mlxcel-bench-de] +2026-10-09T02:38:39+0900 sample 15: clean gpu=[904504:mlxcel-bench-de] +2026-10-09T02:38:41+0900 sample 16: clean gpu=[904504:mlxcel-bench-de] +2026-10-09T02:38:42+0900 sample 17: clean gpu=[904504:mlxcel-bench-de] +2026-10-09T02:38:43+0900 monitor: 17 samples +2026-10-09T02:38:43+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gpt-oss-20b-MXFP4-Q4_pp512_r3_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gpt-oss-20b-MXFP4-Q4_pp512_r3_new.log new file mode 100644 index 000000000..a7718e62e --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/gpt-oss-20b-MXFP4-Q4_pp512_r3_new.log @@ -0,0 +1,21 @@ +2026-10-09T02:38:48+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:38:48+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:39:03+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:39:03+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/gpt-oss-20b-MXFP4-Q4 -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:39:03+0900 sample 1: clean gpu=[] +2026-10-09T02:39:05+0900 sample 2: clean gpu=[930024:mlxcel-bench-de] +2026-10-09T02:39:06+0900 sample 3: clean gpu=[930024:mlxcel-bench-de] +2026-10-09T02:39:08+0900 sample 4: clean gpu=[930024:mlxcel-bench-de] +2026-10-09T02:39:09+0900 sample 5: clean gpu=[930024:mlxcel-bench-de] +2026-10-09T02:39:11+0900 sample 6: clean gpu=[930024:mlxcel-bench-de] +2026-10-09T02:39:13+0900 sample 7: clean gpu=[930024:mlxcel-bench-de] +2026-10-09T02:39:14+0900 sample 8: clean gpu=[930024:mlxcel-bench-de] +2026-10-09T02:39:16+0900 sample 9: clean gpu=[930024:mlxcel-bench-de] +2026-10-09T02:39:17+0900 sample 10: clean gpu=[930024:mlxcel-bench-de] +2026-10-09T02:39:19+0900 sample 11: clean gpu=[930024:mlxcel-bench-de] +2026-10-09T02:39:20+0900 sample 12: clean gpu=[930024:mlxcel-bench-de] +2026-10-09T02:39:22+0900 sample 13: clean gpu=[930024:mlxcel-bench-de] +2026-10-09T02:39:23+0900 sample 14: clean gpu=[930024:mlxcel-bench-de] +2026-10-09T02:39:24+0900 sample 15: clean gpu=[930024:mlxcel-bench-de] +2026-10-09T02:39:25+0900 monitor: 15 samples +2026-10-09T02:39:25+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/granite-4.0-h-tiny-4bit_pp512_r1_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/granite-4.0-h-tiny-4bit_pp512_r1_base-null.log new file mode 100644 index 000000000..fe68df73c --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/granite-4.0-h-tiny-4bit_pp512_r1_base-null.log @@ -0,0 +1,10 @@ +2026-10-09T01:55:19+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T01:55:19+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T01:55:34+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T01:55:34+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/granite-4.0-h-tiny-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T01:55:34+0900 sample 1: clean gpu=[] +2026-10-09T01:55:36+0900 sample 2: clean gpu=[3699816:mlxcel-bench-de] +2026-10-09T01:55:38+0900 sample 3: clean gpu=[3699816:mlxcel-bench-de] +2026-10-09T01:55:39+0900 sample 4: clean gpu=[3699816:mlxcel-bench-de] +2026-10-09T01:55:40+0900 monitor: 4 samples +2026-10-09T01:55:40+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/granite-4.0-h-tiny-4bit_pp512_r1_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/granite-4.0-h-tiny-4bit_pp512_r1_base.log new file mode 100644 index 000000000..13fb69f38 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/granite-4.0-h-tiny-4bit_pp512_r1_base.log @@ -0,0 +1,12 @@ +2026-10-09T01:54:26+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T01:54:26+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T01:54:40+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T01:54:40+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/granite-4.0-h-tiny-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T01:54:41+0900 sample 1: clean gpu=[] +2026-10-09T01:54:42+0900 sample 2: clean gpu=[3671563:mlxcel-bench-de] +2026-10-09T01:54:44+0900 sample 3: clean gpu=[3671563:mlxcel-bench-de] +2026-10-09T01:54:45+0900 sample 4: clean gpu=[3671563:mlxcel-bench-de] +2026-10-09T01:54:46+0900 sample 5: clean gpu=[3671563:mlxcel-bench-de] +2026-10-09T01:54:48+0900 sample 6: clean gpu=[3671563:mlxcel-bench-de] +2026-10-09T01:54:49+0900 monitor: 6 samples +2026-10-09T01:54:49+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/granite-4.0-h-tiny-4bit_pp512_r1_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/granite-4.0-h-tiny-4bit_pp512_r1_new.log new file mode 100644 index 000000000..14f6606ca --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/granite-4.0-h-tiny-4bit_pp512_r1_new.log @@ -0,0 +1,10 @@ +2026-10-09T01:54:54+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T01:54:54+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T01:55:08+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T01:55:08+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/granite-4.0-h-tiny-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T01:55:09+0900 sample 1: clean gpu=[] +2026-10-09T01:55:10+0900 sample 2: clean gpu=[3686559:mlxcel-bench-de] +2026-10-09T01:55:12+0900 sample 3: clean gpu=[3686559:mlxcel-bench-de] +2026-10-09T01:55:13+0900 sample 4: clean gpu=[3686559:mlxcel-bench-de] +2026-10-09T01:55:14+0900 monitor: 4 samples +2026-10-09T01:55:14+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/granite-4.0-h-tiny-4bit_pp512_r2_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/granite-4.0-h-tiny-4bit_pp512_r2_base-null.log new file mode 100644 index 000000000..b256f3565 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/granite-4.0-h-tiny-4bit_pp512_r2_base-null.log @@ -0,0 +1,10 @@ +2026-10-09T02:14:45+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:14:45+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:15:00+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:15:00+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/granite-4.0-h-tiny-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:15:00+0900 sample 1: clean gpu=[] +2026-10-09T02:15:02+0900 sample 2: clean gpu=[147991:mlxcel-bench-de] +2026-10-09T02:15:03+0900 sample 3: clean gpu=[147991:mlxcel-bench-de] +2026-10-09T02:15:05+0900 sample 4: clean gpu=[147991:mlxcel-bench-de] +2026-10-09T02:15:06+0900 monitor: 4 samples +2026-10-09T02:15:06+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/granite-4.0-h-tiny-4bit_pp512_r2_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/granite-4.0-h-tiny-4bit_pp512_r2_base.log new file mode 100644 index 000000000..bda10692f --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/granite-4.0-h-tiny-4bit_pp512_r2_base.log @@ -0,0 +1,10 @@ +2026-10-09T02:15:11+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:15:11+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:15:25+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:15:25+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/granite-4.0-h-tiny-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:15:26+0900 sample 1: clean gpu=[] +2026-10-09T02:15:27+0900 sample 2: clean gpu=[161122:mlxcel-bench-de] +2026-10-09T02:15:29+0900 sample 3: clean gpu=[161122:mlxcel-bench-de] +2026-10-09T02:15:30+0900 sample 4: clean gpu=[161122:mlxcel-bench-de] +2026-10-09T02:15:31+0900 monitor: 4 samples +2026-10-09T02:15:31+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/granite-4.0-h-tiny-4bit_pp512_r2_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/granite-4.0-h-tiny-4bit_pp512_r2_new.log new file mode 100644 index 000000000..4b5b60c21 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/granite-4.0-h-tiny-4bit_pp512_r2_new.log @@ -0,0 +1,11 @@ +2026-10-09T02:14:18+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:14:18+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:14:33+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:14:33+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/granite-4.0-h-tiny-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:14:33+0900 sample 1: clean gpu=[] +2026-10-09T02:14:35+0900 sample 2: clean gpu=[133603:mlxcel-bench-de] +2026-10-09T02:14:36+0900 sample 3: clean gpu=[133603:mlxcel-bench-de] +2026-10-09T02:14:38+0900 sample 4: clean gpu=[133603:mlxcel-bench-de] +2026-10-09T02:14:39+0900 sample 5: clean gpu=[133603:mlxcel-bench-de] +2026-10-09T02:14:40+0900 monitor: 5 samples +2026-10-09T02:14:40+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/granite-4.0-h-tiny-4bit_pp512_r3_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/granite-4.0-h-tiny-4bit_pp512_r3_base-null.log new file mode 100644 index 000000000..583bb9b7b --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/granite-4.0-h-tiny-4bit_pp512_r3_base-null.log @@ -0,0 +1,12 @@ +2026-10-09T02:34:02+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:34:02+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:34:17+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:34:17+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/granite-4.0-h-tiny-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:34:17+0900 sample 1: clean gpu=[] +2026-10-09T02:34:19+0900 sample 2: clean gpu=[775576:mlxcel-bench-de] +2026-10-09T02:34:20+0900 sample 3: clean gpu=[775576:mlxcel-bench-de] +2026-10-09T02:34:22+0900 sample 4: clean gpu=[775576:mlxcel-bench-de] +2026-10-09T02:34:23+0900 sample 5: clean gpu=[775576:mlxcel-bench-de] +2026-10-09T02:34:24+0900 sample 6: clean gpu=[775576:mlxcel-bench-de] +2026-10-09T02:34:25+0900 monitor: 6 samples +2026-10-09T02:34:25+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/granite-4.0-h-tiny-4bit_pp512_r3_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/granite-4.0-h-tiny-4bit_pp512_r3_base.log new file mode 100644 index 000000000..0785bd5a8 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/granite-4.0-h-tiny-4bit_pp512_r3_base.log @@ -0,0 +1,10 @@ +2026-10-09T02:34:30+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:34:30+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:34:45+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:34:45+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/granite-4.0-h-tiny-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:34:45+0900 sample 1: clean gpu=[] +2026-10-09T02:34:47+0900 sample 2: clean gpu=[790307:mlxcel-bench-de] +2026-10-09T02:34:48+0900 sample 3: clean gpu=[790307:mlxcel-bench-de] +2026-10-09T02:34:50+0900 sample 4: clean gpu=[790307:mlxcel-bench-de] +2026-10-09T02:34:51+0900 monitor: 4 samples +2026-10-09T02:34:51+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/granite-4.0-h-tiny-4bit_pp512_r3_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/granite-4.0-h-tiny-4bit_pp512_r3_new.log new file mode 100644 index 000000000..17728d15a --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/guard/logs/rounds/guard/granite-4.0-h-tiny-4bit_pp512_r3_new.log @@ -0,0 +1,10 @@ +2026-10-09T02:34:56+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +2026-10-09T02:34:56+0900 waiting for 10s of idle GPU and no compiler +2026-10-09T02:35:10+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +2026-10-09T02:35:10+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/granite-4.0-h-tiny-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +2026-10-09T02:35:11+0900 sample 1: clean gpu=[] +2026-10-09T02:35:12+0900 sample 2: clean gpu=[803170:mlxcel-bench-de] +2026-10-09T02:35:14+0900 sample 3: clean gpu=[803170:mlxcel-bench-de] +2026-10-09T02:35:15+0900 sample 4: clean gpu=[803170:mlxcel-bench-de] +2026-10-09T02:35:16+0900 monitor: 4 samples +2026-10-09T02:35:16+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw-logs.tar.gz b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw-logs.tar.gz deleted file mode 100644 index fc22a43bb..000000000 Binary files a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw-logs.tar.gz and /dev/null differ diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r1_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r1_base-null.log new file mode 100644 index 000000000..7bc5c487e --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r1_base-null.log @@ -0,0 +1,22 @@ +rocm_gpu_guard: 2026-10-09T01:47:04+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T01:47:04+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T01:47:18+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T01:47:18+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Meta-Llama-3.1-8B-Instruct-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.171 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 4.75 GB, cache 0.81 GB, active+cache 5.56 GB, phase peak 5.10 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[Memory] after measured pass: active 4.75 GB, cache 0.42 GB, active+cache 5.18 GB, phase peak 6.17 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 510.48 ms (1002.98 tok/s) + Decode: 3399.29 ms (37.65 tok/s) + MLX peak memory: 6.17 GB +rocm_gpu_guard: 2026-10-09T01:47:26+0900 monitor: 5 samples +rocm_gpu_guard: 2026-10-09T01:47:26+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r1_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r1_base.log new file mode 100644 index 000000000..701febea2 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r1_base.log @@ -0,0 +1,22 @@ +rocm_gpu_guard: 2026-10-09T01:46:06+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T01:46:06+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T01:46:21+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T01:46:21+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Meta-Llama-3.1-8B-Instruct-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.207 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 4.75 GB, cache 0.68 GB, active+cache 5.43 GB, phase peak 5.10 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[Memory] after measured pass: active 4.75 GB, cache 0.38 GB, active+cache 5.13 GB, phase peak 6.15 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 506.72 ms (1010.42 tok/s) + Decode: 3397.42 ms (37.68 tok/s) + MLX peak memory: 6.15 GB +rocm_gpu_guard: 2026-10-09T01:46:31+0900 monitor: 7 samples +rocm_gpu_guard: 2026-10-09T01:46:31+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r1_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r1_new.log new file mode 100644 index 000000000..8d333433a --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r1_new.log @@ -0,0 +1,23 @@ +rocm_gpu_guard: 2026-10-09T01:46:36+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T01:46:36+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T01:46:51+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T01:46:51+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/Meta-Llama-3.1-8B-Instruct-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.200 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 4.75 GB, cache 0.38 GB, active+cache 5.13 GB, phase peak 5.10 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[Memory] after measured pass: active 4.75 GB, cache 0.38 GB, active+cache 5.13 GB, phase peak 6.15 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 506.60 ms (1010.65 tok/s) + Decode: 3356.95 ms (38.13 tok/s) + Decode path: engine + MLX peak memory: 6.15 GB +rocm_gpu_guard: 2026-10-09T01:46:59+0900 monitor: 5 samples +rocm_gpu_guard: 2026-10-09T01:46:59+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r2_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r2_base-null.log new file mode 100644 index 000000000..9fc0d9f1a --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r2_base-null.log @@ -0,0 +1,22 @@ +rocm_gpu_guard: 2026-10-09T02:06:37+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:06:37+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:06:52+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:06:52+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Meta-Llama-3.1-8B-Instruct-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.186 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 4.75 GB, cache 0.69 GB, active+cache 5.44 GB, phase peak 5.10 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[Memory] after measured pass: active 4.75 GB, cache 0.43 GB, active+cache 5.18 GB, phase peak 6.15 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 513.71 ms (996.67 tok/s) + Decode: 3382.28 ms (37.84 tok/s) + MLX peak memory: 6.15 GB +rocm_gpu_guard: 2026-10-09T02:07:00+0900 monitor: 5 samples +rocm_gpu_guard: 2026-10-09T02:07:00+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r2_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r2_base.log new file mode 100644 index 000000000..11e0ad7bf --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r2_base.log @@ -0,0 +1,22 @@ +rocm_gpu_guard: 2026-10-09T02:07:05+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:07:05+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:07:20+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:07:20+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Meta-Llama-3.1-8B-Instruct-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.189 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 4.75 GB, cache 0.81 GB, active+cache 5.56 GB, phase peak 5.10 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[Memory] after measured pass: active 4.75 GB, cache 0.43 GB, active+cache 5.18 GB, phase peak 6.15 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 507.96 ms (1007.95 tok/s) + Decode: 3388.98 ms (37.77 tok/s) + MLX peak memory: 6.15 GB +rocm_gpu_guard: 2026-10-09T02:07:27+0900 monitor: 5 samples +rocm_gpu_guard: 2026-10-09T02:07:27+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r2_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r2_new.log new file mode 100644 index 000000000..cb54b69d5 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r2_new.log @@ -0,0 +1,23 @@ +rocm_gpu_guard: 2026-10-09T02:06:07+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:06:07+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:06:22+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:06:22+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/Meta-Llama-3.1-8B-Instruct-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.190 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 4.75 GB, cache 0.42 GB, active+cache 5.18 GB, phase peak 5.11 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[Memory] after measured pass: active 4.75 GB, cache 0.38 GB, active+cache 5.13 GB, phase peak 6.15 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 932.40 ms (549.12 tok/s) + Decode: 3416.33 ms (37.47 tok/s) + Decode path: engine + MLX peak memory: 6.15 GB +rocm_gpu_guard: 2026-10-09T02:06:32+0900 monitor: 7 samples +rocm_gpu_guard: 2026-10-09T02:06:32+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r3_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r3_base-null.log new file mode 100644 index 000000000..c0c263329 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r3_base-null.log @@ -0,0 +1,22 @@ +rocm_gpu_guard: 2026-10-09T02:26:00+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:26:00+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:26:15+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:26:15+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Meta-Llama-3.1-8B-Instruct-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.193 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 4.75 GB, cache 0.68 GB, active+cache 5.43 GB, phase peak 5.10 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[Memory] after measured pass: active 4.75 GB, cache 0.43 GB, active+cache 5.18 GB, phase peak 6.15 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 516.54 ms (991.20 tok/s) + Decode: 3403.14 ms (37.61 tok/s) + MLX peak memory: 6.15 GB +rocm_gpu_guard: 2026-10-09T02:26:25+0900 monitor: 7 samples +rocm_gpu_guard: 2026-10-09T02:26:25+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r3_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r3_base.log new file mode 100644 index 000000000..aff3ad82c --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r3_base.log @@ -0,0 +1,22 @@ +rocm_gpu_guard: 2026-10-09T02:26:30+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:26:30+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:26:45+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:26:45+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Meta-Llama-3.1-8B-Instruct-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.190 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 4.75 GB, cache 0.81 GB, active+cache 5.56 GB, phase peak 5.10 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[Memory] after measured pass: active 4.75 GB, cache 0.43 GB, active+cache 5.18 GB, phase peak 6.24 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 509.34 ms (1005.22 tok/s) + Decode: 3397.71 ms (37.67 tok/s) + MLX peak memory: 6.24 GB +rocm_gpu_guard: 2026-10-09T02:26:53+0900 monitor: 5 samples +rocm_gpu_guard: 2026-10-09T02:26:53+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r3_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r3_new.log new file mode 100644 index 000000000..16455f3bc --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Meta-Llama-3.1-8B-Instruct-4bit_pp512_r3_new.log @@ -0,0 +1,23 @@ +rocm_gpu_guard: 2026-10-09T02:26:58+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:26:58+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:27:12+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:27:12+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/Meta-Llama-3.1-8B-Instruct-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.191 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 4.75 GB, cache 0.38 GB, active+cache 5.13 GB, phase peak 5.10 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[Memory] after measured pass: active 4.75 GB, cache 0.38 GB, active+cache 5.13 GB, phase peak 6.25 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 513.78 ms (996.53 tok/s) + Decode: 3398.23 ms (37.67 tok/s) + Decode path: engine + MLX peak memory: 6.25 GB +rocm_gpu_guard: 2026-10-09T02:27:20+0900 monitor: 5 samples +rocm_gpu_guard: 2026-10-09T02:27:20+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r1_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r1_base-null.log new file mode 100644 index 000000000..1efd8e5a7 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r1_base-null.log @@ -0,0 +1,22 @@ +rocm_gpu_guard: 2026-10-09T02:02:31+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:02:31+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:02:46+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:02:46+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Mixtral-8x7B-Instruct-v0.1-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.055 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(32768), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 26.40 GB, cache 2.26 GB, active+cache 28.66 GB, phase peak 27.52 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(32768), reserved 128 for generation) +[Memory] after measured pass: active 26.40 GB, cache 0.37 GB, active+cache 26.76 GB, phase peak 27.47 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 19862.45 ms (25.78 tok/s) + Decode: 12226.42 ms (10.47 tok/s) + MLX peak memory: 27.52 GB +rocm_gpu_guard: 2026-10-09T02:03:44+0900 monitor: 38 samples +rocm_gpu_guard: 2026-10-09T02:03:44+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r1_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r1_base.log new file mode 100644 index 000000000..53e1d3596 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r1_base.log @@ -0,0 +1,22 @@ +rocm_gpu_guard: 2026-10-09T01:59:53+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T01:59:53+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:00:07+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:00:07+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Mixtral-8x7B-Instruct-v0.1-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.057 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(32768), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 26.40 GB, cache 2.20 GB, active+cache 28.60 GB, phase peak 27.52 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(32768), reserved 128 for generation) +[Memory] after measured pass: active 26.40 GB, cache 0.36 GB, active+cache 26.76 GB, phase peak 27.47 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 19714.36 ms (25.97 tok/s) + Decode: 13213.77 ms (9.69 tok/s) + MLX peak memory: 27.52 GB +rocm_gpu_guard: 2026-10-09T02:01:07+0900 monitor: 39 samples +rocm_gpu_guard: 2026-10-09T02:01:07+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r1_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r1_new.log new file mode 100644 index 000000000..d6ee35222 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r1_new.log @@ -0,0 +1,23 @@ +rocm_gpu_guard: 2026-10-09T02:01:12+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:01:12+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:01:27+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:01:27+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/Mixtral-8x7B-Instruct-v0.1-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.057 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(32768), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 26.40 GB, cache 0.22 GB, active+cache 26.61 GB, phase peak 27.51 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(32768), reserved 128 for generation) +[Memory] after measured pass: active 26.40 GB, cache 0.22 GB, active+cache 26.61 GB, phase peak 27.47 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 19839.49 ms (25.81 tok/s) + Decode: 13411.61 ms (9.54 tok/s) + Decode path: engine + MLX peak memory: 27.51 GB +rocm_gpu_guard: 2026-10-09T02:02:26+0900 monitor: 39 samples +rocm_gpu_guard: 2026-10-09T02:02:26+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r2_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r2_base-null.log new file mode 100644 index 000000000..7c8d955a1 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r2_base-null.log @@ -0,0 +1,22 @@ +rocm_gpu_guard: 2026-10-09T02:21:04+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:21:04+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:21:19+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:21:19+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Mixtral-8x7B-Instruct-v0.1-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.053 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(32768), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 26.40 GB, cache 1.72 GB, active+cache 28.11 GB, phase peak 27.52 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(32768), reserved 128 for generation) +[Memory] after measured pass: active 26.40 GB, cache 0.37 GB, active+cache 26.76 GB, phase peak 27.47 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 19823.17 ms (25.83 tok/s) + Decode: 12615.91 ms (10.15 tok/s) + MLX peak memory: 27.52 GB +rocm_gpu_guard: 2026-10-09T02:22:20+0900 monitor: 40 samples +rocm_gpu_guard: 2026-10-09T02:22:20+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r2_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r2_base.log new file mode 100644 index 000000000..1458cbfe7 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r2_base.log @@ -0,0 +1,22 @@ +rocm_gpu_guard: 2026-10-09T02:22:25+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:22:25+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:22:39+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:22:39+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Mixtral-8x7B-Instruct-v0.1-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.070 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(32768), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 26.40 GB, cache 2.26 GB, active+cache 28.66 GB, phase peak 27.52 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(32768), reserved 128 for generation) +[Memory] after measured pass: active 26.40 GB, cache 0.37 GB, active+cache 26.76 GB, phase peak 27.47 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 19874.59 ms (25.76 tok/s) + Decode: 12839.71 ms (9.97 tok/s) + MLX peak memory: 27.52 GB +rocm_gpu_guard: 2026-10-09T02:23:38+0900 monitor: 38 samples +rocm_gpu_guard: 2026-10-09T02:23:38+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r2_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r2_new.log new file mode 100644 index 000000000..d8c70518a --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r2_new.log @@ -0,0 +1,23 @@ +rocm_gpu_guard: 2026-10-09T02:19:45+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:19:45+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:20:00+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:20:00+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/Mixtral-8x7B-Instruct-v0.1-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.054 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(32768), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 26.40 GB, cache 0.34 GB, active+cache 26.73 GB, phase peak 27.51 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(32768), reserved 128 for generation) +[Memory] after measured pass: active 26.40 GB, cache 0.33 GB, active+cache 26.73 GB, phase peak 27.47 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 19734.06 ms (25.94 tok/s) + Decode: 12681.93 ms (10.09 tok/s) + Decode path: engine + MLX peak memory: 27.51 GB +rocm_gpu_guard: 2026-10-09T02:20:59+0900 monitor: 39 samples +rocm_gpu_guard: 2026-10-09T02:20:59+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r3_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r3_base-null.log new file mode 100644 index 000000000..113b99b6d --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r3_base-null.log @@ -0,0 +1,22 @@ +rocm_gpu_guard: 2026-10-09T02:39:31+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:39:31+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:39:45+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:39:45+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Mixtral-8x7B-Instruct-v0.1-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.055 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(32768), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 26.40 GB, cache 2.26 GB, active+cache 28.66 GB, phase peak 27.52 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(32768), reserved 128 for generation) +[Memory] after measured pass: active 26.40 GB, cache 0.37 GB, active+cache 26.76 GB, phase peak 27.47 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 19711.80 ms (25.97 tok/s) + Decode: 13062.31 ms (9.80 tok/s) + MLX peak memory: 27.52 GB +rocm_gpu_guard: 2026-10-09T02:40:45+0900 monitor: 39 samples +rocm_gpu_guard: 2026-10-09T02:40:45+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r3_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r3_base.log new file mode 100644 index 000000000..0ddc91c4e --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r3_base.log @@ -0,0 +1,22 @@ +rocm_gpu_guard: 2026-10-09T02:40:50+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:40:50+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:41:04+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:41:04+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Mixtral-8x7B-Instruct-v0.1-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.054 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(32768), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 26.40 GB, cache 2.26 GB, active+cache 28.66 GB, phase peak 27.52 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(32768), reserved 128 for generation) +[Memory] after measured pass: active 26.40 GB, cache 0.37 GB, active+cache 26.76 GB, phase peak 27.47 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 19663.82 ms (26.04 tok/s) + Decode: 12839.24 ms (9.97 tok/s) + MLX peak memory: 27.52 GB +rocm_gpu_guard: 2026-10-09T02:42:18+0900 monitor: 48 samples +rocm_gpu_guard: 2026-10-09T02:42:18+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r3_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r3_new.log new file mode 100644 index 000000000..9ae178084 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Mixtral-8x7B-Instruct-v0.1-4bit_pp512_r3_new.log @@ -0,0 +1,23 @@ +rocm_gpu_guard: 2026-10-09T02:42:23+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:42:23+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:42:38+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:42:38+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/Mixtral-8x7B-Instruct-v0.1-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.065 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(32768), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 26.40 GB, cache 0.37 GB, active+cache 26.77 GB, phase peak 27.51 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(32768), reserved 128 for generation) +[Memory] after measured pass: active 26.40 GB, cache 0.34 GB, active+cache 26.74 GB, phase peak 27.47 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 19878.37 ms (25.76 tok/s) + Decode: 12548.46 ms (10.20 tok/s) + Decode path: engine + MLX peak memory: 27.51 GB +rocm_gpu_guard: 2026-10-09T02:43:37+0900 monitor: 38 samples +rocm_gpu_guard: 2026-10-09T02:43:37+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r1_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r1_base-null.log new file mode 100644 index 000000000..1b37a199e --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r1_base-null.log @@ -0,0 +1,27 @@ +rocm_gpu_guard: 2026-10-09T01:56:56+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T01:56:56+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T01:57:11+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T01:57:11+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[NemotronH] Loading config... +[NemotronH] Config loaded: 52 layers (23 mamba, 6 attention, 0 mlp, 23 moe) +[NemotronH] Loading weights from safetensors... +[NemotronH] Building model... +[NemotronH] Model loaded successfully +[Load] wall: 0.321 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=None, reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 18.01 GB, cache 0.30 GB, active+cache 18.32 GB, phase peak 18.17 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=None, reserved 128 for generation) +[Memory] after measured pass: active 18.07 GB, cache 0.21 GB, active+cache 18.27 GB, phase peak 18.66 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 1851.40 ms (276.55 tok/s) + Decode: 1697.24 ms (75.42 tok/s) + MLX peak memory: 18.66 GB +rocm_gpu_guard: 2026-10-09T01:57:26+0900 monitor: 10 samples +rocm_gpu_guard: 2026-10-09T01:57:26+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r1_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r1_base.log new file mode 100644 index 000000000..19ea57f93 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r1_base.log @@ -0,0 +1,27 @@ +rocm_gpu_guard: 2026-10-09T01:55:45+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T01:55:45+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T01:56:00+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T01:56:00+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[NemotronH] Loading config... +[NemotronH] Config loaded: 52 layers (23 mamba, 6 attention, 0 mlp, 23 moe) +[NemotronH] Loading weights from safetensors... +[NemotronH] Building model... +[NemotronH] Model loaded successfully +[Load] wall: 0.327 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=None, reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 18.01 GB, cache 0.30 GB, active+cache 18.32 GB, phase peak 18.17 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=None, reserved 128 for generation) +[Memory] after measured pass: active 18.07 GB, cache 0.21 GB, active+cache 18.27 GB, phase peak 18.66 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 1827.53 ms (280.16 tok/s) + Decode: 1698.51 ms (75.36 tok/s) + MLX peak memory: 18.66 GB +rocm_gpu_guard: 2026-10-09T01:56:16+0900 monitor: 11 samples +rocm_gpu_guard: 2026-10-09T01:56:16+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r1_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r1_new.log new file mode 100644 index 000000000..66f2305b7 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r1_new.log @@ -0,0 +1,23 @@ +rocm_gpu_guard: 2026-10-09T01:56:21+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T01:56:21+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T01:56:36+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T01:56:36+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.330 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=None, reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 18.01 GB, cache 0.17 GB, active+cache 18.18 GB, phase peak 18.17 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=None, reserved 128 for generation) +[Memory] after measured pass: active 18.01 GB, cache 0.26 GB, active+cache 18.27 GB, phase peak 18.66 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 1719.00 ms (297.85 tok/s) + Decode: 1695.18 ms (75.51 tok/s) + Decode path: engine + MLX peak memory: 18.66 GB +rocm_gpu_guard: 2026-10-09T01:56:51+0900 monitor: 10 samples +rocm_gpu_guard: 2026-10-09T01:56:51+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r2_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r2_base-null.log new file mode 100644 index 000000000..faad783e9 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r2_base-null.log @@ -0,0 +1,27 @@ +rocm_gpu_guard: 2026-10-09T02:16:12+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:16:12+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:16:27+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:16:27+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[NemotronH] Loading config... +[NemotronH] Config loaded: 52 layers (23 mamba, 6 attention, 0 mlp, 23 moe) +[NemotronH] Loading weights from safetensors... +[NemotronH] Building model... +[NemotronH] Model loaded successfully +[Load] wall: 0.330 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=None, reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 18.01 GB, cache 0.30 GB, active+cache 18.32 GB, phase peak 18.17 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=None, reserved 128 for generation) +[Memory] after measured pass: active 18.07 GB, cache 0.21 GB, active+cache 18.27 GB, phase peak 18.66 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 1819.72 ms (281.36 tok/s) + Decode: 1698.86 ms (75.34 tok/s) + MLX peak memory: 18.66 GB +rocm_gpu_guard: 2026-10-09T02:16:42+0900 monitor: 10 samples +rocm_gpu_guard: 2026-10-09T02:16:42+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r2_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r2_base.log new file mode 100644 index 000000000..04b8096e8 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r2_base.log @@ -0,0 +1,27 @@ +rocm_gpu_guard: 2026-10-09T02:16:47+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:16:47+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:17:01+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:17:01+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[NemotronH] Loading config... +[NemotronH] Config loaded: 52 layers (23 mamba, 6 attention, 0 mlp, 23 moe) +[NemotronH] Loading weights from safetensors... +[NemotronH] Building model... +[NemotronH] Model loaded successfully +[Load] wall: 0.329 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=None, reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 18.01 GB, cache 0.30 GB, active+cache 18.31 GB, phase peak 18.17 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=None, reserved 128 for generation) +[Memory] after measured pass: active 18.07 GB, cache 0.14 GB, active+cache 18.21 GB, phase peak 18.66 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 1767.78 ms (289.63 tok/s) + Decode: 1703.63 ms (75.13 tok/s) + MLX peak memory: 18.66 GB +rocm_gpu_guard: 2026-10-09T02:17:16+0900 monitor: 10 samples +rocm_gpu_guard: 2026-10-09T02:17:16+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r2_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r2_new.log new file mode 100644 index 000000000..2d77f0b75 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r2_new.log @@ -0,0 +1,23 @@ +rocm_gpu_guard: 2026-10-09T02:15:36+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:15:36+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:15:51+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:15:51+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.337 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=None, reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 18.01 GB, cache 0.18 GB, active+cache 18.19 GB, phase peak 18.17 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=None, reserved 128 for generation) +[Memory] after measured pass: active 18.01 GB, cache 0.26 GB, active+cache 18.27 GB, phase peak 18.66 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 2000.43 ms (255.95 tok/s) + Decode: 1700.67 ms (75.26 tok/s) + Decode path: engine + MLX peak memory: 18.66 GB +rocm_gpu_guard: 2026-10-09T02:16:07+0900 monitor: 11 samples +rocm_gpu_guard: 2026-10-09T02:16:07+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r3_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r3_base-null.log new file mode 100644 index 000000000..301464bad --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r3_base-null.log @@ -0,0 +1,27 @@ +rocm_gpu_guard: 2026-10-09T02:35:21+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:35:21+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:35:36+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:35:36+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[NemotronH] Loading config... +[NemotronH] Config loaded: 52 layers (23 mamba, 6 attention, 0 mlp, 23 moe) +[NemotronH] Loading weights from safetensors... +[NemotronH] Building model... +[NemotronH] Model loaded successfully +[Load] wall: 0.332 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=None, reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 18.01 GB, cache 0.30 GB, active+cache 18.31 GB, phase peak 18.17 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=None, reserved 128 for generation) +[Memory] after measured pass: active 18.07 GB, cache 0.21 GB, active+cache 18.27 GB, phase peak 18.66 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 1878.15 ms (272.61 tok/s) + Decode: 1696.57 ms (75.45 tok/s) + MLX peak memory: 18.66 GB +rocm_gpu_guard: 2026-10-09T02:35:52+0900 monitor: 11 samples +rocm_gpu_guard: 2026-10-09T02:35:52+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r3_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r3_base.log new file mode 100644 index 000000000..89793f6f2 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r3_base.log @@ -0,0 +1,27 @@ +rocm_gpu_guard: 2026-10-09T02:35:57+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:35:57+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:36:12+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:36:12+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[NemotronH] Loading config... +[NemotronH] Config loaded: 52 layers (23 mamba, 6 attention, 0 mlp, 23 moe) +[NemotronH] Loading weights from safetensors... +[NemotronH] Building model... +[NemotronH] Model loaded successfully +[Load] wall: 0.333 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=None, reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 18.01 GB, cache 0.30 GB, active+cache 18.32 GB, phase peak 18.17 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=None, reserved 128 for generation) +[Memory] after measured pass: active 18.07 GB, cache 0.21 GB, active+cache 18.27 GB, phase peak 18.66 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 1794.91 ms (285.25 tok/s) + Decode: 1700.18 ms (75.29 tok/s) + MLX peak memory: 18.66 GB +rocm_gpu_guard: 2026-10-09T02:36:27+0900 monitor: 10 samples +rocm_gpu_guard: 2026-10-09T02:36:27+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r3_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r3_new.log new file mode 100644 index 000000000..38c5bd715 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit_pp512_r3_new.log @@ -0,0 +1,23 @@ +rocm_gpu_guard: 2026-10-09T02:36:32+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:36:32+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:36:46+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:36:46+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.336 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=None, reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 18.01 GB, cache 0.17 GB, active+cache 18.18 GB, phase peak 18.17 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=None, reserved 128 for generation) +[Memory] after measured pass: active 18.01 GB, cache 0.23 GB, active+cache 18.24 GB, phase peak 18.66 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 1856.01 ms (275.86 tok/s) + Decode: 1694.83 ms (75.52 tok/s) + Decode path: engine + MLX peak memory: 18.66 GB +rocm_gpu_guard: 2026-10-09T02:37:01+0900 monitor: 10 samples +rocm_gpu_guard: 2026-10-09T02:37:01+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp2048_r1_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp2048_r1_base-null.log new file mode 100644 index 000000000..04c95815c --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp2048_r1_base-null.log @@ -0,0 +1,22 @@ +rocm_gpu_guard: 2026-10-09T01:19:10+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T01:19:10+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T01:45:57+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T01:45:57+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-0.6B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.273 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=Some(40960), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 0.39 GB, cache 1.80 GB, active+cache 2.18 GB, phase peak 1.84 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=Some(40960), reserved 128 for generation) +[Memory] after measured pass: active 0.39 GB, cache 0.88 GB, active+cache 1.27 GB, phase peak 1.85 GB +[Profile Results] + Prompt tokens: 2048 + Generated tokens: 128 + Prefill: 448.08 ms (4570.58 tok/s) + Decode: 590.32 ms (216.83 tok/s) + MLX peak memory: 1.85 GB +rocm_gpu_guard: 2026-10-09T01:46:01+0900 monitor: 3 samples +rocm_gpu_guard: 2026-10-09T01:46:01+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp2048_r1_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp2048_r1_base.log new file mode 100644 index 000000000..31b35ff19 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp2048_r1_base.log @@ -0,0 +1,25 @@ +rocm_gpu_guard: 2026-10-08T21:33:51+0900 waiting for guard lock /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-08T22:19:00+0900 acquired guard lock /tmp/mlxcel-rocm-gpu-guard.lock after 2709s +rocm_gpu_guard: 2026-10-08T22:19:00+0900 waiting for 60s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-08T22:19:18+0900 idle streak reset after 12s: gpu=[] compilers=[1173803:cargo] +rocm_gpu_guard: 2026-10-08T22:23:33+0900 idle streak reset after 1s: gpu=[] compilers=[1334471:cargo] +rocm_gpu_guard: 2026-10-08T23:05:01+0900 idle for 60s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-08T23:05:01+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-0.6B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.252 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=Some(40960), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 0.39 GB, cache 1.80 GB, active+cache 2.18 GB, phase peak 1.84 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=Some(40960), reserved 128 for generation) +[Memory] after measured pass: active 0.39 GB, cache 0.88 GB, active+cache 1.27 GB, phase peak 1.85 GB +[Profile Results] + Prompt tokens: 2048 + Generated tokens: 128 + Prefill: 432.66 ms (4733.48 tok/s) + Decode: 594.48 ms (215.31 tok/s) + MLX peak memory: 1.85 GB +rocm_gpu_guard: 2026-10-08T23:05:05+0900 monitor: 3 samples +rocm_gpu_guard: 2026-10-08T23:05:05+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp2048_r1_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp2048_r1_new.log new file mode 100644 index 000000000..f449f5bf9 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp2048_r1_new.log @@ -0,0 +1,26 @@ +rocm_gpu_guard: 2026-10-08T23:05:15+0900 waiting for guard lock /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-08T23:13:56+0900 acquired guard lock /tmp/mlxcel-rocm-gpu-guard.lock after 521s +rocm_gpu_guard: 2026-10-08T23:13:56+0900 waiting for 60s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-08T23:14:31+0900 idle streak reset after 23s: gpu=[3262038:rocminfo] compilers=[3262000:cargo] +rocm_gpu_guard: 2026-10-08T23:23:02+0900 idle streak reset after 14s: gpu=[] compilers=[3571564:cargo,3571605:rustc,3571606:rustc,3571607:rustc,3571608:rustc,3571609:rustc] +rocm_gpu_guard: 2026-10-08T23:28:13+0900 idle for 60s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-08T23:28:13+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-0.6B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.250 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=Some(40960), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 0.39 GB, cache 0.88 GB, active+cache 1.27 GB, phase peak 1.84 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=Some(40960), reserved 128 for generation) +[Memory] after measured pass: active 0.39 GB, cache 0.88 GB, active+cache 1.27 GB, phase peak 1.85 GB +[Profile Results] + Prompt tokens: 2048 + Generated tokens: 128 + Prefill: 474.47 ms (4316.35 tok/s) + Decode: 580.45 ms (220.52 tok/s) + Decode path: engine + MLX peak memory: 1.85 GB +rocm_gpu_guard: 2026-10-08T23:28:17+0900 monitor: 3 samples +rocm_gpu_guard: 2026-10-08T23:28:17+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp2048_r2_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp2048_r2_base-null.log new file mode 100644 index 000000000..f3165af6c --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp2048_r2_base-null.log @@ -0,0 +1,22 @@ +rocm_gpu_guard: 2026-10-09T02:05:19+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:05:19+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:05:34+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:05:34+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-0.6B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.195 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=Some(40960), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 0.39 GB, cache 1.81 GB, active+cache 2.19 GB, phase peak 1.84 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=Some(40960), reserved 128 for generation) +[Memory] after measured pass: active 0.39 GB, cache 0.88 GB, active+cache 1.27 GB, phase peak 1.85 GB +[Profile Results] + Prompt tokens: 2048 + Generated tokens: 128 + Prefill: 460.43 ms (4447.97 tok/s) + Decode: 586.82 ms (218.13 tok/s) + MLX peak memory: 1.85 GB +rocm_gpu_guard: 2026-10-09T02:05:38+0900 monitor: 3 samples +rocm_gpu_guard: 2026-10-09T02:05:38+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp2048_r2_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp2048_r2_base.log new file mode 100644 index 000000000..34c9ffe01 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp2048_r2_base.log @@ -0,0 +1,22 @@ +rocm_gpu_guard: 2026-10-09T02:05:43+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:05:43+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:05:58+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:05:58+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-0.6B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.199 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=Some(40960), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 0.39 GB, cache 1.81 GB, active+cache 2.19 GB, phase peak 1.84 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=Some(40960), reserved 128 for generation) +[Memory] after measured pass: active 0.39 GB, cache 1.00 GB, active+cache 1.39 GB, phase peak 1.85 GB +[Profile Results] + Prompt tokens: 2048 + Generated tokens: 128 + Prefill: 453.61 ms (4514.86 tok/s) + Decode: 574.66 ms (222.74 tok/s) + MLX peak memory: 1.85 GB +rocm_gpu_guard: 2026-10-09T02:06:02+0900 monitor: 3 samples +rocm_gpu_guard: 2026-10-09T02:06:02+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp2048_r2_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp2048_r2_new.log new file mode 100644 index 000000000..c9068650d --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp2048_r2_new.log @@ -0,0 +1,23 @@ +rocm_gpu_guard: 2026-10-09T02:04:57+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:04:57+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:05:11+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:05:11+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-0.6B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.193 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=Some(40960), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 0.39 GB, cache 0.99 GB, active+cache 1.37 GB, phase peak 1.84 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=Some(40960), reserved 128 for generation) +[Memory] after measured pass: active 0.39 GB, cache 0.88 GB, active+cache 1.27 GB, phase peak 1.85 GB +[Profile Results] + Prompt tokens: 2048 + Generated tokens: 128 + Prefill: 450.84 ms (4542.64 tok/s) + Decode: 590.98 ms (216.59 tok/s) + Decode path: engine + MLX peak memory: 1.85 GB +rocm_gpu_guard: 2026-10-09T02:05:14+0900 monitor: 2 samples +rocm_gpu_guard: 2026-10-09T02:05:14+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp2048_r3_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp2048_r3_base-null.log new file mode 100644 index 000000000..aff771ccf --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp2048_r3_base-null.log @@ -0,0 +1,22 @@ +rocm_gpu_guard: 2026-10-09T02:24:51+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:24:51+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:25:05+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:25:05+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-0.6B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.200 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=Some(40960), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 0.39 GB, cache 1.82 GB, active+cache 2.20 GB, phase peak 1.86 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=Some(40960), reserved 128 for generation) +[Memory] after measured pass: active 0.39 GB, cache 1.00 GB, active+cache 1.39 GB, phase peak 1.85 GB +[Profile Results] + Prompt tokens: 2048 + Generated tokens: 128 + Prefill: 452.00 ms (4530.98 tok/s) + Decode: 588.40 ms (217.54 tok/s) + MLX peak memory: 1.86 GB +rocm_gpu_guard: 2026-10-09T02:25:08+0900 monitor: 2 samples +rocm_gpu_guard: 2026-10-09T02:25:08+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp2048_r3_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp2048_r3_base.log new file mode 100644 index 000000000..ba360683d --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp2048_r3_base.log @@ -0,0 +1,22 @@ +rocm_gpu_guard: 2026-10-09T02:25:13+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:25:13+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:25:28+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:25:28+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-0.6B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.205 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=Some(40960), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 0.39 GB, cache 1.81 GB, active+cache 2.19 GB, phase peak 1.84 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=Some(40960), reserved 128 for generation) +[Memory] after measured pass: active 0.39 GB, cache 1.00 GB, active+cache 1.39 GB, phase peak 1.85 GB +[Profile Results] + Prompt tokens: 2048 + Generated tokens: 128 + Prefill: 462.41 ms (4428.99 tok/s) + Decode: 586.97 ms (218.07 tok/s) + MLX peak memory: 1.85 GB +rocm_gpu_guard: 2026-10-09T02:25:32+0900 monitor: 3 samples +rocm_gpu_guard: 2026-10-09T02:25:32+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp2048_r3_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp2048_r3_new.log new file mode 100644 index 000000000..4e6ad39a4 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp2048_r3_new.log @@ -0,0 +1,23 @@ +rocm_gpu_guard: 2026-10-09T02:25:37+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:25:37+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:25:52+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:25:52+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-0.6B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.198 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=Some(40960), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 0.39 GB, cache 0.88 GB, active+cache 1.27 GB, phase peak 1.84 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=Some(40960), reserved 128 for generation) +[Memory] after measured pass: active 0.39 GB, cache 0.96 GB, active+cache 1.35 GB, phase peak 1.85 GB +[Profile Results] + Prompt tokens: 2048 + Generated tokens: 128 + Prefill: 454.49 ms (4506.18 tok/s) + Decode: 591.13 ms (216.53 tok/s) + Decode path: engine + MLX peak memory: 1.85 GB +rocm_gpu_guard: 2026-10-09T02:25:55+0900 monitor: 2 samples +rocm_gpu_guard: 2026-10-09T02:25:55+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp512_r1_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp512_r1_base-null.log new file mode 100644 index 000000000..d4f5a5e65 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp512_r1_base-null.log @@ -0,0 +1,23 @@ +rocm_gpu_guard: 2026-10-08T21:22:36+0900 waiting for guard lock /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-08T21:32:09+0900 acquired guard lock /tmp/mlxcel-rocm-gpu-guard.lock after 573s +rocm_gpu_guard: 2026-10-08T21:32:09+0900 waiting for 60s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-08T21:33:38+0900 idle for 60s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-08T21:33:38+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-0.6B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.192 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(40960), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 0.39 GB, cache 0.77 GB, active+cache 1.15 GB, phase peak 1.04 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(40960), reserved 128 for generation) +[Memory] after measured pass: active 0.39 GB, cache 0.31 GB, active+cache 0.69 GB, phase peak 1.46 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 70.41 ms (7271.58 tok/s) + Decode: 462.36 ms (276.84 tok/s) + MLX peak memory: 1.46 GB +rocm_gpu_guard: 2026-10-08T21:33:41+0900 monitor: 2 samples +rocm_gpu_guard: 2026-10-08T21:33:41+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp512_r1_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp512_r1_base.log new file mode 100644 index 000000000..35457a283 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp512_r1_base.log @@ -0,0 +1,24 @@ +rocm_gpu_guard: 2026-10-08T19:17:44+0900 waiting for guard lock /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-08T19:41:27+0900 acquired guard lock /tmp/mlxcel-rocm-gpu-guard.lock after 1423s +rocm_gpu_guard: 2026-10-08T19:41:27+0900 waiting for 60s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-08T19:42:09+0900 idle streak reset after 28s: gpu=[] compilers=[3313432:cargo] +rocm_gpu_guard: 2026-10-08T20:39:08+0900 idle for 60s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-08T20:39:08+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-0.6B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.741 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(40960), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 0.39 GB, cache 0.36 GB, active+cache 0.74 GB, phase peak 0.63 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(40960), reserved 128 for generation) +[Memory] after measured pass: active 0.39 GB, cache 0.26 GB, active+cache 0.65 GB, phase peak 1.46 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 73.12 ms (7002.12 tok/s) + Decode: 465.33 ms (275.07 tok/s) + MLX peak memory: 1.46 GB +rocm_gpu_guard: 2026-10-08T20:39:15+0900 monitor: 5 samples +rocm_gpu_guard: 2026-10-08T20:39:15+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp512_r1_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp512_r1_new.log new file mode 100644 index 000000000..7112e48e7 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp512_r1_new.log @@ -0,0 +1,24 @@ +rocm_gpu_guard: 2026-10-08T20:39:25+0900 waiting for guard lock /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-08T21:20:56+0900 acquired guard lock /tmp/mlxcel-rocm-gpu-guard.lock after 2491s +rocm_gpu_guard: 2026-10-08T21:20:56+0900 waiting for 60s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-08T21:22:23+0900 idle for 60s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-08T21:22:23+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-0.6B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.256 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(40960), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 0.39 GB, cache 0.26 GB, active+cache 0.64 GB, phase peak 0.60 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(40960), reserved 128 for generation) +[Memory] after measured pass: active 0.39 GB, cache 0.26 GB, active+cache 0.65 GB, phase peak 1.46 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 59.61 ms (8589.40 tok/s) + Decode: 468.02 ms (273.49 tok/s) + Decode path: engine + MLX peak memory: 1.46 GB +rocm_gpu_guard: 2026-10-08T21:22:26+0900 monitor: 2 samples +rocm_gpu_guard: 2026-10-08T21:22:26+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp512_r2_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp512_r2_base-null.log new file mode 100644 index 000000000..a8b18a976 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp512_r2_base-null.log @@ -0,0 +1,22 @@ +rocm_gpu_guard: 2026-10-09T02:04:12+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:04:12+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:04:26+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:04:26+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-0.6B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.197 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(40960), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 0.39 GB, cache 0.67 GB, active+cache 1.06 GB, phase peak 0.95 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(40960), reserved 128 for generation) +[Memory] after measured pass: active 0.39 GB, cache 0.31 GB, active+cache 0.69 GB, phase peak 1.47 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 61.93 ms (8267.07 tok/s) + Decode: 464.92 ms (275.31 tok/s) + MLX peak memory: 1.47 GB +rocm_gpu_guard: 2026-10-09T02:04:29+0900 monitor: 2 samples +rocm_gpu_guard: 2026-10-09T02:04:29+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp512_r2_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp512_r2_base.log new file mode 100644 index 000000000..85f974b44 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp512_r2_base.log @@ -0,0 +1,22 @@ +rocm_gpu_guard: 2026-10-09T02:04:34+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:04:34+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:04:49+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:04:49+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-0.6B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.192 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(40960), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 0.39 GB, cache 0.61 GB, active+cache 1.00 GB, phase peak 0.87 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(40960), reserved 128 for generation) +[Memory] after measured pass: active 0.39 GB, cache 0.31 GB, active+cache 0.69 GB, phase peak 1.47 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 68.65 ms (7458.21 tok/s) + Decode: 463.13 ms (276.38 tok/s) + MLX peak memory: 1.47 GB +rocm_gpu_guard: 2026-10-09T02:04:52+0900 monitor: 2 samples +rocm_gpu_guard: 2026-10-09T02:04:52+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp512_r2_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp512_r2_new.log new file mode 100644 index 000000000..d92cd768f --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp512_r2_new.log @@ -0,0 +1,23 @@ +rocm_gpu_guard: 2026-10-09T02:03:49+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:03:49+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:04:04+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:04:04+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-0.6B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.211 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(40960), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 0.39 GB, cache 0.26 GB, active+cache 0.64 GB, phase peak 0.60 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(40960), reserved 128 for generation) +[Memory] after measured pass: active 0.39 GB, cache 0.26 GB, active+cache 0.65 GB, phase peak 1.46 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 59.82 ms (8559.12 tok/s) + Decode: 467.00 ms (274.09 tok/s) + Decode path: engine + MLX peak memory: 1.46 GB +rocm_gpu_guard: 2026-10-09T02:04:06+0900 monitor: 2 samples +rocm_gpu_guard: 2026-10-09T02:04:06+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp512_r3_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp512_r3_base-null.log new file mode 100644 index 000000000..504582ada --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp512_r3_base-null.log @@ -0,0 +1,22 @@ +rocm_gpu_guard: 2026-10-09T02:23:43+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:23:43+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:23:58+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:23:58+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-0.6B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.195 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(40960), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 0.39 GB, cache 0.56 GB, active+cache 0.95 GB, phase peak 0.83 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(40960), reserved 128 for generation) +[Memory] after measured pass: active 0.39 GB, cache 0.31 GB, active+cache 0.69 GB, phase peak 1.46 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 65.70 ms (7793.54 tok/s) + Decode: 462.61 ms (276.69 tok/s) + MLX peak memory: 1.46 GB +rocm_gpu_guard: 2026-10-09T02:24:01+0900 monitor: 2 samples +rocm_gpu_guard: 2026-10-09T02:24:01+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp512_r3_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp512_r3_base.log new file mode 100644 index 000000000..ae61fa0ba --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp512_r3_base.log @@ -0,0 +1,22 @@ +rocm_gpu_guard: 2026-10-09T02:24:06+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:24:06+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:24:20+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:24:20+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-0.6B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.198 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(40960), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 0.39 GB, cache 0.55 GB, active+cache 0.93 GB, phase peak 0.82 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(40960), reserved 128 for generation) +[Memory] after measured pass: active 0.39 GB, cache 0.31 GB, active+cache 0.69 GB, phase peak 1.46 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 68.96 ms (7424.31 tok/s) + Decode: 465.01 ms (275.26 tok/s) + MLX peak memory: 1.46 GB +rocm_gpu_guard: 2026-10-09T02:24:23+0900 monitor: 2 samples +rocm_gpu_guard: 2026-10-09T02:24:23+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp512_r3_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp512_r3_new.log new file mode 100644 index 000000000..3639461b2 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-0.6B-4bit_pp512_r3_new.log @@ -0,0 +1,23 @@ +rocm_gpu_guard: 2026-10-09T02:24:28+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:24:28+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:24:43+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:24:43+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-0.6B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.194 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(40960), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 0.39 GB, cache 0.30 GB, active+cache 0.69 GB, phase peak 0.83 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(40960), reserved 128 for generation) +[Memory] after measured pass: active 0.39 GB, cache 0.31 GB, active+cache 0.69 GB, phase peak 1.46 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 68.81 ms (7440.48 tok/s) + Decode: 465.67 ms (274.88 tok/s) + Decode path: engine + MLX peak memory: 1.46 GB +rocm_gpu_guard: 2026-10-09T02:24:46+0900 monitor: 2 samples +rocm_gpu_guard: 2026-10-09T02:24:46+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp2048_r1_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp2048_r1_base-null.log new file mode 100644 index 000000000..7c2287d83 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp2048_r1_base-null.log @@ -0,0 +1,22 @@ +rocm_gpu_guard: 2026-10-09T01:53:41+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T01:53:41+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T01:53:56+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T01:53:56+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.212 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=Some(40960), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 17.27 GB, cache 1.91 GB, active+cache 19.17 GB, phase peak 18.80 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=Some(40960), reserved 128 for generation) +[Memory] after measured pass: active 17.27 GB, cache 1.37 GB, active+cache 18.64 GB, phase peak 18.91 GB +[Profile Results] + Prompt tokens: 2048 + Generated tokens: 128 + Prefill: 7150.03 ms (286.43 tok/s) + Decode: 2200.86 ms (58.16 tok/s) + MLX peak memory: 18.91 GB +rocm_gpu_guard: 2026-10-09T01:54:21+0900 monitor: 17 samples +rocm_gpu_guard: 2026-10-09T01:54:21+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp2048_r1_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp2048_r1_base.log new file mode 100644 index 000000000..c90caa743 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp2048_r1_base.log @@ -0,0 +1,22 @@ +rocm_gpu_guard: 2026-10-09T01:52:12+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T01:52:12+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T01:52:26+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T01:52:26+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.210 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=Some(40960), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 17.27 GB, cache 1.91 GB, active+cache 19.17 GB, phase peak 18.80 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=Some(40960), reserved 128 for generation) +[Memory] after measured pass: active 17.27 GB, cache 1.54 GB, active+cache 18.81 GB, phase peak 18.91 GB +[Profile Results] + Prompt tokens: 2048 + Generated tokens: 128 + Prefill: 7161.06 ms (285.99 tok/s) + Decode: 2205.06 ms (58.05 tok/s) + MLX peak memory: 18.91 GB +rocm_gpu_guard: 2026-10-09T01:52:52+0900 monitor: 17 samples +rocm_gpu_guard: 2026-10-09T01:52:52+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp2048_r1_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp2048_r1_new.log new file mode 100644 index 000000000..5130d3887 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp2048_r1_new.log @@ -0,0 +1,23 @@ +rocm_gpu_guard: 2026-10-09T01:52:57+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T01:52:57+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T01:53:11+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T01:53:11+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.207 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=Some(40960), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 17.27 GB, cache 1.37 GB, active+cache 18.64 GB, phase peak 18.80 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=Some(40960), reserved 128 for generation) +[Memory] after measured pass: active 17.27 GB, cache 1.54 GB, active+cache 18.81 GB, phase peak 18.91 GB +[Profile Results] + Prompt tokens: 2048 + Generated tokens: 128 + Prefill: 7249.95 ms (282.48 tok/s) + Decode: 2196.16 ms (58.28 tok/s) + Decode path: engine + MLX peak memory: 18.91 GB +rocm_gpu_guard: 2026-10-09T01:53:36+0900 monitor: 17 samples +rocm_gpu_guard: 2026-10-09T01:53:36+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp2048_r2_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp2048_r2_base-null.log new file mode 100644 index 000000000..e00939b08 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp2048_r2_base-null.log @@ -0,0 +1,22 @@ +rocm_gpu_guard: 2026-10-09T02:12:50+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:12:50+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:13:04+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:13:04+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.217 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=Some(40960), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 17.27 GB, cache 1.90 GB, active+cache 19.17 GB, phase peak 18.80 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=Some(40960), reserved 128 for generation) +[Memory] after measured pass: active 17.27 GB, cache 1.54 GB, active+cache 18.81 GB, phase peak 18.91 GB +[Profile Results] + Prompt tokens: 2048 + Generated tokens: 128 + Prefill: 7272.66 ms (281.60 tok/s) + Decode: 2200.73 ms (58.16 tok/s) + MLX peak memory: 18.91 GB +rocm_gpu_guard: 2026-10-09T02:13:28+0900 monitor: 16 samples +rocm_gpu_guard: 2026-10-09T02:13:28+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp2048_r2_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp2048_r2_base.log new file mode 100644 index 000000000..7fc69a5b3 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp2048_r2_base.log @@ -0,0 +1,22 @@ +rocm_gpu_guard: 2026-10-09T02:13:33+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:13:33+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:13:48+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:13:48+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.251 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=Some(40960), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 17.27 GB, cache 1.90 GB, active+cache 19.17 GB, phase peak 18.80 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=Some(40960), reserved 128 for generation) +[Memory] after measured pass: active 17.27 GB, cache 1.54 GB, active+cache 18.81 GB, phase peak 18.91 GB +[Profile Results] + Prompt tokens: 2048 + Generated tokens: 128 + Prefill: 7144.89 ms (286.64 tok/s) + Decode: 2202.88 ms (58.11 tok/s) + MLX peak memory: 18.91 GB +rocm_gpu_guard: 2026-10-09T02:14:13+0900 monitor: 17 samples +rocm_gpu_guard: 2026-10-09T02:14:13+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp2048_r2_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp2048_r2_new.log new file mode 100644 index 000000000..7cf1a0ea4 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp2048_r2_new.log @@ -0,0 +1,23 @@ +rocm_gpu_guard: 2026-10-09T02:11:54+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:11:54+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:12:09+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:12:09+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.221 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=Some(40960), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 17.27 GB, cache 1.37 GB, active+cache 18.64 GB, phase peak 18.80 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=Some(40960), reserved 128 for generation) +[Memory] after measured pass: active 17.27 GB, cache 1.37 GB, active+cache 18.64 GB, phase peak 18.91 GB +[Profile Results] + Prompt tokens: 2048 + Generated tokens: 128 + Prefill: 7053.70 ms (290.34 tok/s) + Decode: 2201.71 ms (58.14 tok/s) + Decode path: engine + MLX peak memory: 18.91 GB +rocm_gpu_guard: 2026-10-09T02:12:44+0900 monitor: 24 samples +rocm_gpu_guard: 2026-10-09T02:12:44+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp2048_r3_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp2048_r3_base-null.log new file mode 100644 index 000000000..d8f0ae959 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp2048_r3_base-null.log @@ -0,0 +1,22 @@ +rocm_gpu_guard: 2026-10-09T02:31:48+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:31:48+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:32:03+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:32:03+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.217 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=Some(40960), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 17.27 GB, cache 1.90 GB, active+cache 19.17 GB, phase peak 18.80 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=Some(40960), reserved 128 for generation) +[Memory] after measured pass: active 17.27 GB, cache 1.54 GB, active+cache 18.81 GB, phase peak 18.91 GB +[Profile Results] + Prompt tokens: 2048 + Generated tokens: 128 + Prefill: 7198.39 ms (284.51 tok/s) + Decode: 2208.97 ms (57.95 tok/s) + MLX peak memory: 18.91 GB +rocm_gpu_guard: 2026-10-09T02:32:28+0900 monitor: 17 samples +rocm_gpu_guard: 2026-10-09T02:32:28+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp2048_r3_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp2048_r3_base.log new file mode 100644 index 000000000..1822eec3f --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp2048_r3_base.log @@ -0,0 +1,22 @@ +rocm_gpu_guard: 2026-10-09T02:32:33+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:32:33+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:32:47+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:32:47+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.215 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=Some(40960), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 17.27 GB, cache 1.90 GB, active+cache 19.17 GB, phase peak 18.80 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=Some(40960), reserved 128 for generation) +[Memory] after measured pass: active 17.27 GB, cache 1.54 GB, active+cache 18.81 GB, phase peak 18.91 GB +[Profile Results] + Prompt tokens: 2048 + Generated tokens: 128 + Prefill: 7080.17 ms (289.26 tok/s) + Decode: 2192.55 ms (58.38 tok/s) + MLX peak memory: 18.91 GB +rocm_gpu_guard: 2026-10-09T02:33:13+0900 monitor: 17 samples +rocm_gpu_guard: 2026-10-09T02:33:13+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp2048_r3_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp2048_r3_new.log new file mode 100644 index 000000000..e7c91f7b0 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp2048_r3_new.log @@ -0,0 +1,23 @@ +rocm_gpu_guard: 2026-10-09T02:33:18+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:33:18+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:33:32+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:33:32+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.215 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=Some(40960), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 17.27 GB, cache 1.37 GB, active+cache 18.63 GB, phase peak 18.80 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=Some(40960), reserved 128 for generation) +[Memory] after measured pass: active 17.27 GB, cache 1.37 GB, active+cache 18.64 GB, phase peak 18.91 GB +[Profile Results] + Prompt tokens: 2048 + Generated tokens: 128 + Prefill: 7148.04 ms (286.51 tok/s) + Decode: 2204.57 ms (58.06 tok/s) + Decode path: engine + MLX peak memory: 18.91 GB +rocm_gpu_guard: 2026-10-09T02:33:57+0900 monitor: 17 samples +rocm_gpu_guard: 2026-10-09T02:33:57+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp512_r1_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp512_r1_base-null.log new file mode 100644 index 000000000..a7916b02e --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp512_r1_base-null.log @@ -0,0 +1,22 @@ +rocm_gpu_guard: 2026-10-09T01:51:37+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T01:51:37+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T01:51:52+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T01:51:52+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.209 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(40960), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 17.27 GB, cache 0.65 GB, active+cache 17.92 GB, phase peak 17.66 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(40960), reserved 128 for generation) +[Memory] after measured pass: active 17.27 GB, cache 0.50 GB, active+cache 17.77 GB, phase peak 18.66 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 1806.26 ms (283.46 tok/s) + Decode: 2042.81 ms (62.66 tok/s) + MLX peak memory: 18.66 GB +rocm_gpu_guard: 2026-10-09T01:52:07+0900 monitor: 10 samples +rocm_gpu_guard: 2026-10-09T01:52:07+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp512_r1_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp512_r1_base.log new file mode 100644 index 000000000..20dccea75 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp512_r1_base.log @@ -0,0 +1,22 @@ +rocm_gpu_guard: 2026-10-09T01:50:11+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T01:50:11+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T01:50:26+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T01:50:26+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.216 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(40960), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 17.27 GB, cache 0.65 GB, active+cache 17.92 GB, phase peak 17.66 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(40960), reserved 128 for generation) +[Memory] after measured pass: active 17.27 GB, cache 0.50 GB, active+cache 17.77 GB, phase peak 18.66 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 2292.18 ms (223.37 tok/s) + Decode: 2046.99 ms (62.53 tok/s) + MLX peak memory: 18.66 GB +rocm_gpu_guard: 2026-10-09T01:50:58+0900 monitor: 21 samples +rocm_gpu_guard: 2026-10-09T01:50:58+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp512_r1_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp512_r1_new.log new file mode 100644 index 000000000..87bc3e5b4 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp512_r1_new.log @@ -0,0 +1,23 @@ +rocm_gpu_guard: 2026-10-09T01:51:03+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T01:51:03+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T01:51:17+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T01:51:17+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.216 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(40960), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 17.27 GB, cache 0.38 GB, active+cache 17.65 GB, phase peak 17.66 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(40960), reserved 128 for generation) +[Memory] after measured pass: active 17.27 GB, cache 0.50 GB, active+cache 17.77 GB, phase peak 18.66 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 1889.81 ms (270.93 tok/s) + Decode: 2047.51 ms (62.52 tok/s) + Decode path: engine + MLX peak memory: 18.66 GB +rocm_gpu_guard: 2026-10-09T01:51:32+0900 monitor: 10 samples +rocm_gpu_guard: 2026-10-09T01:51:32+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp512_r2_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp512_r2_base-null.log new file mode 100644 index 000000000..0f456150f --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp512_r2_base-null.log @@ -0,0 +1,22 @@ +rocm_gpu_guard: 2026-10-09T02:10:44+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:10:44+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:10:59+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:10:59+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.208 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(40960), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 17.27 GB, cache 0.65 GB, active+cache 17.92 GB, phase peak 17.66 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(40960), reserved 128 for generation) +[Memory] after measured pass: active 17.27 GB, cache 0.50 GB, active+cache 17.77 GB, phase peak 18.66 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 1605.19 ms (318.97 tok/s) + Decode: 2048.39 ms (62.49 tok/s) + MLX peak memory: 18.66 GB +rocm_gpu_guard: 2026-10-09T02:11:14+0900 monitor: 10 samples +rocm_gpu_guard: 2026-10-09T02:11:14+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp512_r2_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp512_r2_base.log new file mode 100644 index 000000000..634199821 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp512_r2_base.log @@ -0,0 +1,22 @@ +rocm_gpu_guard: 2026-10-09T02:11:19+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:11:19+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:11:34+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:11:34+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.213 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(40960), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 17.27 GB, cache 0.65 GB, active+cache 17.92 GB, phase peak 17.66 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(40960), reserved 128 for generation) +[Memory] after measured pass: active 17.27 GB, cache 0.50 GB, active+cache 17.76 GB, phase peak 18.66 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 1736.22 ms (294.89 tok/s) + Decode: 2055.99 ms (62.26 tok/s) + MLX peak memory: 18.66 GB +rocm_gpu_guard: 2026-10-09T02:11:49+0900 monitor: 10 samples +rocm_gpu_guard: 2026-10-09T02:11:49+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp512_r2_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp512_r2_new.log new file mode 100644 index 000000000..484b481ff --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp512_r2_new.log @@ -0,0 +1,23 @@ +rocm_gpu_guard: 2026-10-09T02:10:10+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:10:10+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:10:25+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:10:25+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.220 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(40960), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 17.27 GB, cache 0.38 GB, active+cache 17.65 GB, phase peak 17.66 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(40960), reserved 128 for generation) +[Memory] after measured pass: active 17.27 GB, cache 0.39 GB, active+cache 17.66 GB, phase peak 18.66 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 1964.26 ms (260.66 tok/s) + Decode: 2050.47 ms (62.42 tok/s) + Decode path: engine + MLX peak memory: 18.66 GB +rocm_gpu_guard: 2026-10-09T02:10:39+0900 monitor: 10 samples +rocm_gpu_guard: 2026-10-09T02:10:39+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp512_r3_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp512_r3_base-null.log new file mode 100644 index 000000000..47da1dbfb --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp512_r3_base-null.log @@ -0,0 +1,22 @@ +rocm_gpu_guard: 2026-10-09T02:30:03+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:30:03+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:30:18+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:30:18+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.213 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(40960), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 17.27 GB, cache 0.65 GB, active+cache 17.92 GB, phase peak 17.66 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(40960), reserved 128 for generation) +[Memory] after measured pass: active 17.27 GB, cache 0.50 GB, active+cache 17.76 GB, phase peak 18.66 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 1982.61 ms (258.25 tok/s) + Decode: 2039.94 ms (62.75 tok/s) + MLX peak memory: 18.66 GB +rocm_gpu_guard: 2026-10-09T02:30:34+0900 monitor: 11 samples +rocm_gpu_guard: 2026-10-09T02:30:34+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp512_r3_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp512_r3_base.log new file mode 100644 index 000000000..f63e12679 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp512_r3_base.log @@ -0,0 +1,22 @@ +rocm_gpu_guard: 2026-10-09T02:30:39+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:30:39+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:30:54+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:30:54+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.220 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(40960), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 17.27 GB, cache 0.65 GB, active+cache 17.92 GB, phase peak 17.66 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(40960), reserved 128 for generation) +[Memory] after measured pass: active 17.27 GB, cache 0.50 GB, active+cache 17.76 GB, phase peak 18.66 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 1807.65 ms (283.24 tok/s) + Decode: 2039.27 ms (62.77 tok/s) + MLX peak memory: 18.66 GB +rocm_gpu_guard: 2026-10-09T02:31:09+0900 monitor: 10 samples +rocm_gpu_guard: 2026-10-09T02:31:09+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp512_r3_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp512_r3_new.log new file mode 100644 index 000000000..f268c4011 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/Qwen3-30B-A3B-4bit_pp512_r3_new.log @@ -0,0 +1,23 @@ +rocm_gpu_guard: 2026-10-09T02:31:14+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:31:14+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:31:28+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:31:28+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/Qwen3-30B-A3B-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.223 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(40960), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 17.27 GB, cache 0.38 GB, active+cache 17.65 GB, phase peak 17.66 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(40960), reserved 128 for generation) +[Memory] after measured pass: active 17.27 GB, cache 0.50 GB, active+cache 17.76 GB, phase peak 18.66 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 1903.10 ms (269.03 tok/s) + Decode: 2041.74 ms (62.69 tok/s) + Decode path: engine + MLX peak memory: 18.66 GB +rocm_gpu_guard: 2026-10-09T02:31:43+0900 monitor: 10 samples +rocm_gpu_guard: 2026-10-09T02:31:43+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp2048_r1_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp2048_r1_base-null.log new file mode 100644 index 000000000..90c19449e --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp2048_r1_base-null.log @@ -0,0 +1,23 @@ +rocm_gpu_guard: 2026-10-09T01:49:44+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T01:49:44+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T01:49:58+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T01:49:58+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/gemma-3-4b-it-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +Warning: model.safetensors.index.json in models/mlx/gemma-3-4b-it-4bit references shards that don't match the on-disk files (likely a repackaged mlx-community quant). Falling back to all *.safetensors files in the directory. +[Load] wall: 0.630 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=None, reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 2.80 GB, cache 2.47 GB, active+cache 5.28 GB, phase peak 4.27 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=None, reserved 128 for generation) +[Memory] after measured pass: active 2.97 GB, cache 1.04 GB, active+cache 4.02 GB, phase peak 4.52 GB +[Profile Results] + Prompt tokens: 2048 + Generated tokens: 128 + Prefill: 702.59 ms (2914.94 tok/s) + Decode: 2046.94 ms (62.53 tok/s) + MLX peak memory: 4.52 GB +rocm_gpu_guard: 2026-10-09T01:50:06+0900 monitor: 5 samples +rocm_gpu_guard: 2026-10-09T01:50:06+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp2048_r1_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp2048_r1_base.log new file mode 100644 index 000000000..06a2380fd --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp2048_r1_base.log @@ -0,0 +1,23 @@ +rocm_gpu_guard: 2026-10-09T01:48:50+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T01:48:50+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T01:49:04+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T01:49:04+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/gemma-3-4b-it-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +Warning: model.safetensors.index.json in models/mlx/gemma-3-4b-it-4bit references shards that don't match the on-disk files (likely a repackaged mlx-community quant). Falling back to all *.safetensors files in the directory. +[Load] wall: 0.633 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=None, reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 2.80 GB, cache 2.47 GB, active+cache 5.27 GB, phase peak 4.27 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=None, reserved 128 for generation) +[Memory] after measured pass: active 2.97 GB, cache 1.21 GB, active+cache 4.18 GB, phase peak 4.52 GB +[Profile Results] + Prompt tokens: 2048 + Generated tokens: 128 + Prefill: 720.27 ms (2843.38 tok/s) + Decode: 2037.11 ms (62.83 tok/s) + MLX peak memory: 4.52 GB +rocm_gpu_guard: 2026-10-09T01:49:12+0900 monitor: 5 samples +rocm_gpu_guard: 2026-10-09T01:49:12+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp2048_r1_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp2048_r1_new.log new file mode 100644 index 000000000..813cdab26 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp2048_r1_new.log @@ -0,0 +1,24 @@ +rocm_gpu_guard: 2026-10-09T01:49:17+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T01:49:17+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T01:49:31+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T01:49:31+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/gemma-3-4b-it-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +Warning: model.safetensors.index.json in models/mlx/gemma-3-4b-it-4bit references shards that don't match the on-disk files (likely a repackaged mlx-community quant). Falling back to all *.safetensors files in the directory. +[Load] wall: 0.639 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=None, reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 2.80 GB, cache 1.24 GB, active+cache 4.04 GB, phase peak 4.27 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=None, reserved 128 for generation) +[Memory] after measured pass: active 2.80 GB, cache 1.33 GB, active+cache 4.14 GB, phase peak 4.52 GB +[Profile Results] + Prompt tokens: 2048 + Generated tokens: 128 + Prefill: 702.60 ms (2914.89 tok/s) + Decode: 2061.91 ms (62.08 tok/s) + Decode path: engine + MLX peak memory: 4.52 GB +rocm_gpu_guard: 2026-10-09T01:49:39+0900 monitor: 5 samples +rocm_gpu_guard: 2026-10-09T01:49:39+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp2048_r2_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp2048_r2_base-null.log new file mode 100644 index 000000000..310273323 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp2048_r2_base-null.log @@ -0,0 +1,23 @@ +rocm_gpu_guard: 2026-10-09T02:09:16+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:09:16+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:09:31+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:09:31+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/gemma-3-4b-it-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +Warning: model.safetensors.index.json in models/mlx/gemma-3-4b-it-4bit references shards that don't match the on-disk files (likely a repackaged mlx-community quant). Falling back to all *.safetensors files in the directory. +[Load] wall: 0.668 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=None, reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 2.80 GB, cache 2.47 GB, active+cache 5.28 GB, phase peak 4.27 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=None, reserved 128 for generation) +[Memory] after measured pass: active 2.97 GB, cache 1.21 GB, active+cache 4.18 GB, phase peak 4.52 GB +[Profile Results] + Prompt tokens: 2048 + Generated tokens: 128 + Prefill: 703.54 ms (2911.00 tok/s) + Decode: 2046.37 ms (62.55 tok/s) + MLX peak memory: 4.52 GB +rocm_gpu_guard: 2026-10-09T02:09:38+0900 monitor: 5 samples +rocm_gpu_guard: 2026-10-09T02:09:38+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp2048_r2_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp2048_r2_base.log new file mode 100644 index 000000000..a1da50093 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp2048_r2_base.log @@ -0,0 +1,23 @@ +rocm_gpu_guard: 2026-10-09T02:09:43+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:09:43+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:09:58+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:09:58+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/gemma-3-4b-it-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +Warning: model.safetensors.index.json in models/mlx/gemma-3-4b-it-4bit references shards that don't match the on-disk files (likely a repackaged mlx-community quant). Falling back to all *.safetensors files in the directory. +[Load] wall: 0.662 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=None, reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 2.80 GB, cache 2.47 GB, active+cache 5.28 GB, phase peak 4.27 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=None, reserved 128 for generation) +[Memory] after measured pass: active 2.97 GB, cache 1.04 GB, active+cache 4.02 GB, phase peak 4.52 GB +[Profile Results] + Prompt tokens: 2048 + Generated tokens: 128 + Prefill: 706.78 ms (2897.64 tok/s) + Decode: 2045.58 ms (62.57 tok/s) + MLX peak memory: 4.52 GB +rocm_gpu_guard: 2026-10-09T02:10:05+0900 monitor: 5 samples +rocm_gpu_guard: 2026-10-09T02:10:05+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp2048_r2_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp2048_r2_new.log new file mode 100644 index 000000000..d77e36575 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp2048_r2_new.log @@ -0,0 +1,24 @@ +rocm_gpu_guard: 2026-10-09T02:08:49+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:08:49+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:09:04+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:09:04+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/gemma-3-4b-it-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +Warning: model.safetensors.index.json in models/mlx/gemma-3-4b-it-4bit references shards that don't match the on-disk files (likely a repackaged mlx-community quant). Falling back to all *.safetensors files in the directory. +[Load] wall: 0.649 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=None, reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 2.80 GB, cache 1.24 GB, active+cache 4.04 GB, phase peak 4.27 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=None, reserved 128 for generation) +[Memory] after measured pass: active 2.80 GB, cache 1.42 GB, active+cache 4.22 GB, phase peak 4.52 GB +[Profile Results] + Prompt tokens: 2048 + Generated tokens: 128 + Prefill: 709.08 ms (2888.25 tok/s) + Decode: 2074.55 ms (61.70 tok/s) + Decode path: engine + MLX peak memory: 4.52 GB +rocm_gpu_guard: 2026-10-09T02:09:11+0900 monitor: 5 samples +rocm_gpu_guard: 2026-10-09T02:09:11+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp2048_r3_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp2048_r3_base-null.log new file mode 100644 index 000000000..76c7e1a9e --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp2048_r3_base-null.log @@ -0,0 +1,23 @@ +rocm_gpu_guard: 2026-10-09T02:28:42+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:28:42+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:28:57+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:28:57+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/gemma-3-4b-it-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +Warning: model.safetensors.index.json in models/mlx/gemma-3-4b-it-4bit references shards that don't match the on-disk files (likely a repackaged mlx-community quant). Falling back to all *.safetensors files in the directory. +[Load] wall: 0.648 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=None, reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 2.80 GB, cache 2.47 GB, active+cache 5.27 GB, phase peak 4.27 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=None, reserved 128 for generation) +[Memory] after measured pass: active 2.97 GB, cache 1.21 GB, active+cache 4.18 GB, phase peak 4.52 GB +[Profile Results] + Prompt tokens: 2048 + Generated tokens: 128 + Prefill: 703.00 ms (2913.23 tok/s) + Decode: 2043.34 ms (62.64 tok/s) + MLX peak memory: 4.52 GB +rocm_gpu_guard: 2026-10-09T02:29:04+0900 monitor: 5 samples +rocm_gpu_guard: 2026-10-09T02:29:04+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp2048_r3_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp2048_r3_base.log new file mode 100644 index 000000000..9fd239a73 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp2048_r3_base.log @@ -0,0 +1,23 @@ +rocm_gpu_guard: 2026-10-09T02:29:09+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:29:09+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:29:24+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:29:24+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/gemma-3-4b-it-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +Warning: model.safetensors.index.json in models/mlx/gemma-3-4b-it-4bit references shards that don't match the on-disk files (likely a repackaged mlx-community quant). Falling back to all *.safetensors files in the directory. +[Load] wall: 0.648 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=None, reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 2.80 GB, cache 2.47 GB, active+cache 5.28 GB, phase peak 4.27 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=None, reserved 128 for generation) +[Memory] after measured pass: active 2.97 GB, cache 1.21 GB, active+cache 4.18 GB, phase peak 4.52 GB +[Profile Results] + Prompt tokens: 2048 + Generated tokens: 128 + Prefill: 706.11 ms (2900.41 tok/s) + Decode: 2042.94 ms (62.65 tok/s) + MLX peak memory: 4.52 GB +rocm_gpu_guard: 2026-10-09T02:29:31+0900 monitor: 5 samples +rocm_gpu_guard: 2026-10-09T02:29:31+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp2048_r3_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp2048_r3_new.log new file mode 100644 index 000000000..5bed9a721 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp2048_r3_new.log @@ -0,0 +1,24 @@ +rocm_gpu_guard: 2026-10-09T02:29:36+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:29:36+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:29:51+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:29:51+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/gemma-3-4b-it-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 2048 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +Warning: model.safetensors.index.json in models/mlx/gemma-3-4b-it-4bit references shards that don't match the on-disk files (likely a repackaged mlx-community quant). Falling back to all *.safetensors files in the directory. +[Load] wall: 0.641 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=None, reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 2.80 GB, cache 1.24 GB, active+cache 4.04 GB, phase peak 4.27 GB +[long-prompt] target=2048 tokens -> using 2048 tokens (max_context=None, reserved 128 for generation) +[Memory] after measured pass: active 2.80 GB, cache 1.25 GB, active+cache 4.05 GB, phase peak 4.52 GB +[Profile Results] + Prompt tokens: 2048 + Generated tokens: 128 + Prefill: 701.77 ms (2918.35 tok/s) + Decode: 2060.93 ms (62.11 tok/s) + Decode path: engine + MLX peak memory: 4.52 GB +rocm_gpu_guard: 2026-10-09T02:29:58+0900 monitor: 5 samples +rocm_gpu_guard: 2026-10-09T02:29:58+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp512_r1_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp512_r1_base-null.log new file mode 100644 index 000000000..69eba7246 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp512_r1_base-null.log @@ -0,0 +1,23 @@ +rocm_gpu_guard: 2026-10-09T01:48:24+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T01:48:24+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T01:48:39+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T01:48:39+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/gemma-3-4b-it-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +Warning: model.safetensors.index.json in models/mlx/gemma-3-4b-it-4bit references shards that don't match the on-disk files (likely a repackaged mlx-community quant). Falling back to all *.safetensors files in the directory. +[Load] wall: 0.623 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=None, reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 2.80 GB, cache 0.53 GB, active+cache 3.33 GB, phase peak 3.10 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=None, reserved 128 for generation) +[Memory] after measured pass: active 2.91 GB, cache 0.25 GB, active+cache 3.16 GB, phase peak 3.92 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 228.61 ms (2239.65 tok/s) + Decode: 1856.04 ms (68.96 tok/s) + MLX peak memory: 3.92 GB +rocm_gpu_guard: 2026-10-09T01:48:45+0900 monitor: 4 samples +rocm_gpu_guard: 2026-10-09T01:48:45+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp512_r1_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp512_r1_base.log new file mode 100644 index 000000000..5002b4b18 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp512_r1_base.log @@ -0,0 +1,23 @@ +rocm_gpu_guard: 2026-10-09T01:47:31+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T01:47:31+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T01:47:46+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T01:47:46+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/gemma-3-4b-it-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +Warning: model.safetensors.index.json in models/mlx/gemma-3-4b-it-4bit references shards that don't match the on-disk files (likely a repackaged mlx-community quant). Falling back to all *.safetensors files in the directory. +[Load] wall: 0.666 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=None, reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 2.80 GB, cache 0.52 GB, active+cache 3.33 GB, phase peak 3.09 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=None, reserved 128 for generation) +[Memory] after measured pass: active 2.91 GB, cache 0.24 GB, active+cache 3.15 GB, phase peak 3.92 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 480.68 ms (1065.16 tok/s) + Decode: 1853.53 ms (69.06 tok/s) + MLX peak memory: 3.92 GB +rocm_gpu_guard: 2026-10-09T01:47:53+0900 monitor: 5 samples +rocm_gpu_guard: 2026-10-09T01:47:53+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp512_r1_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp512_r1_new.log new file mode 100644 index 000000000..b5b637887 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp512_r1_new.log @@ -0,0 +1,24 @@ +rocm_gpu_guard: 2026-10-09T01:47:58+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T01:47:58+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T01:48:13+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T01:48:13+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/gemma-3-4b-it-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +Warning: model.safetensors.index.json in models/mlx/gemma-3-4b-it-4bit references shards that don't match the on-disk files (likely a repackaged mlx-community quant). Falling back to all *.safetensors files in the directory. +[Load] wall: 0.660 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=None, reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 2.80 GB, cache 0.38 GB, active+cache 3.18 GB, phase peak 3.10 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=None, reserved 128 for generation) +[Memory] after measured pass: active 2.80 GB, cache 0.37 GB, active+cache 3.18 GB, phase peak 3.92 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 231.93 ms (2207.57 tok/s) + Decode: 1852.54 ms (69.09 tok/s) + Decode path: engine + MLX peak memory: 3.92 GB +rocm_gpu_guard: 2026-10-09T01:48:19+0900 monitor: 4 samples +rocm_gpu_guard: 2026-10-09T01:48:19+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp512_r2_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp512_r2_base-null.log new file mode 100644 index 000000000..76a3c6c70 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp512_r2_base-null.log @@ -0,0 +1,23 @@ +rocm_gpu_guard: 2026-10-09T02:07:58+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:07:58+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:08:12+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:08:12+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/gemma-3-4b-it-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +Warning: model.safetensors.index.json in models/mlx/gemma-3-4b-it-4bit references shards that don't match the on-disk files (likely a repackaged mlx-community quant). Falling back to all *.safetensors files in the directory. +[Load] wall: 0.661 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=None, reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 2.80 GB, cache 0.53 GB, active+cache 3.33 GB, phase peak 3.09 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=None, reserved 128 for generation) +[Memory] after measured pass: active 2.91 GB, cache 0.28 GB, active+cache 3.19 GB, phase peak 3.92 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 236.92 ms (2161.07 tok/s) + Decode: 1852.86 ms (69.08 tok/s) + MLX peak memory: 3.92 GB +rocm_gpu_guard: 2026-10-09T02:08:18+0900 monitor: 4 samples +rocm_gpu_guard: 2026-10-09T02:08:18+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp512_r2_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp512_r2_base.log new file mode 100644 index 000000000..390c92703 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp512_r2_base.log @@ -0,0 +1,23 @@ +rocm_gpu_guard: 2026-10-09T02:08:23+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:08:23+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:08:38+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:08:38+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/gemma-3-4b-it-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +Warning: model.safetensors.index.json in models/mlx/gemma-3-4b-it-4bit references shards that don't match the on-disk files (likely a repackaged mlx-community quant). Falling back to all *.safetensors files in the directory. +[Load] wall: 0.662 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=None, reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 2.80 GB, cache 0.53 GB, active+cache 3.33 GB, phase peak 3.10 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=None, reserved 128 for generation) +[Memory] after measured pass: active 2.91 GB, cache 0.29 GB, active+cache 3.20 GB, phase peak 3.92 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 237.18 ms (2158.68 tok/s) + Decode: 1847.37 ms (69.29 tok/s) + MLX peak memory: 3.92 GB +rocm_gpu_guard: 2026-10-09T02:08:44+0900 monitor: 4 samples +rocm_gpu_guard: 2026-10-09T02:08:44+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp512_r2_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp512_r2_new.log new file mode 100644 index 000000000..43ebfe808 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp512_r2_new.log @@ -0,0 +1,24 @@ +rocm_gpu_guard: 2026-10-09T02:07:32+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:07:32+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:07:47+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:07:47+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/gemma-3-4b-it-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +Warning: model.safetensors.index.json in models/mlx/gemma-3-4b-it-4bit references shards that don't match the on-disk files (likely a repackaged mlx-community quant). Falling back to all *.safetensors files in the directory. +[Load] wall: 0.658 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=None, reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 2.80 GB, cache 0.34 GB, active+cache 3.14 GB, phase peak 3.09 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=None, reserved 128 for generation) +[Memory] after measured pass: active 2.80 GB, cache 0.39 GB, active+cache 3.20 GB, phase peak 3.92 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 234.46 ms (2183.76 tok/s) + Decode: 1857.67 ms (68.90 tok/s) + Decode path: engine + MLX peak memory: 3.92 GB +rocm_gpu_guard: 2026-10-09T02:07:53+0900 monitor: 4 samples +rocm_gpu_guard: 2026-10-09T02:07:53+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp512_r3_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp512_r3_base-null.log new file mode 100644 index 000000000..6fb346d2a --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp512_r3_base-null.log @@ -0,0 +1,23 @@ +rocm_gpu_guard: 2026-10-09T02:27:25+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:27:25+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:27:40+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:27:40+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/gemma-3-4b-it-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +Warning: model.safetensors.index.json in models/mlx/gemma-3-4b-it-4bit references shards that don't match the on-disk files (likely a repackaged mlx-community quant). Falling back to all *.safetensors files in the directory. +[Load] wall: 0.671 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=None, reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 2.80 GB, cache 0.53 GB, active+cache 3.33 GB, phase peak 3.10 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=None, reserved 128 for generation) +[Memory] after measured pass: active 2.91 GB, cache 0.25 GB, active+cache 3.16 GB, phase peak 3.92 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 229.13 ms (2234.52 tok/s) + Decode: 1834.46 ms (69.78 tok/s) + MLX peak memory: 3.92 GB +rocm_gpu_guard: 2026-10-09T02:27:46+0900 monitor: 4 samples +rocm_gpu_guard: 2026-10-09T02:27:46+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp512_r3_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp512_r3_base.log new file mode 100644 index 000000000..29eb6b311 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp512_r3_base.log @@ -0,0 +1,23 @@ +rocm_gpu_guard: 2026-10-09T02:27:51+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:27:51+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:28:05+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:28:05+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/gemma-3-4b-it-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +Warning: model.safetensors.index.json in models/mlx/gemma-3-4b-it-4bit references shards that don't match the on-disk files (likely a repackaged mlx-community quant). Falling back to all *.safetensors files in the directory. +[Load] wall: 0.648 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=None, reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 2.80 GB, cache 0.53 GB, active+cache 3.33 GB, phase peak 3.10 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=None, reserved 128 for generation) +[Memory] after measured pass: active 2.91 GB, cache 0.29 GB, active+cache 3.20 GB, phase peak 3.92 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 237.50 ms (2155.78 tok/s) + Decode: 1849.93 ms (69.19 tok/s) + MLX peak memory: 3.92 GB +rocm_gpu_guard: 2026-10-09T02:28:11+0900 monitor: 4 samples +rocm_gpu_guard: 2026-10-09T02:28:11+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp512_r3_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp512_r3_new.log new file mode 100644 index 000000000..6e00f6967 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gemma-3-4b-it-4bit_pp512_r3_new.log @@ -0,0 +1,24 @@ +rocm_gpu_guard: 2026-10-09T02:28:16+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:28:16+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:28:31+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:28:31+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/gemma-3-4b-it-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +Warning: model.safetensors.index.json in models/mlx/gemma-3-4b-it-4bit references shards that don't match the on-disk files (likely a repackaged mlx-community quant). Falling back to all *.safetensors files in the directory. +[Load] wall: 0.638 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=None, reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 2.80 GB, cache 0.38 GB, active+cache 3.18 GB, phase peak 3.09 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=None, reserved 128 for generation) +[Memory] after measured pass: active 2.80 GB, cache 0.39 GB, active+cache 3.20 GB, phase peak 3.92 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 229.49 ms (2231.04 tok/s) + Decode: 1865.00 ms (68.63 tok/s) + Decode path: engine + MLX peak memory: 3.92 GB +rocm_gpu_guard: 2026-10-09T02:28:37+0900 monitor: 4 samples +rocm_gpu_guard: 2026-10-09T02:28:37+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gpt-oss-20b-MXFP4-Q4_pp512_r1_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gpt-oss-20b-MXFP4-Q4_pp512_r1_base-null.log new file mode 100644 index 000000000..3c51ef697 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gpt-oss-20b-MXFP4-Q4_pp512_r1_base-null.log @@ -0,0 +1,23 @@ +rocm_gpu_guard: 2026-10-09T01:59:12+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T01:59:12+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T01:59:26+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T01:59:26+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/gpt-oss-20b-MXFP4-Q4 -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +Quantization mode mxfp4: running on native ROCm kernels, no load-time conversion (backend quantization capability table) +[Load] wall: 0.489 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 11.26 GB, cache 1.90 GB, active+cache 13.16 GB, phase peak 13.67 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[Memory] after measured pass: active 11.28 GB, cache 0.27 GB, active+cache 11.55 GB, phase peak 13.84 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 968.89 ms (528.44 tok/s) + Decode: 15015.86 ms (8.52 tok/s) + MLX peak memory: 13.84 GB +rocm_gpu_guard: 2026-10-09T01:59:48+0900 monitor: 14 samples +rocm_gpu_guard: 2026-10-09T01:59:48+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gpt-oss-20b-MXFP4-Q4_pp512_r1_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gpt-oss-20b-MXFP4-Q4_pp512_r1_base.log new file mode 100644 index 000000000..56373792d --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gpt-oss-20b-MXFP4-Q4_pp512_r1_base.log @@ -0,0 +1,23 @@ +rocm_gpu_guard: 2026-10-09T01:57:31+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T01:57:31+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T01:57:46+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T01:57:46+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/gpt-oss-20b-MXFP4-Q4 -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +Quantization mode mxfp4: running on native ROCm kernels, no load-time conversion (backend quantization capability table) +[Load] wall: 0.570 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 11.26 GB, cache 1.82 GB, active+cache 13.08 GB, phase peak 12.88 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[Memory] after measured pass: active 11.28 GB, cache 0.27 GB, active+cache 11.55 GB, phase peak 13.84 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 969.43 ms (528.15 tok/s) + Decode: 14985.87 ms (8.54 tok/s) + MLX peak memory: 13.84 GB +rocm_gpu_guard: 2026-10-09T01:58:22+0900 monitor: 24 samples +rocm_gpu_guard: 2026-10-09T01:58:22+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gpt-oss-20b-MXFP4-Q4_pp512_r1_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gpt-oss-20b-MXFP4-Q4_pp512_r1_new.log new file mode 100644 index 000000000..1d75c1fda --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gpt-oss-20b-MXFP4-Q4_pp512_r1_new.log @@ -0,0 +1,24 @@ +rocm_gpu_guard: 2026-10-09T01:58:27+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T01:58:27+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T01:58:42+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T01:58:42+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/gpt-oss-20b-MXFP4-Q4 -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +Quantization mode mxfp4: running on native ROCm kernels, no load-time conversion (backend quantization capability table) +[Load] wall: 0.566 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 11.28 GB, cache 0.27 GB, active+cache 11.55 GB, phase peak 13.67 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[Memory] after measured pass: active 11.30 GB, cache 0.28 GB, active+cache 11.57 GB, phase peak 13.60 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 1004.76 ms (509.57 tok/s) + Decode: 15016.75 ms (8.52 tok/s) + Decode path: engine + MLX peak memory: 13.67 GB +rocm_gpu_guard: 2026-10-09T01:59:07+0900 monitor: 16 samples +rocm_gpu_guard: 2026-10-09T01:59:07+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gpt-oss-20b-MXFP4-Q4_pp512_r2_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gpt-oss-20b-MXFP4-Q4_pp512_r2_base-null.log new file mode 100644 index 000000000..aba3ec3cd --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gpt-oss-20b-MXFP4-Q4_pp512_r2_base-null.log @@ -0,0 +1,23 @@ +rocm_gpu_guard: 2026-10-09T02:18:19+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:18:19+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:18:33+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:18:33+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/gpt-oss-20b-MXFP4-Q4 -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +Quantization mode mxfp4: running on native ROCm kernels, no load-time conversion (backend quantization capability table) +[Load] wall: 0.555 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 11.26 GB, cache 1.90 GB, active+cache 13.16 GB, phase peak 13.67 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[Memory] after measured pass: active 11.28 GB, cache 0.27 GB, active+cache 11.55 GB, phase peak 13.84 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 973.88 ms (525.73 tok/s) + Decode: 15085.56 ms (8.48 tok/s) + MLX peak memory: 13.84 GB +rocm_gpu_guard: 2026-10-09T02:18:58+0900 monitor: 16 samples +rocm_gpu_guard: 2026-10-09T02:18:58+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gpt-oss-20b-MXFP4-Q4_pp512_r2_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gpt-oss-20b-MXFP4-Q4_pp512_r2_base.log new file mode 100644 index 000000000..111adbc8a --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gpt-oss-20b-MXFP4-Q4_pp512_r2_base.log @@ -0,0 +1,23 @@ +rocm_gpu_guard: 2026-10-09T02:19:03+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:19:03+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:19:17+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:19:17+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/gpt-oss-20b-MXFP4-Q4 -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +Quantization mode mxfp4: running on native ROCm kernels, no load-time conversion (backend quantization capability table) +[Load] wall: 0.502 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 11.26 GB, cache 1.90 GB, active+cache 13.16 GB, phase peak 13.67 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[Memory] after measured pass: active 11.28 GB, cache 0.27 GB, active+cache 11.55 GB, phase peak 13.84 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 971.68 ms (526.92 tok/s) + Decode: 15123.50 ms (8.46 tok/s) + MLX peak memory: 13.84 GB +rocm_gpu_guard: 2026-10-09T02:19:40+0900 monitor: 15 samples +rocm_gpu_guard: 2026-10-09T02:19:40+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gpt-oss-20b-MXFP4-Q4_pp512_r2_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gpt-oss-20b-MXFP4-Q4_pp512_r2_new.log new file mode 100644 index 000000000..c3d0c9d91 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gpt-oss-20b-MXFP4-Q4_pp512_r2_new.log @@ -0,0 +1,24 @@ +rocm_gpu_guard: 2026-10-09T02:17:21+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:17:21+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:17:36+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:17:36+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/gpt-oss-20b-MXFP4-Q4 -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +Quantization mode mxfp4: running on native ROCm kernels, no load-time conversion (backend quantization capability table) +[Load] wall: 0.579 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 11.28 GB, cache 0.27 GB, active+cache 11.55 GB, phase peak 12.88 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[Memory] after measured pass: active 11.30 GB, cache 0.28 GB, active+cache 11.57 GB, phase peak 13.60 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 1007.14 ms (508.37 tok/s) + Decode: 15144.73 ms (8.45 tok/s) + Decode path: engine + MLX peak memory: 13.60 GB +rocm_gpu_guard: 2026-10-09T02:18:14+0900 monitor: 25 samples +rocm_gpu_guard: 2026-10-09T02:18:14+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gpt-oss-20b-MXFP4-Q4_pp512_r3_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gpt-oss-20b-MXFP4-Q4_pp512_r3_base-null.log new file mode 100644 index 000000000..4e6a5e281 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gpt-oss-20b-MXFP4-Q4_pp512_r3_base-null.log @@ -0,0 +1,23 @@ +rocm_gpu_guard: 2026-10-09T02:37:06+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:37:06+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:37:21+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:37:21+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/gpt-oss-20b-MXFP4-Q4 -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +Quantization mode mxfp4: running on native ROCm kernels, no load-time conversion (backend quantization capability table) +[Load] wall: 0.615 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 11.26 GB, cache 1.83 GB, active+cache 13.09 GB, phase peak 12.88 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[Memory] after measured pass: active 11.28 GB, cache 0.27 GB, active+cache 11.55 GB, phase peak 13.84 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 971.76 ms (526.88 tok/s) + Decode: 15159.36 ms (8.44 tok/s) + MLX peak memory: 13.84 GB +rocm_gpu_guard: 2026-10-09T02:37:58+0900 monitor: 25 samples +rocm_gpu_guard: 2026-10-09T02:37:58+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gpt-oss-20b-MXFP4-Q4_pp512_r3_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gpt-oss-20b-MXFP4-Q4_pp512_r3_base.log new file mode 100644 index 000000000..315b97ec5 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gpt-oss-20b-MXFP4-Q4_pp512_r3_base.log @@ -0,0 +1,23 @@ +rocm_gpu_guard: 2026-10-09T02:38:03+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:38:03+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:38:18+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:38:18+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/gpt-oss-20b-MXFP4-Q4 -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +Quantization mode mxfp4: running on native ROCm kernels, no load-time conversion (backend quantization capability table) +[Load] wall: 0.546 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 11.26 GB, cache 1.90 GB, active+cache 13.16 GB, phase peak 13.67 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[Memory] after measured pass: active 11.28 GB, cache 0.27 GB, active+cache 11.55 GB, phase peak 13.84 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 1097.23 ms (466.63 tok/s) + Decode: 15250.46 ms (8.39 tok/s) + MLX peak memory: 13.84 GB +rocm_gpu_guard: 2026-10-09T02:38:43+0900 monitor: 17 samples +rocm_gpu_guard: 2026-10-09T02:38:43+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gpt-oss-20b-MXFP4-Q4_pp512_r3_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gpt-oss-20b-MXFP4-Q4_pp512_r3_new.log new file mode 100644 index 000000000..ad6ea9f63 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/gpt-oss-20b-MXFP4-Q4_pp512_r3_new.log @@ -0,0 +1,24 @@ +rocm_gpu_guard: 2026-10-09T02:38:48+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:38:48+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:39:03+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:39:03+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/gpt-oss-20b-MXFP4-Q4 -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +Quantization mode mxfp4: running on native ROCm kernels, no load-time conversion (backend quantization capability table) +[Load] wall: 0.499 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 11.28 GB, cache 0.27 GB, active+cache 11.55 GB, phase peak 13.67 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[Memory] after measured pass: active 11.30 GB, cache 0.28 GB, active+cache 11.57 GB, phase peak 13.60 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 1002.98 ms (510.48 tok/s) + Decode: 15075.33 ms (8.49 tok/s) + Decode path: engine + MLX peak memory: 13.67 GB +rocm_gpu_guard: 2026-10-09T02:39:25+0900 monitor: 15 samples +rocm_gpu_guard: 2026-10-09T02:39:25+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/granite-4.0-h-tiny-4bit_pp512_r1_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/granite-4.0-h-tiny-4bit_pp512_r1_base-null.log new file mode 100644 index 000000000..c46dd80b3 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/granite-4.0-h-tiny-4bit_pp512_r1_base-null.log @@ -0,0 +1,22 @@ +rocm_gpu_guard: 2026-10-09T01:55:19+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T01:55:19+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T01:55:34+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T01:55:34+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/granite-4.0-h-tiny-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.118 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 3.96 GB, cache 3.76 GB, active+cache 7.71 GB, phase peak 7.22 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[Memory] after measured pass: active 4.02 GB, cache 1.78 GB, active+cache 5.80 GB, phase peak 7.31 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 865.72 ms (591.41 tok/s) + Decode: 1435.18 ms (89.19 tok/s) + MLX peak memory: 7.31 GB +rocm_gpu_guard: 2026-10-09T01:55:40+0900 monitor: 4 samples +rocm_gpu_guard: 2026-10-09T01:55:40+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/granite-4.0-h-tiny-4bit_pp512_r1_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/granite-4.0-h-tiny-4bit_pp512_r1_base.log new file mode 100644 index 000000000..726c5040c --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/granite-4.0-h-tiny-4bit_pp512_r1_base.log @@ -0,0 +1,22 @@ +rocm_gpu_guard: 2026-10-09T01:54:26+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T01:54:26+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T01:54:40+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T01:54:40+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/granite-4.0-h-tiny-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.137 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 3.96 GB, cache 1.85 GB, active+cache 5.81 GB, phase peak 6.61 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[Memory] after measured pass: active 4.02 GB, cache 1.78 GB, active+cache 5.80 GB, phase peak 7.31 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 1141.73 ms (448.44 tok/s) + Decode: 1426.12 ms (89.75 tok/s) + MLX peak memory: 7.31 GB +rocm_gpu_guard: 2026-10-09T01:54:49+0900 monitor: 6 samples +rocm_gpu_guard: 2026-10-09T01:54:49+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/granite-4.0-h-tiny-4bit_pp512_r1_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/granite-4.0-h-tiny-4bit_pp512_r1_new.log new file mode 100644 index 000000000..059b91b1d --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/granite-4.0-h-tiny-4bit_pp512_r1_new.log @@ -0,0 +1,23 @@ +rocm_gpu_guard: 2026-10-09T01:54:54+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T01:54:54+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T01:55:08+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T01:55:08+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/granite-4.0-h-tiny-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.122 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 3.96 GB, cache 1.85 GB, active+cache 5.81 GB, phase peak 7.22 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[Memory] after measured pass: active 3.96 GB, cache 1.85 GB, active+cache 5.81 GB, phase peak 7.31 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 869.89 ms (588.58 tok/s) + Decode: 1444.81 ms (88.59 tok/s) + Decode path: engine + MLX peak memory: 7.31 GB +rocm_gpu_guard: 2026-10-09T01:55:14+0900 monitor: 4 samples +rocm_gpu_guard: 2026-10-09T01:55:14+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/granite-4.0-h-tiny-4bit_pp512_r2_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/granite-4.0-h-tiny-4bit_pp512_r2_base-null.log new file mode 100644 index 000000000..4b892dd4a --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/granite-4.0-h-tiny-4bit_pp512_r2_base-null.log @@ -0,0 +1,22 @@ +rocm_gpu_guard: 2026-10-09T02:14:45+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:14:45+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:15:00+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:15:00+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/granite-4.0-h-tiny-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.126 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 3.96 GB, cache 3.76 GB, active+cache 7.71 GB, phase peak 7.22 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[Memory] after measured pass: active 4.02 GB, cache 1.78 GB, active+cache 5.80 GB, phase peak 7.31 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 894.81 ms (572.19 tok/s) + Decode: 1429.58 ms (89.54 tok/s) + MLX peak memory: 7.31 GB +rocm_gpu_guard: 2026-10-09T02:15:06+0900 monitor: 4 samples +rocm_gpu_guard: 2026-10-09T02:15:06+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/granite-4.0-h-tiny-4bit_pp512_r2_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/granite-4.0-h-tiny-4bit_pp512_r2_base.log new file mode 100644 index 000000000..3fcd771ef --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/granite-4.0-h-tiny-4bit_pp512_r2_base.log @@ -0,0 +1,22 @@ +rocm_gpu_guard: 2026-10-09T02:15:11+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:15:11+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:15:25+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:15:25+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/granite-4.0-h-tiny-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.121 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 3.96 GB, cache 3.76 GB, active+cache 7.71 GB, phase peak 7.22 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[Memory] after measured pass: active 4.02 GB, cache 1.78 GB, active+cache 5.80 GB, phase peak 7.31 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 894.08 ms (572.66 tok/s) + Decode: 1429.87 ms (89.52 tok/s) + MLX peak memory: 7.31 GB +rocm_gpu_guard: 2026-10-09T02:15:31+0900 monitor: 4 samples +rocm_gpu_guard: 2026-10-09T02:15:31+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/granite-4.0-h-tiny-4bit_pp512_r2_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/granite-4.0-h-tiny-4bit_pp512_r2_new.log new file mode 100644 index 000000000..7f9a607db --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/granite-4.0-h-tiny-4bit_pp512_r2_new.log @@ -0,0 +1,23 @@ +rocm_gpu_guard: 2026-10-09T02:14:18+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:14:18+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:14:33+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:14:33+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/granite-4.0-h-tiny-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.131 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 3.96 GB, cache 1.85 GB, active+cache 5.81 GB, phase peak 6.61 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[Memory] after measured pass: active 3.96 GB, cache 1.84 GB, active+cache 5.80 GB, phase peak 7.31 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 1128.64 ms (453.64 tok/s) + Decode: 1438.29 ms (88.99 tok/s) + Decode path: engine + MLX peak memory: 7.31 GB +rocm_gpu_guard: 2026-10-09T02:14:40+0900 monitor: 5 samples +rocm_gpu_guard: 2026-10-09T02:14:40+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/granite-4.0-h-tiny-4bit_pp512_r3_base-null.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/granite-4.0-h-tiny-4bit_pp512_r3_base-null.log new file mode 100644 index 000000000..d82645bb6 --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/granite-4.0-h-tiny-4bit_pp512_r3_base-null.log @@ -0,0 +1,22 @@ +rocm_gpu_guard: 2026-10-09T02:34:02+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:34:02+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:34:17+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:34:17+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/granite-4.0-h-tiny-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.131 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 3.96 GB, cache 1.85 GB, active+cache 5.81 GB, phase peak 6.61 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[Memory] after measured pass: active 4.02 GB, cache 1.78 GB, active+cache 5.80 GB, phase peak 7.31 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 1331.37 ms (384.57 tok/s) + Decode: 1427.41 ms (89.67 tok/s) + MLX peak memory: 7.31 GB +rocm_gpu_guard: 2026-10-09T02:34:25+0900 monitor: 6 samples +rocm_gpu_guard: 2026-10-09T02:34:25+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/granite-4.0-h-tiny-4bit_pp512_r3_base.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/granite-4.0-h-tiny-4bit_pp512_r3_base.log new file mode 100644 index 000000000..c855ae2bb --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/granite-4.0-h-tiny-4bit_pp512_r3_base.log @@ -0,0 +1,22 @@ +rocm_gpu_guard: 2026-10-09T02:34:30+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:34:30+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:34:45+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:34:45+0900 attempt 1/10: start: /tmp/claude-1001/-home-devops-Development-mlxcel/66a03ced-36c6-4e0a-9e39-155a3eb57353/scratchpad/issue-2192/pre-epic/target/release/mlxcel-bench-decode -m models/mlx/granite-4.0-h-tiny-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.123 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 3.96 GB, cache 3.20 GB, active+cache 7.16 GB, phase peak 7.22 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[Memory] after measured pass: active 4.02 GB, cache 1.78 GB, active+cache 5.80 GB, phase peak 7.31 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 868.14 ms (589.77 tok/s) + Decode: 1431.48 ms (89.42 tok/s) + MLX peak memory: 7.31 GB +rocm_gpu_guard: 2026-10-09T02:34:51+0900 monitor: 4 samples +rocm_gpu_guard: 2026-10-09T02:34:51+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/granite-4.0-h-tiny-4bit_pp512_r3_new.log b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/granite-4.0-h-tiny-4bit_pp512_r3_new.log new file mode 100644 index 000000000..ab486a6fb --- /dev/null +++ b/docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/rounds/raw/granite-4.0-h-tiny-4bit_pp512_r3_new.log @@ -0,0 +1,23 @@ +rocm_gpu_guard: 2026-10-09T02:34:56+0900 lock held by outer guard 2263193: not taking /tmp/mlxcel-rocm-gpu-guard.lock +rocm_gpu_guard: 2026-10-09T02:34:56+0900 waiting for 10s of idle GPU and no compiler +rocm_gpu_guard: 2026-10-09T02:35:10+0900 idle for 10s: kfd proc empty and no compiler in every 1 Hz sample +rocm_gpu_guard: 2026-10-09T02:35:10+0900 attempt 1/10: start: /home/devops/Development/wt-20261006T1212-issue-2192/target/release/mlxcel-bench-decode -m models/mlx/granite-4.0-h-tiny-4bit -p x -n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens 512 +[mlx-rocm] bound HIP device 0: gfx1151 (AMD Radeon 8060S Graphics) cus=20 warp=32 lds=64KB +[Memory] cache limit: 2.15 GB +[Load] wall: 0.121 s MLX peak: 0.00 GB +[Memory] after load: active 0.00 GB, cache 0.00 GB, active+cache 0.00 GB, phase peak 0.00 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[mlx-rocm] gfx1151 native_wmma=1 cus=20 warp=32 low_cu=0 host_WARP_SIZE=32 +[hipBLASLt caps] device 0 (gfx1151): bf16=1 fp8_e4m3=0 fp8_e5m2=0 int8=1 +[Memory] after warmup (prefill + warmup tokens): active 3.96 GB, cache 1.85 GB, active+cache 5.81 GB, phase peak 7.22 GB +[long-prompt] target=512 tokens -> using 512 tokens (max_context=Some(131072), reserved 128 for generation) +[Memory] after measured pass: active 3.96 GB, cache 1.84 GB, active+cache 5.80 GB, phase peak 7.31 GB +[Profile Results] + Prompt tokens: 512 + Generated tokens: 128 + Prefill: 855.19 ms (598.70 tok/s) + Decode: 1428.92 ms (89.58 tok/s) + Decode path: engine + MLX peak memory: 7.31 GB +rocm_gpu_guard: 2026-10-09T02:35:16+0900 monitor: 4 samples +rocm_gpu_guard: 2026-10-09T02:35:16+0900 attempt 1: CLEAN, exit 0 diff --git a/docs/benchmark_results/rocm-unified-engine-gfx1151-2026-10-08.md b/docs/benchmark_results/rocm-unified-engine-gfx1151-2026-10-08.md index f54f96dfa..6ea0d56d1 100644 --- a/docs/benchmark_results/rocm-unified-engine-gfx1151-2026-10-08.md +++ b/docs/benchmark_results/rocm-unified-engine-gfx1151-2026-10-08.md @@ -126,7 +126,7 @@ ADR 0007's ROCm paragraph: the CLI and the server engine with dense storage must ### Method -`data/.../harness/bench_rounds.sh`: for each (model, prompt length) cell, three interleaved rounds of three arms, each a fresh `mlxcel-bench-decode` process at `scripts/bench_decode.sh`'s shape (`-n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens N`, greedy): `base` is the pre-epic binary (`4c44e317`, `CxxGenerator`), `new` is `main` (`DirectEngine`, `Decode path: engine`), and `base-null` is the pre-epic binary again, the method's noise floor. The arm order rotates each round. Every run was its own `scripts/rocm_gpu_guard.sh` command inside one host-wide lock hold (`data/.../harness/full_session.sh`): 10 s of idle GPU and no compiler before each run, a 1 Hz monitor during it, and all 99 runs were CLEAN on the first attempt (`data/.../guard/guard-logs.tar.gz`). The session ran 2026-10-09 01:19 to 02:43 KST. `data/.../harness/summarize_rounds.py` computes the medians and the paired per-round deltas below (median and range over the three rounds, percent of `base`). The threshold is ADR 0007's 1.0 percent on the paired decode delta. +`data/.../harness/bench_rounds.sh`: for each (model, prompt length) cell, three interleaved rounds of three arms, each a fresh `mlxcel-bench-decode` process at `scripts/bench_decode.sh`'s shape (`-n 128 --warmup-tokens 20 --ignore-eos --prompt-tokens N`, greedy): `base` is the pre-epic binary (`4c44e317`, `CxxGenerator`), `new` is `main` (`DirectEngine`, `Decode path: engine`), and `base-null` is the pre-epic binary again, the method's noise floor. The arm order rotates each round. Every run was its own `scripts/rocm_gpu_guard.sh` command inside one host-wide lock hold (`data/.../harness/full_session.sh`): 10 s of idle GPU and no compiler before each run, a 1 Hz monitor during it, and all 99 runs were CLEAN on the first attempt (`data/.../guard/logs/`). The session ran 2026-10-09 01:19 to 02:43 KST. `data/.../harness/summarize_rounds.py` computes the medians and the paired per-round deltas below (median and range over the three rounds, percent of `base`). The threshold is ADR 0007's 1.0 percent on the paired decode delta. ### Decode, `mlxcel generate` (DirectEngine) vs `CxxGenerator`