InsnAlign: Instruction Alignment for Binary Code Representation Learning

InsnAlign builds instruction-level embeddings on top of binary-function encoders (jTrans and CLAP) and uses them to (1) recover source-level correspondences between functions compiled with different compilers / optimization levels, and (2) synergize function-level retrieval with instruction-level alignment scores (MAS reranking) to improve binary code similarity analysis.

This README focuses on artifact evaluation: how to reproduce the main results (Tables 2-5) and the case-study figure (Figure 9) with the provided scripts, models, and data.

To reproduce our results, please download the released data and models from https://doi.org/10.5281/zenodo.21696749

---

Repository Layout

src/align/                  Core implementation
  model.py                  BinBertAlignModel (jTrans BERT + instruction pooler)
  evaluate.py               MRR / R@K evaluation, MAS synergy reranking
  eval_mrr.py               CLI: function-level, instruction-level, and synergy eval
  eval_mrr_clap.py          Same CLI for CLAP-based models
  dataset.py, tokenizer.py  jTrans data loading and tokenization
  dataset_clap.py           CLAP data loading
  train*.py                 Training scripts (not needed for evaluation)
  explainability/           MAS, AUC-ROC, Cohen's d (discriminability experiments)
  view_align/               Interactive alignment viewer (Figure 9 case study)

shell/                      End-to-end evaluation scripts
  jtrans-eval-synergy.sh    Synergy eval for jTrans models  -> output-synergy/*.json
  clap-eval-synergy.sh      Synergy eval for CLAP models    -> output-synergy/*.json
  table3.sh                 Discriminability eval (Table 3) -> output-exp/rand-<seed>/*.json

output-synergy/             JSON results from the synergy evaluation
  show_result.py            Renders baseline / synergy / delta tables from a JSON file

output-exp/                 Discriminability results for Table 3
  rand-<seed>/              Per-seed JSON outputs (one per model x sampling mode)
  draw_table.py             Averages metrics across seeds and renders Table 3

models/                     Trained model checkpoints
  jtrans-align/             jTrans + instruction alignment (used for the case study)
  jtrans-finetune/          jTrans fine-tuned baseline
  clap-align/               CLAP + alignment head
  clap-beta-000/            CLAP baseline (beta = 0)
  clap-align-hard-neg-beta-002/  CLAP + alignment, hard negatives, beta = 0.02

data/
  binkit/                   Function-level paired test set (BinKit), e.g. clean-test2.pkl
  patch_pkl/coreutils/      Raw per-instruction pool pickles for the case study
                            (coreutils 9.2 / 9.5, gcc-11.4, O0 / O3)

jTrans/                     jTrans tokenizer/vocab (required by the jTrans models)
  jtrans_tokenizer/vocab.txt

---

Environment Setup

Python 3.10+ and PyTorch (CUDA recommended) are required.

# (recommended) create a virtual environment
python -m venv venv && source venv/bin/activate

# install PyTorch with CUDA support first (adjust the index to your CUDA version)
pip install torch --index-url https://download.pytorch.org/whl/cu121

# install the remaining dependencies
pip install -r requirements.txt

Notes:

• The jTrans tokenizer vocab is loaded from jTrans/jtrans_tokenizer/vocab.txt. This directory is git-ignored, so make sure it is present in the artifact bundle (or obtained from the upstream jTrans release) before running any jTrans-based evaluation.
• All commands below are run from the repository root.
• The evaluation scripts auto-select CUDA if available and otherwise fall back to CPU. The synergy evaluation encodes a pool of up to 10,000 functions per pair, so a GPU is strongly recommended to keep runtime reasonable.
• For smoke testing, adjust the poolsize value of scripts in ./shell (e.g., 100)

---

Reproducing the Main Results (Tables 2, 4, 5)

Tables 2, 4, and 5 report function-level retrieval accuracy over the 16 compiler-pair configurations (gcc-11.2.0, clang-13.0, gcc-4.9.4, clang-4.0 × O0/O3), before and after the synergy reranking.

Step 1 — Run the evaluation scripts

# CLAP-based models  -> output-synergy/clap-align.json, clap-beta-000.json, clap-align-hard-neg-beta-002.json
bash shell/clap-eval-synergy.sh

# jTrans-based models -> output-synergy/jtrans-align.json, jtrans-finetune.json
bash shell/jtrans-eval-synergy.sh

Each script evaluates its models in --mode synergy on data/binkit/clean-test2.pkl with K=100 and gamma=0.5, and dumps the per-pair results (baseline cosine metrics plus synergy metrics for the tested (K, gamma) grid) into output-synergy/.

Step 2 — Render the tables

# one invocation per result file
python output-synergy/show_result.py output-synergy/jtrans-align.json --K 100 --gamma 0.5
python output-synergy/show_result.py output-synergy/jtrans-finetune.json --K 100 --gamma 0.5
python output-synergy/show_result.py output-synergy/clap-align.json --K 100 --gamma 0.5

# Table 5 CLAP-based model with hard-negative training
python output-synergy/show_result.py output-synergy/clap-align-hard-neg-beta-002.json --K 100 --gamma 0.5

show_result.py prints three 4×4 tables for each model / metric:

Table	Contents
Table 2	Baseline R@1 — pure function-level cosine retrieval
Table 4	Synergy R@1 — cosine + MAS reranking at the given (K, gamma)
Table 5	CLAP-based model with hard-negative training

Note: the JSON is keyed by config string K{K}_g{gamma}. --K and --gamma must match the configs present in the file. After re-running the shell scripts (which use K=100, gamma=0.5), use --K 100 --gamma 0.5. You can also change K and gamma of shell scripts to dump more reranking settings such as using K='5 10 20 50 100' and g= '0.1 0.3 0.5 0.7 0.9 1.0'. The dump results will contain configs like K5_g0.5 / K50_g0.7, then you can pass arguments like --K 5 --gamma 0.5 accordingly.

---

Reproducing Table 3 (Discriminability)

Table 3 reports whether instruction-level alignment is a reliable discriminability signal: it evaluates how well the instruction-level MAS and the function-level cosine score each separate similar function pairs from dissimilar ones, on the gcc-11.2.0_O0 -> gcc-11.2.0_O3 variant pair, using AUC-ROC and Cohen's d as discrepancy metrics. Each model is tested under two negative-sampling modes:

• Top-5 — hard negatives: the anchor (O0) function is paired with the 5 nearest (by cosine) same-source O3 functions of other symbols, so the negative set consists of the most confusing distractors.
• Random — negatives sampled uniformly from different symbols.

Both metrics are computed for the MAS (instruction alignment) and the function cosine score, and the results are averaged over 4 random seeds.

Step 1 — Run the evaluation script

bash shell/table3.sh

The script evaluates four models — jtrans-finetune, jtrans-align, clap-beta-000, and clap-align — for each seed in 42 3 5 7, and writes one JSON per model x sampling-mode combination into output-exp/rand-<seed>/ (e.g. output-exp/rand-42/jtrans-align.json for Top-5 and output-exp/rand-42/rand-jtrans-align.json for Random). It uses max_pairs=30000 per mode, so a GPU is recommended; for smoke testing, lower the npairs variable in shell/table3.sh.

Step 2 — Render the table

python output-exp/draw_table.py

draw_table.py averages the four metrics (AUC MAS, AUC cos, Cohen's d MAS, Cohen's d cos) across the seeds in output-exp/rand-3/, rand-5/, rand-7/, and rand-42/, and prints one row per model x sampling-mode combination — this is the table reported in the paper.

---

Case Study: Alignment Viewer (Figure 9)

The case study uses the interactive alignment viewer to produce the vuln-by-vuln instruction-alignment figure. It loads two raw instruction pools (side A and side B), embeds every instruction with models/jtrans-align, and serves a local web UI (function list, ranked matches, instruction-level similarity heatmap, and per-function MRR / R@1 metrics).

Right side of Figure 9 (vuln-by-vuln across optimization levels)

python -m src.align.view_align \
    --poola data/patch_pkl/coreutils/9.2-gcc-11.4-O0/coreutils.pkl \
    --poolb data/patch_pkl/coreutils/9.2-gcc-11.4-O3/coreutils.pkl \
    --model-path models/jtrans-align \
    --vocab jTrans/jtrans_tokenizer/vocab.txt

After the server starts, open http://localhost:5000 (override with --port/--host). Select a vulnerable function on left, inspect its ranked matches on right, and click through instructions to see the similarity coloring, ground-truth dots, and the instruction-level metrics panel — this is the view captured in Figure 9.

At the middle part, click the ground-truth switch (i.e., remove the blue points).

To view the right subfigure, please type the function name line_bytes_split into the upper-left filtering, select the function, and the right side of the webpage will change accordingly. Line 812 and 813 are key instructions of the patch, and you can see they are matched correctly.

Left side of Figure 9 (vulnerable version comparison)

Same command but point --poolb at the 9.5 pool:

python -m src.align.view_align \
    --poola data/patch_pkl/coreutils/9.2-gcc-11.4-O0/coreutils.pkl \
    --poolb data/patch_pkl/coreutils/9.5-gcc-11.4-O3/coreutils.pkl \
    --model-path models/jtrans-align \
    --vocab jTrans/jtrans_tokenizer/vocab.txt

Please type the function name line_bytes_split into the upper-left filtering, select the function, and the right side of the webpage will change accordingly.

NOTE: Line 812 and 813 of the ver. 9.2 side will not have a high similarity counterpart in the ver. 9.5 binary. The ground-truth label in this figure is meaningless because the two coreutils are compiled with different versions.

---

Notes and Troubleshooting

• CLAP models: the CLAP tokenizer/model are loaded with trust_remote_code=True from the local model directory (models/clap-*), so no network access is needed.
• Reproducibility: --seed 42 is used by the evaluation scripts.
• If show_result.py prints N/A in the synergy tables, the requested (K, gamma) config is not in the JSON — adjust --K/--gamma to a key present in the file.

---

Please consider citing our paper

@misc{wang2026instructionalignmentbinarycode,
      title={Instruction Alignment for Binary Code Representation Learning}, 
      author={Huaijin Wang and Shuai Wang},
      year={2026},
      eprint={2608.11766},
      archivePrefix={arXiv},
      primaryClass={cs.SE},
      url={https://arxiv.org/abs/2608.11766}, 
}