The Value of Perplexity

[itifwork]

Hello, I noticed that you reported the NVFP4 perplexity value for Llama 3.1-8B, and it shows a slight difference (around one point) compared with the results from FourOverSix and FAAR. I'm curious about the reason for this difference. Could you please explain it?

[actypedef]

Hi @itifwork ,

Could you share the exact PPL results reproduced from these repos and the experimental environment? That would help us make a fair comparison and better understand the source of the difference.

[itifwork]

OK, I have attached the link to the paper and the result of the perplexity below. For FouroverSix, I have replicated the code in the paper and the results are basically similar to those of the perplexity in the table.

FourOverSix: https://arxiv.org/abs/2512.02010

FAAR: https://arxiv.org/abs/2603.22370

[actypedef]

Thanks for sharing the details. It seems that the results you attached are for Llama 3, while the NVFP4 perplexity we reported is for Llama 3.1-8B. Since these are different model versions, their perplexity values are not directly comparable, and a discrepancy of around one point could be expected.

[uccs123]

The FouroverSix paper uses Llama-3.1-8B for the 8B Llama 3 experiments. When I evaluated perplexity on the same Llama-3.1-8B model using both FouroverSix and your proposed ArcQuant, the results differed by approximately one point. Nevertheless, the downstream task results are almost the same.