Add percentile PTQ method in ncnn2table

[Roundaboutt]

PR description

Add percentile calibration method to ncnn2table

For quantization of Transformer blocks, the percentile calibration method performs slightly better than KL-divergence calibration.

Users can specify the percentile calibration value:

./ncnn2table \
  vit_tiny.ncnn.param \
  vit_tiny.ncnn.bin \
  calib.list \
  vit_tiny_percentile.table \
  shape=[224,224,3] method=percentile percentile=0.9999 type=1 thread=8

Alternatively, users can omit the percentile parameter to use the default automatic percentile search mode:

./ncnn2table \
  vit_tiny.ncnn.param \
  vit_tiny.ncnn.bin \
  calib.list \
  vit_tiny_percentile.table \
  shape=[224,224,3] method=percentile type=1 thread=8

In automatic search mode, the percentile value for each activation layer is selected from percentile=[0.999,0.9995,0.9999,0.99995,0.99999,1.0], choosing the value with the smallest quantization error.

Users can also specify the candidate percentile values for automatic search:

./ncnn2table \
  vit_tiny.ncnn.param \
  vit_tiny.ncnn.bin \
  calib.list \
  vit_tiny_percentile.table \
  shape=[224,224,3] method=percentile percentile=[0.999,0.9995,0.9999,0.99995] type=1 thread=8

Move Gemm weight scale init

Move Gemm weight quantization from ncnn2int8 to ncnn2table. The quantization scale is written to and stored in the table file instead of being computed online.

Experiments

PTQ experiments were performed with vit-tiny-patch16-224. The calibration dataset and test dataset both use the first 500 images from ImageNet.

Model	Accuracy
FP32 PyTorch	73.00%
FP32 ncnn	73.20%
KL int8 ncnn	62.80%
percentile int8 ncnn	63.40%

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 536aea4898

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Propagate Gemm scale lookup failures before saving

When a Gemm has both constant operands and the table contains a valid _param_0 entry but omits or has an invalid _param_1 entry, this return occurs after A has already been transposed and quantized. The caller ignores the result of quantize_gemm() and proceeds to save, leaving int8 A data alongside fp32 B data without setting int8_scale_term, which produces a corrupted model instead of rejecting the incomplete table; validate both entries before mutation or propagate the failure from main.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Include zero activations in percentile ranks

For sparse activations such as ReLU outputs, build_activation_histogram() skips every exact zero at lines 1049-1050, so sum and this target count represent only nonzero samples. Consequently, an explicit percentile=0.999 selects the 99.9th percentile of nonzero values rather than the documented 99.9th percentile of the activation distribution (for 50% zeros, approximately the overall 99.95th percentile), yielding systematically inflated thresholds and potentially worse quantization; track the skipped-zero count when computing the percentile rank.

Useful? React with 👍 / 👎.