Repository navigation
Py interface - #9
Open
stellon-admin wants to merge 11 commits into
Open
stellon-admin wants to merge 11 commits into
stellon-admin wants to merge 11 commits into
Conversation
Assisted-by: Codex
Assisted-by: Codex
Assisted-by: Codex
Assisted-by: Codex
Exempting silence (codec 4299) from the repetition penalty lets the main checkpoint stall in silence runs. Only speech_end and stop stay exempt, matching the Python sampler. Long silence runs could also drive the LM to all-NaN logits, after which top-p indexed an empty candidate list and segfaulted; skip top-p then so sample() reports 'no finite sampling candidates'. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Penalise the silence token; guard top-p against non-finite logits
The wheels published as 0.1.0 were built on 5 October, before the silence token was brought under the repetition penalty and top-p was guarded against non-finite logits. Generated output changes for a given seed, so this is a minor bump rather than a patch. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Bump version to 0.2.0
No GPU backend has TQ2_1 kernels. When a TQ2_1 weight is assigned to a GPU buffer type that supports the op with Q2_0, create it as Q2_0 and repack it while loading. The repack is lossless: both formats store 2-bit codes with FP16 scales, and each 128-weight TQ2_1 scale is copied bit-exact onto two 64-weight Q2_0 blocks. CPU buffers are unchanged. Repacked tensors are not served from the file mapping. - ggml: ggml_repack_tq2_1_to_q2_0(), which rejects code 3 - llama: llama_model_params.gpu_repack (default true), load log lines for the repacked tensors and the repack time - llama-ext.h staging API: llama_model_n_tensors(), llama_model_get_tensor_by_index(), llama_get_graph_splits() and llama_get_flash_attn(), so tools can check weight placement, CPU fallbacks and the resolved flash attention setting - ggml-cuda: cache the device's integrated flag at registration instead of calling cudaGetDeviceProperties on every get_type call - tests: repack round trips and invalid codes in test-tq2_1, Q2_0 mul_mat cases at Qwen3-1.7B shapes in test-backend-ops, and test-tq2_1-load, which loads a TQ2_1 GGUF with all layers on the GPU, checks that every TQ2_1 weight is on the GPU as Q2_0 and dequantizes to the same values as the file, and that with gpu_repack off the weights stay on the CPU as TQ2_1
Assisted-by: Codex
kitten-tts: NVIDIA GPU support and CUDA wheels (kitten-tts-cpp-cu12 / -cu13)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Additional information
Requirements