[snimu]

Gated Residual variants for prime_moe (pretraining)

Stacked on #4062 (feat/pretrain). Options for the pretraining MoE's Gated Residual (GR) connections and residual-stream inputs, used by the GR-variants screen (pretrain-experiments experiments/arch/2026-10-19-gr-variants/report.md). All options default off; with them off the model is unchanged (the control's step-1 loss is bitwise identical to 2769909d22 / the modern base).

New PrimeMoeConfig fields:

field	what it does
hc_qkv_reads	separate GR reads (own up-projection / gate, shared norm and down-projection) for the attention's query, key and value inputs
hc_moe_reads	separate reads for the MoE router and the experts (routed and shared)
hc_qkv_conv	causal depthwise convolutions (kernel hc_qkv_conv, residual form, zero-initialized) after the attention read, one per q/k/v input; fla causal_conv1d with cu_seqlens, so they reset at document boundaries
hc_stream_width	narrower streams (e.g. 8 x d/2); reads assemble d from chunks, writes use one scalar per (stream, chunk)
hc_stream_inputs	what the streams start from: copy (default: the token embedding in every stream), tables (one token table per stream), or token tables + ngram / bytes streams (several joined by _, e.g. ngram_bytes, ngram_ngram, ngram_ngram_ngram; repeated kinds are independent copies with their own hash layouts)
hc_ngram_vocab_size	rows per hash head of an n-gram stream (NGramEmbedding: bi- and trigrams, document- and EOS-aware)
hc_byte_window, hc_byte_table	byte stream (ByteWindowEmbedding): the last K UTF-8 bytes ending with each token, pulled within the document from a static token-to-bytes table
hc_byte_add	add an own byte embedding to every token-table stream (tables) or n-gram stream (ngrams) instead of a byte stream
hc_parallel_block	parallel attention + MLP with GR (shared_read / separate_reads), both outputs written to all streams with their own scalars
hc_write	GR write variants of sequential blocks: norm (RMS-normalize the block output before the write), output (write scalars also from the normalized block output, zero-initialized term), channel (a gate per stream and channel from the read's bottleneck)

Where the code lives: the generic, default-off HyperConnection extensions (read_count, write_count, stream_width, the read convolution, write_mode) are in the shared models/layers/hyper_connection.py (which Qwen3.8-Flash-Next also uses; its defaults keep the original kernels, and the Qwen3.8 tiny-model forward is unchanged); everything prime_moe-specific (stream inputs, ByteWindowEmbedding, separate attention / MoE inputs, the parallel block) is in models/prime_moe/. Supporting changes: the attention and the MoE accept separate inputs (separate q/k/v inputs need model.fusions without qkv); init_weights zero-initializes the read convolutions; GR projections and convolutions train with AdamW (existing hyper_connection role); extra embedding tables, n-gram and byte tables are embedding-role parameters (AdamW, weight decay as the token embedding); grad norms of extra tables log under embed.

Checks (details in the report): control step-1 loss bitwise equal to the base tree (and, after merging feat/pretrain 5c0bfa6129, bitwise equal to unmodified feat/pretrain: 11.792655944824219 on the d384 control; the Qwen3.8-Flash-Next tiny-model forward is identical on feat/pretrain and the merged branch); byte windows equal a direct reference on packed multi-document rows (K 16 / 32); n-gram ids reset at document starts; each later commit leaves the previous configs' step-1 loss unchanged; 30-step smokes of every option on H100 / H200.

Commits = the frozen source trees the screen ran (sha256 of git diff 2769909d22):

• 05db158f9e = code-f95bfdda (stage 1-3 screen)
• ed0a0d713b = code-fec107db (zero-init read convolutions)
• n-gram + byte combination = code-e4bf0f7b (round 2)
• parallel blocks = code-dca8a291
• repeated stream kinds = code-969c64ea (r2nn / r2bb)
• three n-gram streams + additive bytes = code-e894a72a (input program)
• write variants 509d6e7b2e = code-80e2e174
• merge of feat/pretrain 5c0bfa6129 (hyper_connection.py moved to models/layers/, the merged branch's diff vs feat/pretrain sha256 84bdc669..., frozen as code-m84bdc669)