alethic-300m
A 300M-parameter model trained for code, English and Uzbek, with a 4,096-token context and a Unigram tokenizer. Comes as a matched pair: SGCA and a GQA baseline.
A 300M-parameter model trained for code, English and Uzbek, with a 4,096-token context and a Unigram tokenizer. Comes as a matched pair: SGCA and a GQA baseline.
Pretraining and SinGatedAttention exploration from scratch on a single Colab T4. No matched baseline, which is why the 300M pair exists.
Model card →The 116M text-generation model that came before the Alethic line.
Model card →Parameter-matched small LM experiment with a sinusoidal attention gate, trained on Tiny Shakespeare.
Model card →The supervised fine-tuning data behind the Alethic assistant: instruction answers, thinking blocks, tool use, context maps, and long project sessions. English and Uzbek.
Request access594 of the examples are long project sessions, written from the author's own projects and analysed with Gemini. Their ids start with D.
Main line: does separating attention retrieval from learned residual control help? Tested against conventional baselines at matched parameters and compute.
Attention does retrieval. A separate, learned per-channel control path handles the residual. Definition and notes live on the author's research page.
A 2026 proposal: learnable sinusoidal components alongside linear pathways, multiplicative harmonic terms, and adaptive complexity.
Parameter-matched comparison of a sinusoidal attention gate on Tiny Shakespeare.
Pretraining from scratch on Colab. Useful signal, but no matched baseline, so no claim.
SGCA against GQA with a narrow-GQA ablation. At 1B tokens the GQA baseline came out ahead, and the code-focused model continues from the GQA checkpoint.
Overtrained 10M pair on the same 1B tokens, then supervised fine-tuning of Alethic-300M for real answers in English, Uzbek and code.
Held-out evaluation across 1,017,192 tokens from six sources (seed 1234, 1× H100 80 GB).
C = g · (α · sin A)Illustration of the SGCA residual update vs. GQA identity. Near A = 0, sin(A) ≈ A; beyond ±π/2, the gradient cos(A) vanishes and flips sign, bounding residual magnitude to |g·α|.
Every run is published with its configs, tokenizer and data notes, checkpoints, logs and measured comparisons.
All artifacts on Hugging Face