GitHub - 0arch-io/kisoku: Kisoku 1.6B: a from-scratch language model trained solo on a TPU grant. Weights， code， evals and report.
Kisoku 1.6B is a language model pretrained from scratch by a single researcher on a TPU v4-32, achieving performance comparable to Meta's Llama 3.2 1B with significantly less training data. The project includes a chat fine-tune, long-context extensions, and a detailed technical report covering the training process, contamination audit, and cost analysis.