GPT-2 in Pure CMake: Because Why Not

AlpinDale's gpt2.cmake is a proof-of-concept that implements a GPT-2 model entirely in CMake, using Q16.16 fixed-point arithmetic. The project provides two paths: a toy model for quick experimentation and a full model that loads the actual OpenAI GPT-2 weights from Hugging Face.

How it works

The toy model generates tables and a model definition via Python scripts, then runs inference through cmake -P gpt2.cmake. The full model downloads the safetensors, vocab, and merges files, converts them with tools/gen_full.py, and executes gpt2_full.cmake with a prompt and token count.

All arithmetic is done in Q16.16 fixed-point, avoiding floating-point operations entirely. This is a deliberate choice to keep the implementation deterministic and portable across platforms where CMake's math() command only supports integers.

Why this matters

CMake is not a general-purpose language. It lacks native data structures, has no standard library for neural networks, and its math() command is limited to 32-bit integers. Yet this project manages to implement attention, feed-forward layers, and tokenization—all within that constraint.

The result is a testament to the flexibility of CMake's scripting mode, but also a reminder that just because you can, doesn't mean you should. The code is likely unmaintainable and slow, but as a technical exercise it's impressive.

Trade-offs and limitations

Fixed-point arithmetic introduces quantization errors, which may degrade output quality compared to floating-point inference. The full model requires downloading ~500MB of weights and preprocessing them, so it's not a lightweight demo.

Performance is another concern: CMake scripts are interpreted, and even a small model will run slower than a native implementation. This is not a production tool—it's a curiosity that pushes the boundaries of what a build system can do.

For engineers, the takeaway is twofold: it's a fun hack to study if you want to understand fixed-point neural networks, and it's a cautionary tale about using the wrong tool for the job. If you need to run GPT-2, use a real runtime. If you want to impress your colleagues, this is the repo to link.