llama.cpp as a pure Go library
A useful example of local inference becoming importable infrastructure for Go services, without cgo, shared libraries, or a separate wasm runtime.
Links: Original source · Shared link · Related link 1 · Related link 2 · Related link 3 · Related link 4
Logged at IST: 2026-08-25 20:15 IST
What it is: goccy/go-llama, a pure-Go llama.cpp-style inference library for running GGUF models from Go applications.
Gist: The interesting bit is the packaging path. go-llama does not bind to llama.cpp through cgo, and it does not embed a wasm runtime. Instead, llama.cpp is compiled to WASI WebAssembly, then translated ahead-of-time into standalone Go through wasm2go and llamawasm2go. The result is meant to be a normal Go dependency: no shared library, no cgo, and static-binary friendly.
The README claims a fairly complete local-inference surface: model/context management, streaming, interruption, sampling controls, speculative decoding, LoRA adapters, chat templates, embeddings, scoring, and state save/load. It also exposes sandbox-ish knobs around preopened directories, memory caps, in-memory filesystems, and stdio capture.
The caveat is also clear: wasm32 linear memory caps the engine at 4 GiB, so this is for small quantized models, roughly the README's “3B parameters at Q4” target, not giant local serving. But that is still a useful deployment shape: embedding small LLM-derived features directly inside Go services without shipping a C++ runtime edge.
Newsletter angle: Good AI-infra/developer-tools item: local inference is becoming something a Go service can import like a library, not a sidecar process or cgo deployment project.