KTransformers and heterogeneous MoE inference
Useful systems item because it shows the same recurring inference theme again: once models are sparse, the interesting optimization surface shifts from raw GPU count to placement, scheduling, quantization, cache locality, and heterogeneous memory hierarchy.
Logged at IST: 2026-07-19 02:31 IST
What it is: X post pointing to kvcache-ai/ktransformers, a Tsinghua MADSys Lab project for CPU-GPU heterogeneous inference and fine-tuning of large MoE models.
Gist: The viral framing is a little breathless, but the underlying project is real and interesting: KTransformers is about heterogeneous CPU/GPU execution for large MoE models, keeping hot experts on GPU and offloading colder expert work to CPU/DRAM so very large models can be explored on commodity-ish hardware. The repo’s own docs claim DeepSeek-V3/R1 support on 24GB VRAM with long-context paths, 3x to 28x speedups in the relevant setup, and fine-tuning paths for large MoEs through LLaMA-Factory on small GPU clusters.
Newsletter angle: Useful systems item because it shows the same recurring inference theme again: once models are sparse, the interesting optimization surface shifts from raw GPU count to placement, scheduling, quantization, cache locality, and heterogeneous memory hierarchy.