Qwen-2.5-1B-RLCD uses parallel constrained decoding for typed JSON work
It shows that a lot of typed JSON decision work can be reframed as bounded parallel choice evaluation rather than slow token-by-token generation.
Links: Original source · Shared link · Related link 2 · Related link 3
Logged at IST: 2026-09-17 16:02 IST
What it is: Harsha Gundala open-sourced Qwen-2.5-1B-RLCD, a small Apple Silicon oriented engine for type-safe JSON workloads.
Gist: The model card frames the trick as parallel constrained decoding. Instead of asking a model to autoregressively emit a whole JSON object, it treats schema fields as bounded choices, broadcasts a single KV-cache state across fields, slices logits to valid candidates, and assembles the JSON programmatically. On an M4 Max, the reported benchmarks show 5.6x to 7.0x latency reductions versus autoregressive structured generation, with 100% schema validity and field-level confidence scores.
The interesting part is how close this sits to the TypeSafe/Jev argument without requiring a new frontier model: for many routing, classification, triage, and extraction jobs, the output space is already typed and finite. If you can turn generation into parallel categorical decisions, a small local model can become a low-latency decision primitive.
Newsletter angle: A practical open-source echo of the typed decision-model idea: structured inference gets faster when you stop treating every field as prose generation.