Go/security post on building a self-hosted LLM security proxy with sub-2ms prompt inspection
useful practical architecture for regulated multi-provider LLM deployments where cloud gateways or provider-native guardrails are insufficient
What it is: Go/security post on building a self-hosted LLM security proxy with sub-2ms prompt inspection
Gist: author built an OpenAI-compatible reverse proxy (“Tamga”) that scans prompts for PII, secrets, and prompt-injection patterns before forwarding to providers; key engineering lesson is a hybrid scan pipeline where cheap CPU-bound detectors run sequentially while slower network/model-backed scanners run in parallel, because goroutine orchestration overhead dominated when everything fanned out
Newsletter angle: concrete infra pattern for “LLM middleware” that is more about latency budgets and data residency than model eval hype