Your team asked for GPU budget for self-hosted inference. You said "not yet" because last time you checked, the tooling wasn't production-grade. That was true 18 months ago. It's not true now, and the delay is costing you leverage you don't know you're losing.
I'm writing this because most decision-makers I talk to are still running on an outdated mental model of what self-hosted LLM infrastructure looks like. The software moved. The org didn't.
The Team That Celebrated Too Early
I watched a team spin up on-prem inference, celebrate for a week, then watch it rot because nobody owned it. Six months later they were back on the API, having spent the budget anyway.
This is the failure mode nobody talks about. The software works. It's been working for a while now. The problem is everything around it.
Nobody owns the stack. Running self-hosted inference in production means someone on your team owns model updates, hardware failures, quantization tradeoffs, and latency tuning. That's a different job than calling an API. If you don't staff it, the deployment decays.
Procurement kills momentum. GPU capacity is a capital expenditure conversation, not a software download. If you don't already have data center access or cloud-GPU contracts, the blocker isn't the code. It's a procurement cycle that takes months. By the time the hardware arrives, the team that asked for it has moved on.
Model selection is real work. The quantized model that runs great for summarization falls apart on code generation. There is no default. Every use case needs evaluation, and evaluation takes time nobody budgets for.
These are solvable problems. But teams that skip them end up with on-prem deployments that nobody trusts, and leadership that says "see, I told you it wasn't ready" when the real issue was organizational, not technical.
What Changed While You Were Waiting
A year ago, I would have told you to hold off. Not anymore.
You can now split inference across multiple GPUs without patching anything yourself. The server mode handles concurrent requests behind a load balancer. 1-bit quantization means models that needed high-end hardware run on modest configs without catastrophic quality loss.
Multi-modal support landed. Speculative decoding shipped, cutting latency on long outputs. The API compatibility layer means your existing code that talks to cloud providers works against a self-hosted endpoint with a URL change.
I deployed a quantized model on a client's on-prem GPU last month. Set up the server, pointed the app at it, ran inference. It worked. First try. That sentence would have been fiction two years ago.
The gap between "experimental" and "production-ready" closed while most orgs were waiting for someone else to go first.
The Decision You're Actually Making
This isn't a permanent binary. It's a portfolio allocation.
Move workloads on-prem when:
- Your inference volume is high enough that API costs became a material line item.
- You need predictable latency without network variability.
- Compliance or data residency requirements mandate it. But verify this. Many teams assume they need on-prem when they don't.
- You have an engineer who wants to own the stack.
Stay on the API when:
- You're prototyping or usage is unpredictable.
- You need frontier models not available as open weights.
- Nobody on your team can own the ops burden.
The mistake I see most often: treating this as all-or-nothing. Start with API. Move specific workloads to self-hosted when economics or data constraints force the conversation. The infrastructure to do it properly exists now. It didn't two years ago.
The Question for Your Next Planning Cycle
The software is ready. The open-weight models are good enough for most production use cases. The tooling matured past the point where "not ready yet" is a defensible position.
The real question isn't whether the technology works. It's whether your org is set up to operate it. That's a staffing decision and a procurement decision, not a technology bet.
If you're still saying "not yet," make sure you're saying it because of an actual blocker, not because of a mental model that expired a year ago.
Related reading:
- Self-Hosted LLM vs API: when the math actually works — the decision framework I use with clients (deutsche Version)
- LLM API comparison 2026 — Claude vs GPT vs Gemini vs Mistral vs DeepSeek for production (deutsche Version)
I help teams navigate this decision. If your org is evaluating self-hosted inference and you want an honest assessment of readiness, reach out.
Get the next field note in your inbox. A new mini case from real AI builds every two weeks. No theory, no pitches.
I write field notes from real builds — AI integration, cron-driven automation, and the parts that break in production. New posts every two weeks at renezander.com.
Top comments (0)