Open-Weight Models, Local Deployment & Routing

The moment you got Qwen running with Ollama in the Quest, you probably felt that "wow, I can actually run an LLM on my own machine" rush. But once the excitement fades, the question that's actually worth money is: in production, which workloads should run on self-hosted open-weight models, and which should just call an API? Get this wrong and you either burn a pile of GPU money for nothing, or hand every dollar you could have saved to OpenAI.

True story. I've seen a team whose boss heard "open-source models are free," rented two A100s, and built an inference cluster -- only to serve a few thousand requests a day. Those cards sat idle 90% of the time, and the monthly bill came out more than four times what the API would have cost. I've seen the opposite too: a team pushing hundreds of thousands of trivial classification requests a day through a GPT-4-class API, until finance came knocking at the end of the month. Both teams skipped the math.

This chapter walks through the math you should do after the Quest -- with real commands, a VRAM cheat sheet, cost worked examples, and a model selection table.

---

Why this matters (and what happens if you skip it)

• Your boss asks, "Can we ditch OpenAI and self-host to save money?" -- you need to calculate whether it saves anything and at what scale, not guess.
• Compliance says "customer data cannot leave the company" -- now you must know local/private deployment, because the API route is simply closed.
• An agent making hundreds of thousands of calls a day, every one hitting a GPT-4-class model, is pure money burning -- many requests only need a small model. That's exactly what "routing" solves.
• "Why did you choose self-hosting / API?" is a staple AI Engineer interview question. Answering "because open source is free" pretty much ends the interview.

---

Three terms that constantly get mixed up

Term: Open-Weight Model

• One-line explanation: a model whose trained parameters are published, so you can download and run it yourself.
• Analogy: a restaurant hands you the finished dish to take home (the weights), but not necessarily the full recipe and ingredient sourcing (training data/code).
• How you'll use it at work: Llama, Qwen, Mistral, DeepSeek, Gemma all qualify -- you can download them, run them locally, fine-tune them.
• Most common trap: treating "open weights" as "fully open source." Most only release weights, not training data, and come with commercial license restrictions (some clauses impose extra requirements at very large user scale). Always read the license before commercial use.

The three paths, side by side:

Quick tour of the major open-weight families (so you don't pick blind)

On sizing: 7B~8B is the sweet spot for local/single-GPU -- good enough quality, VRAM-friendly. 70B is better but demands multiple cards or aggressive quantization. Validate your needs on a small model first, then scale up. My honest advice: start with Qwen 7B for Chinese workloads, Llama 8B for English, and stop drooling over 70B on day one.

---

Where to get models and how to run them

Getting models: the vast majority live on Hugging Face (the GitHub of models). Ollama packages the popular ones so a single command downloads and runs them -- perfect for local experiments. The real commands:

# Option 1: Ollama -- local experiments / single user, least friction
ollama pull qwen2.5:7b
ollama run  qwen2.5:7b "Explain RAG in one sentence"

# Everyday management commands
ollama list          # see models downloaded locally
ollama ps            # see models currently loaded in VRAM
ollama rm qwen2.5:7b # remove what you don't use; don't let it eat your disk

# Ollama's server ships an OpenAI-compatible endpoint (port 11434 by default)
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen2.5:7b", "messages": [{"role": "user", "content": "Hi"}]}'

# Option 2: Hugging Face CLI -- download raw weights (bring your own inference / feed to vLLM)
pip install "huggingface_hub[cli]"
huggingface-cli download Qwen/Qwen2.5-7B-Instruct

# Option 3: vLLM -- production-grade high-throughput serving, exposes an OpenAI-compatible endpoint
pip install vllm          # pin the version; vLLM moves fast and flag names change every few releases
vllm serve Qwen/Qwen2.5-7B-Instruct --port 8000
# Now hit http://localhost:8000/v1 exactly like the OpenAI API -- barely any code changes

In production you'll almost certainly touch these vLLM flags:

vllm serve Qwen/Qwen2.5-72B-Instruct-AWQ \
  --port 8000 \
  --tensor-parallel-size 4 \        # shard one big model across 4 GPUs (70B won't fit on one card)
  --gpu-memory-utilization 0.90 \   # use up to 90% of VRAM, leave the rest for the system; default 0.9
  --max-model-len 8192 \            # cap context length = cap the KV Cache ceiling, prevents OOM
  --quantization awq                # load AWQ-quantized weights

Because both Ollama and vLLM expose an OpenAI-compatible API, your application code switches between "local model ↔ cloud API" by changing one base_url line:

from openai import OpenAI

# Point at local vLLM / Ollama; everything else looks identical to calling OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

resp = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=[{"role": "user", "content": "Rewrite this more formally: don't be late to tomorrow's meeting"}],
)
print(resp.choices[0].message.content)

This is also why "model routing" (covered below) is easy engineering -- the interface shape is identical, and the routing layer just decides which base_url a request goes to. (For the API fundamentals, see LLM API Basics; for unified multi-model access, see LLM Platforms & Gateways.)

Running models splits into two battlefields -- don't bring the wrong tool:

Honestly, I've stepped on this one: took Ollama straight to production, and the moment a few dozen concurrent requests arrived, everything queued up and froze. Ollama is a developer-experience gem, not a high-concurrency server. If you're serving real traffic, use vLLM.

Two server-side concepts you must know

• KV Cache: during generation, the K/V values already computed for earlier tokens get cached instead of recomputed (see the attention mechanics in the Transformer chapter). It's the key to inference speed, but it eats VRAM -- the higher the concurrency and the longer the context, the more VRAM the KV Cache consumes. This, not the model itself, is usually the real culprit behind "out of memory."
• Continuous Batching: vLLM's signature move -- dynamically packing multiple requests together to feed the GPU, delivering several times the throughput of one-at-a-time processing. This is the core difference that lets it handle concurrency where Ollama can't.

---

Estimating VRAM: one table to answer "can my card run this?"

The soul-searching question you'll hit in interviews and in practice alike: "How much VRAM does a 7B model need?" Start with a rough formula:

Model weight VRAM ≈ parameter count × bytes per parameter
  FP16          → 2 bytes per parameter
  INT8          → 1 byte per parameter
  4-bit quant   → ~0.5 bytes per parameter

Actual budget ≈ weights + KV Cache (grows with concurrency & context) + ~1-2GB framework overhead

Applying the formula, the common sizes (weights only, KV Cache not included):

Two field notes:

1. Don't forget the KV Cache. The table above only covers "fitting the model in." In production, at concurrency 50 with 8k context, the KV Cache can eat another few to a dozen GB. The classic incident I've watched: single-request load tests pass flawlessly, then production concurrency spikes and it OOMs -- the model didn't change, the KV Cache ate all the remaining VRAM.
2. Mac users, look at unified memory. On Apple Silicon, VRAM is RAM. A 16GB Mac runs 7B Q4 comfortably, but a 14B starts squeezing system memory and things get sluggish.

---

Quantization formats: don't mix up GGUF / GPTQ / AWQ

Term: Quantization

• One-line explanation: storing model parameters at lower precision (4/8-bit) instead of high precision (16-bit) to save VRAM.
• Analogy: the same movie at different resolutions -- 4K, 1080p, 720p. Weak machine, pick the smaller file.
• How you'll use it at work: 4-bit cuts VRAM usage to roughly 1/4.
• Most common trap: blindly chasing the highest precision. For most chat scenarios, 4-bit is visually indistinguishable while saving half the VRAM. Start small, scale up only if needed.

Three common formats -- pair them with the wrong tool and loading fails:

The GGUF suffix is a mini multiple-choice question worth unpacking:

Smaller number = less VRAM but lower quality; K_M denotes a mixed-precision scheme -- if you can't remember any of this, just use the default Q4_K_M.

(The VRAM-saving principle behind quantization is the same one behind QLoRA -- see the fine-tuning chapter.)

---

The core decision: self-host vs. call an API (with worked numbers)

This is the heart of the chapter. Don't go on gut feeling -- check a few criteria, then do the math:

Call an API (closed-source) when: volume is low or spiky (you don't want to pay a full month for idle GPUs), you need the strongest reasoning available, or nobody on the team wants to babysit GPU ops.

Self-host open weights when: ① data cannot leave the company (compliance -- the hard gate that overrides everything); ② volume is high and steady; ③ you need deep fine-tuning into a specialized model; ④ you want controllable latency and freedom from someone else's API price hikes or model deprecations.

The math: at what volume does self-hosting pay off?

Say you need 7B-class capability. A rough comparison (numbers are illustrative -- use current pricing):

Option A · Call an API
  Assume ~$0.3 per 1M tokens, ~2k tokens per request
  → ~$0.0006 per request; 100k requests/day → ~$60/day → ~$1,800/month

Option B · Self-host (rent one GPU that runs a 7B)
  A mid-tier inference GPU ≈ $1.5/hour × 24 × 30 ≈ $1,080/month (fixed)
  -- whether you serve 1 request or 1 million, you pay it

Break-even, roughly:
  $1,080 ÷ $0.0006 ≈ 1.8M requests/month ≈ ~60k/day
  → Steady daily volume above ~60k: self-hosting starts winning
  → Below that: the API wins, period

Laying out the volume tiers makes it obvious:

* At higher volume you may add cards, but self-host costs rise in steps while API costs rise linearly -- the bigger the volume, the wider the gap.

Beyond the worked example: three hidden costs people forget

1. Ops headcount. GPU drivers, CUDA versions, vLLM upgrades, 3am OOM pages -- someone has to own all of it. If nobody on the team wants to, price 20% of an engineer's time into the "self-host" column and the math may flip instantly.
2. Redundancy and high availability. When the API goes down, it's the vendor's problem (with SLA credits). When your self-hosted service goes down, it's yours. Production-grade means at least two instances -- cost ×2.
3. The opportunity cost of quality gaps. Requests a 7B open model answers poorly translate into churned users. That money never shows up on the GPU bill, but it's real.

Bottom line: with low or spiky volume, the API wins (you only pay for what you use); with high, steady volume, self-hosting amortizes the fixed cost until per-request cost approaches zero -- that's when it pays.

One-line honest advice: use an API to ship the product and validate demand first. Once volume arrives, the math checks out, or compliance forces your hand -- then consider self-hosting. Building a GPU cluster on day one is usually premature optimization. For the fuller toolbox of inference cost control (caching, batching, degradation), see Deployment & Cost Optimization.

---

Model routing: multiple models in one system

Term: Model Routing

• One-line explanation: automatically dispatching each request to the right (cheap/expensive) model based on difficulty.
• Analogy: a hospital triage desk. Headaches and colds go to general practice (small models); only the hard cases get referred to specialists (large models). Never let the specialist see the common cold.
• How you'll use it at work: simple intent detection/classification/rewriting goes to a local small model or a cheap API; complex reasoning/long documents get the flagship model.
• Most common trap: the routing decision itself calls a large model, so the savings get spent on the "deciding." The decision must use ultra-lightweight rules or a small model.

A typical money-saving router:

User request
   │
   ▼
[Lightweight classifier]  ── simple/common? ──▶ Local small model (Qwen 7B / Llama 8B)  cheap, fast
   │
   └── complex/high-value? ──▶ Flagship API (GPT / Claude)  expensive, strong

In code, the plainest version is an if/else -- because both sides speak the OpenAI-compatible API, routing is just swapping the client:

LOCAL = OpenAI(base_url="http://localhost:8000/v1", api_key="x")
CLOUD = OpenAI()  # official API

SIMPLE_TASKS = {"classify", "rewrite", "extract", "faq"}

def route(task_type: str, prompt: str):
    # Decide with "rules + request metadata" -- zero cost; never call a large model just to route
    if task_type in SIMPLE_TASKS and len(prompt) < 2000:
        return LOCAL, "Qwen/Qwen2.5-7B-Instruct"
    return CLOUD, "gpt-5"

client, model = route("classify", user_input)
resp = client.chat.completions.create(model=model, messages=[...])

So what exactly counts as a "simple request"? Here's a field-ready push-down checklist:

Two steps to level up:

1. Add a fallback/escalation path: local model times out or confidence is low → automatically retry on the flagship API. The user just feels "this one was a bit slower," never sees a failure.
2. Add a unified gateway: once you have several models, put routing, rate limiting, logging, and cost accounting in one gateway layer so application code never knows how many models sit behind it. How to build that layer: LLM Platforms & Gateways.

In practice, a large share of requests really are simple, and pulling them off the flagship model often cuts the bill by more than half while users barely notice. This is why "knowing how to run open-weight models" is genuinely valuable engineering -- not to replace APIs, but to pair with them. For which tasks belong on small models and how to evaluate systematically, see the layering table and scorecard in AI Model Selection & Comparison.

---

Self-hosting is not automatically secure

An easy trap to fall into: model running on your own machine ≠ security problems gone. They've just been swapped for a different set.

• Don't leave ports naked. Services started by vLLM / Ollama have no authentication by default. Binding them to a public IP hands your GPU to the whole internet for free. At minimum, put a reverse proxy + API key in front; if it's an internal deployment, keep it strictly on the internal network.
• Prompt injection applies in full. The attack targets "the model + your system prompt" as a combination, regardless of where the model is hosted. Self-hosted models often have weaker safety tuning too -- the guardrails are on you.
• Compliance is a reason to self-host, not a get-out-of-jail card. Keeping data in-house only solves the "where data travels/lives" clause. Log redaction, access control, and auditing all still need doing.

For the full threat checklist and defenses, see Security Threat Modeling.

---

Decision cheat sheet: one diagram to take away

The whole chapter compressed into one decision chain. Next time your boss asks "should we self-host?", walk through it:

Can the data leave the company?
   │
   ├── No (hard compliance gate) ──▶ Self-host, no choice. Go straight to VRAM & GPU-count math
   │
   └── Yes ──▶ Does steady daily volume exceed the break-even point? (compute yours from this chapter's example)
            │
            ├── No / volume is spiky ──▶ Call the API. Don't buy cards; spend the energy on the product
            │
            └── Yes ──▶ Does anyone on the team actually want to own GPU ops?
                     │
                     ├── No ──▶ Still the API, or a managed inference service (someone else feeds the GPUs)
                     │
                     └── Yes ──▶ Self-host + model routing:
                                simple requests → local small model
                                complex requests → flagship API as backstop

Note the final box: self-hosting and the API are not either/or. The most common production endgame is a hybrid architecture -- bulk traffic on your own small model, hard cases and fallback on the API.

---

⚠️ Common pitfalls

---

🔧 Hands-on Exercises

Exercise 1: Is a small model good enough? (~15 minutes)

1. Using the Ollama from your Quest, ask the same question to two model sizes:

ollama run qwen2.5:7b   "Summarize this passage into 3 key points: ..."
ollama run qwen2.5:1.5b "Summarize this passage into 3 key points: ..."

2. Test three task types: summarization, classification ("is this review positive or negative"), and code explanation.
3. Record the gap between 1.5B and 7B for each task type -- you'll find classification is nearly identical while code explanation differs a lot. That intuition is exactly what you'll use to design routing later.

Exercise 2: Compute your own break-even point + write one routing rule (~20 minutes)

1. Take this chapter's cost example and swap in your project's real numbers (N requests/day, tokens per request, current API pricing). Compute where your "self-hosting break-even point" lands in daily requests.
2. List the request types in your project, mark which can be pushed down to a small model and which must go to the flagship, and turn it into a genuinely runnable if/else following this chapter's route().
3. Bonus: add one fallback rule to the router -- local model timing out after 3 seconds automatically retries on the flagship API.

---

📚 Summary

1. Open weights ≠ fully open source: most release only the parameters, with commercial restrictions -- read the license first. Start sizing from the 7B~8B sweet spot (Qwen for Chinese, Llama for ecosystem).
2. Get models from Hugging Face; experiment locally with Ollama/LM Studio, but production concurrency demands vLLM/TGI (continuous batching carries the load). Both expose OpenAI-compatible APIs, so application code switches with one base_url change.
3. VRAM back-of-envelope: FP16 is 2 bytes per parameter, 4-bit about 0.5 -- a 7B at Q4 needs ~4-5GB. But the KV Cache is the real VRAM hog: load tests must include realistic concurrency and context. Don't mismatch quantization formats: GGUF→Ollama, AWQ/GPTQ→GPU; when in doubt, Q4_K_M.
4. Self-host vs. API is arithmetic: low/spiky volume → API wins; high steady volume (roughly 60k+ requests/day in the illustrative example), hard compliance requirements, or deep fine-tuning → self-host. Don't forget ops headcount and HA redundancy -- otherwise ship on the API first and don't build prematurely.
5. Model routing is where open-weight models earn their keep in engineering -- push simple requests down to small models, reserve the flagship for hard ones, and the bill often drops by more than half with users none the wiser. Route with rules/small models; never let the "triage" itself burn a large-model call.

---

Related Reading

• LLM API Basics — request structure of OpenAI-compatible APIs, the prerequisite for local/cloud switching
• AI Model Selection & Comparison — which tasks to push down to small models, and how to build an eval
• LoRA / QLoRA / PEFT Fine-Tuning — the next step after self-hosting: turning an open model into your specialized model
• LLM Platforms & Gateways — the gateway layer for unified multi-model access, rate limiting, and billing
• Deployment & Cost Optimization — caching, batching, degradation, and other inference cost controls
• Production Deployment — fitting the inference service into a full production architecture
• Security Threat Modeling — the threat checklist and defenses for self-hosted services
• Transformers & Attention — why the KV Cache exists and why it eats VRAM