underlay
module 086–8h

Running & Using LLMs

Call, stream, structure, self-host

Here's the liberating secret: every serious model provider — OpenAI, Anthropic, Google, DeepSeek, and every local server (Ollama, llama.cpp, vLLM) — speaks the same dialect. The OpenAI-compatible chat completions API. Learn one, and you can swap models like batteries.

The universal API: one curl, any model

A chat completion is: a model name, a list of messages (system = the rules, user = the ask, assistant = the model's replies), and knobs. Temperature near 0 = deterministic; near 1 = creative. This exact request works against OpenAI, OpenRouter, your own Ollama at localhost:11434/v1 — the only difference is the URL and key:

bashone API shape, every provider
curl https://api.openai.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
    "model": "gpt-5.2",
    "messages": [
      {"role": "system", "content": "You are a concise assistant."},
      {"role": "user", "content": "What is a KV cache? One sentence."}
    ],
    "temperature": 0.7,
    "max_tokens": 100
  }'

# Streaming: add "stream": true — the server replies in text/event-stream
# chunks, each a "data: {...}" line, ending with "data: [DONE]".
# Streaming is how every chat UI feels instant.

Structured output: make the model speak JSON

Unstructured text is useless to code. Function calling declares a tool schema; the model answers with a JSON tool_call instead of prose. This is the primitive agents are built on — module 9 is this plus a loop:

jsonfunction calling — the agent's hands
// request adds a "tools" array with a JSON Schema
{
  "model": "gpt-5.2",
  "messages": [
    {"role": "user", "content": "What's the repair price for an iPhone 15?"}
  ],
  "tools": [{
    "type": "function",
    "function": {
      "name": "get_repair_price",
      "description": "Get the current repair price for a device",
      "parameters": {
        "type": "object",
        "properties": {
          "device": {"type": "string", "enum": ["iphone-15", "pixel-9"]}
        },
        "required": ["device"]
      }
    }
  }],
  "tool_choice": "auto"
}

// response contains tool_calls — YOUR code runs the lookup, then sends
// the result back as a "role": "tool" message. That round trip is an agent.

Running models yourself: the VRAM math that decides everything

Weights need memory: parameters × bytes per parameter. fp16 = 2 bytes; the Q4_K_M quantization ≈ 0.55–0.6 bytes. That single line decides what fits on your hardware:

textthe one formula — worked examples from real GGUF files
weights_GB = params × bytes_per_param
  fp16:  × 2         8B -> 16 GB     70B -> 141 GB
  Q4_K_M: × ~0.58    8B -> 4.9 GB    70B -> 42.5 GB     405B -> ~243 GB

plus the KV cache (the conversation memory):
  8B with GQA at 128K context ≈ 16.4 GB of KV cache
  -> long context + big batch fight each other: this is why
     serving stacks (vLLM) exist and why GQA matters

speed ceiling ≈ GPU bandwidth / bytes per token
  H100 (3.35 TB/s) running 8B Q4 (~5 GB): ~600 tok/s max
bashlocal serving, three tiers
# 1) easiest: Ollama (OpenAI-compatible on :11434)
ollama run llama3.3

# 2) raw llama.cpp — any GGUF from HuggingFace
llama-server -hf bartowski/Llama-3.3-70B-Instruct-GGUF:Q4_K_M --port 8080

# 3) production serving: vLLM (batching, prefix cache, PagedAttention)
pip install vllm
vllm serve meta-llama/Llama-3.3-70B-Instruct --tensor-parallel-size 2

The 2026 landscape in one glance: frontier closed models (GPT-5.x, Claude, Gemini) cost $3–30 per million tokens; open-weight MoE models (DeepSeek V4-Flash at $0.14/$0.28, Qwen3, GPT-OSS) cost cents; local models cost electricity. The pro move is tiering: cheap model first, escalate only when it fails. Your cost sheet for 1,000 conversations, using real published prices, is this module's real check.

test yourself

01An 8B model at Q4_K_M quantization needs roughly:

02Streaming (SSE) in chat APIs exists because:

03Function calling is:

the real check

A quantized model serves OpenAI-shaped JSON on your VPS. Your cost sheet for 1,000 conversations matches published prices.

0 stamps earned·streak: 0 days

Signed out: progress lives in this browser. to carry it everywhere.

vibe-coder payoff

Every agent tool you use is this stack in a trench coat. Build it once by hand and nothing is magic anymore.