Jay Margalus
◆ Agents · Project 5 of 5 · 3 hours

Your own small model

Run a model on your own laptop. Put a router in front of it that sends easy and private work to the local model and hard work to Claude, logs every decision, and lets you grade it.

Show

Jay's local-first router answering a weather question with a tool call and a local model, with a footer showing the route taken, the model used, and the time.
My router on Oct 24. Weather went to a tool, the follow-up to a local model, and the footer says which route and how long. Hard questions go to Claude.
A Terminal window showing route stats: per backend counts, success rate, median seconds, tokens, and cost for local, fireworks, claude, and codex over 14 days.
Two weeks of routing: 81 requests stayed local at 1.5 s median, 27 went to Claude. Graded on cost, wait, and success, an idea from OpenRouter. Success here means the backend answered, not that it was right. The twenty prompts below cover that.

Why local, and why route at all. Models are becoming interchangeable parts. The routing logic, the logs, and the data stay yours. A local model is also the only honest answer for student records or patient notes. It's also worse than Claude at hard reasoning and real coding. So: local for the easy, the bulk, and the private; frontier for the hard; a router that knows the difference and admits when it doesn't. And no, a hundred-billion-parameter model does not fit on a 64 GB laptop.

Do 0 of 8

  1. Install Ollama. ollama.com, one installer. Then:

    ollama pull gemma4:12b-it-qat
    ollama run gemma4:12b-it-qat

    Ask it something. Ask it today's date. Ask what it can't do. A 12B quantized model fits in about 8 GB. With 32 GB or more, also pull a 30B-class model and compare.

  2. Give it a face. Optional: Open WebUI puts a chat window over Ollama in your browser, no account. Notice how often it's enough.

  3. Write the classifier. A short script. For each prompt, ask the local model first: Is this LIGHT (short, factual, rewrite, summary) or HEAVY (multi-step reasoning, code, anything you're unsure of)? One word. Quote the user's prompt inside markers so it labels instead of obeying. Unsure means HEAVY.

  4. Route. LIGHT to the local model. HEAVY to claude -p. One prefilter first: anything you've marked private (a folder, a word like “student” or “patient”) stays local no matter what. Privacy beats the classifier.

  5. Log every decision. One line per request to route.jsonl: first 60 characters of the prompt, label, backend, seconds, came back or not. No log, no router.

  6. Grade it. Twenty prompts you wrote in advance, ten light and ten heavy. A /wrong command to overrule after the fact. Count how many went where you'd have sent them. A heavy question answered badly costs you; a light one answered by Claude costs nothing. Keep two columns: did it come back, and was it right. A model can return a confident wrong answer with a perfect success rate.

  7. Add the escape hatch. When the local model fails (error, empty, timeout), retry once on Claude, unless the request was private. Never silently. The log says it escalated and why.

  8. If you're hooked. Replace the one-word classifier with a scorer: per backend, recent success rate and median wait, route to the cheapest that's allowed. That's the second screenshot. Thirty lines.

Ticks are saved in this browser.

Check

  • Twenty prompts, logged. You can name the routing accuracy and the three decisions you overruled.
  • A private prompt stayed local even though the classifier called it heavy.
  • A local failure escalated to Claude and the log line says so.
  • You can say in one sentence what your local model is for and what it isn't. If the sentence is “everything,” rerun the heavy ten.

In your course

Assign

As a mastery option, not a requirement. It's where the student stops using someone else's system and starts owning the trade-offs.

Grade

The routing log, the twenty-prompt sheet, and one paragraph on “what did you keep local and why.” The reasoning about the two kinds of wrong matters more than the number.

Equipment

A 16 GB laptop runs a 12B model. One old Mac with 32 GB becomes the department's always-on box for Projects 3 to 5.

The whole term's plan is on the faculty guide.