Show
Why local, and why route at all. Models are becoming interchangeable parts. The routing logic, the logs, and the data stay yours. A local model is also the only honest answer for student records or patient notes. It's also worse than Claude at hard reasoning and real coding. So: local for the easy, the bulk, and the private; frontier for the hard; a router that knows the difference and admits when it doesn't. And no, a hundred-billion-parameter model does not fit on a 64 GB laptop.
Do 0 of 8
Install Ollama. ollama.com, one installer. Then:
ollama pull gemma4:12b-it-qat ollama run gemma4:12b-it-qat
Ask it something. Ask it today's date. Ask what it can't do. A 12B quantized model fits in about 8 GB. With 32 GB or more, also pull a 30B-class model and compare.
Give it a face. Optional: Open WebUI puts a chat window over Ollama in your browser, no account. Notice how often it's enough.
Write the classifier. A short script. For each prompt, ask the local model first: Is this LIGHT (short, factual, rewrite, summary) or HEAVY (multi-step reasoning, code, anything you're unsure of)? One word. Quote the user's prompt inside markers so it labels instead of obeying. Unsure means HEAVY.
Route. LIGHT to the local model. HEAVY to
claude -p. One prefilter first: anything you've marked private (a folder, a word like “student” or “patient”) stays local no matter what. Privacy beats the classifier.Log every decision. One line per request to
route.jsonl: first 60 characters of the prompt, label, backend, seconds, came back or not. No log, no router.Grade it. Twenty prompts you wrote in advance, ten light and ten heavy. A
/wrongcommand to overrule after the fact. Count how many went where you'd have sent them. A heavy question answered badly costs you; a light one answered by Claude costs nothing. Keep two columns: did it come back, and was it right. A model can return a confident wrong answer with a perfect success rate.Add the escape hatch. When the local model fails (error, empty, timeout), retry once on Claude, unless the request was private. Never silently. The log says it escalated and why.
If you're hooked. Replace the one-word classifier with a scorer: per backend, recent success rate and median wait, route to the cheapest that's allowed. That's the second screenshot. Thirty lines.
Ticks are saved in this browser.
Check
- Twenty prompts, logged. You can name the routing accuracy and the three decisions you overruled.
- A private prompt stayed local even though the classifier called it heavy.
- A local failure escalated to Claude and the log line says so.
- You can say in one sentence what your local model is for and what it isn't. If the sentence is “everything,” rerun the heavy ten.
In your course
Assign
As a mastery option, not a requirement. It's where the student stops using someone else's system and starts owning the trade-offs.
Grade
The routing log, the twenty-prompt sheet, and one paragraph on “what did you keep local and why.” The reasoning about the two kinds of wrong matters more than the number.
Equipment
A 16 GB laptop runs a 12B model. One old Mac with 32 GB becomes the department's always-on box for Projects 3 to 5.
The whole term's plan is on the faculty guide.