Research

Local models by hardware

This chapter is for a later writer. It says what you may promise a person who wants the model on their own computer, split by about 8 GB, 16 GB, 24 GB, 32 GB, 48 GB, 64–80 GB, and 128 GB or more. The picks, file sizes, and scores below are the 14 November 2026 check. A weight file that fits in RAM still needs extra memory for the operating system and for the conversation (the key-value cache). Ollama’s own quickstart says larger context windows need more memory. This chapter does not add a tokens-per-second figure for the tags it picks. Published speed figures for other files stay labeled as those files. Do not add a watt figure. None was measured.

Where a vendor page says a model can call tools, that is a vendor claim. Where a paper or a personal writeup measured multi-step calling, that is a measured result, and it stays labeled as that author’s result. Publisher benchmark cells below are the publisher’s claim. One local file size in the notes is UNVERIFIED. Leave it that way. An older generation is not the pick at any tier below.

1. Install Ollama, and the local OpenAI-style URL

Ollama is the straightforward way to run an open-weight model and expose it as a local server. Quote the install lines. Do not paraphrase them into a different procedure.

Windows 11. The Windows doc requires “Windows 10 22H2 or newer, Home or Pro,” which covers Windows 11. It says: “The easiest way to install Ollama on Windows is to use the OllamaSetup.exe installer. It installs in your account without requiring Administrator rights.” After that, “Ollama will run in the background and the ollama command line is available in cmd, powershell or your favorite terminal application. As usual the Ollama API will be served on http://localhost:11434.” The installer wants about 4 GB of disk for the program, then more disk for the models (“tens to hundreds of GB”). Docs: docs.ollama.com/windows.

macOS. The macOS doc requires “MacOS Sonoma (v14) or newer” and “Apple M series (CPU and GPU support) or x86 (CPU only).” It says: “The preferred method of installation is to mount the ollama.dmg and drag-and-drop the Ollama application to the system-wide Applications folder.” On first launch the app offers to link the ollama command into /usr/local/bin. Docs: docs.ollama.com/macos. The macOS download page also prints a terminal installer: curl -fsSL https://ollama.com/install.sh | sh (ollama.com/download/mac).

OpenAI-compatible URL. Ollama’s API introduction table lists the local OpenAI-compatibility base URL as http://localhost:11434/v1 (the cloud one is https://ollama.com/v1). The same page says: “Cloud requests need an API key. Local requests do not.” The OpenAI-compatibility page’s local example is:

client = OpenAI(
    base_url='http://localhost:11434/v1/',
    api_key='ollama',  # required but ignored
)

That example calls model='qwen3.8:27b'. The quickstart’s local curl is http://localhost:11434/v1/chat/completions. Native Ollama chat (not the OpenAI shape) is http://localhost:11434/api/chat.

A model name that ends in :cloud, or a call sent to https://ollama.com, is Ollama’s hosted service. That is a different machine from the reader’s. Do not describe that call as local. Never write ollama pull for glm-5.3:cloud, glm-5.3-flash:cloud, deepseek-v4.1-flash:cloud, or muse-spark-open:cloud. https://ollama.com/library/deepseek-v4 was a 404 on 14 November 2026.

2. The other official pointers

llama.cpp and LM Studio are the usual alternatives when someone does not want the Ollama app. The notes do not walk through their build flags. Point at the official pages and stop.

3. What fits on each machine

“Fits” means the published weight file is smaller than the tier, with some room left over. It does not mean a 256K context window will fit. Tool-calling sentences on a vendor page are the vendor’s claim. Publisher scores in this table are the publisher’s claim.

Tier Pick Weights What you may promise
About 8 GB RAM qwen3.5:4b 3.3–4.0 GB. Q4_K_M is 3.3 GB. Vision, tools, and thinking. Apache 2.0. The listing prints 256K. Do not promise 256K on 8 GB. Publisher’s claim on the card: LiveCodeBench v6 55.8, TAU2-Bench 79.9, BFCL-V4 50.3. Source: https://huggingface.co/Qwen/Qwen3.5-4B
About 16 GB RAM mimo-v2.6:9b 6.6–8.7 GB Vision, tools, and thinking. MIT on the Ollama page. The publisher’s own table, not an independent paper: SWE Verified 61.1 vs Qwen3.5-9B 60.0, SWE Pro 44.6 vs 32.0, Terminal Bench 2.1 37.1 vs 27.0. Source: https://ollama.com/library/mimo-v2.6 Do not load the 13.1 GB llama.cpp file on this computer.
About 24 GB, Ollama qwen3.8:27b 18 GB Vision, tools, and thinking. Apache 2.0. 32K fits in 24 GB. Local Agent Bench, which is not a paper, measured this tag at 17 GB with a 32K context. Do not promise 256K. Do not cite that bench to claim a 4B model matches this tag. Publisher’s claim: Terminal Bench 2.1 73.0, SWE-bench Pro 61.7, LiveCodeBench v6 90.3. On that same table, Muse Glimmer-30B is 51.7 and 51.2. Glimmer is not the pick, even though an 18 GB file is the same size class. Sources: https://huggingface.co/Qwen/Qwen3.8-27B and https://localagentbench.com/results/
About 24 GB, llama.cpp only ByteShape GPU-5, tag byteshape/Qwen3.8-27B-GGUF:Qwen3.8-27B-IQ4_XS-3.84bpw 13.1 GB Not an Ollama tag. Do not ollama pull the Hugging Face id. The publisher says GPU-5 keeps 99.63% of the full-precision score. GPU-4 is 11.0 GB and keeps 98.72%. Ollama and LM Studio were not confirmed. The publisher’s commands use llama-server.
32 GB qwen3.8:27b 18 GB, the same tag More room for context than the 24 GB tier. No open Muse exists under 30B.
48 GB qwen3.8:27b-q8_0 30 GB The same model at a larger quant. An older generation is not the pick.
64–80 GB qwen3.8:27b-bf16 56 GB On 64 GB, if the operating system and the context do not fit beside it, stay on qwen3.8:27b-q8_0.
128 GB and above qwen3.5:122b 81 GB Vision, tools, and thinking. Apache 2.0. Publisher’s claim on the Qwen3.5-122B-A10B card: SWE-bench Verified 72.0 vs 62.0, Terminal Bench 2 49.4 vs 18.7. The lower figure in each pair is Qwen’s column for GPT-OSS-120B, not an OpenAI number. Source: https://huggingface.co/Qwen/Qwen3.5-122B-A10B Muse Spark Open is not a local pull.

Two files are easy to promise onto the wrong computer. Do not.

The 13.1 GB llama.cpp file is not the 16 GB computer’s model. On a 16 GB computer the Ollama tag remains mimo-v2.6:9b. GPU-1 is 8.8 GB and does not fit an 8 GB computer once the operating system takes its share. DFlash2 needs llama.cpp b10658 or newer and is text only. The bundled MTP draft is about 250 MB and still accepts images.

The Ollama quickstart model is not an 8 GB-laptop model. The quickstart says: “The model download is about 7.2 GB. We recommend 8 GB of available VRAM, or unified memory on a Mac.” That is 8 GB free for the model, on top of the operating system. Do not put that quickstart download in the 8 GB total-RAM tier. The 8 GB pick is qwen3.5:4b.

What sources say about small models and long tool chains

Keep both the vendor sentence and the measured result. Do not let the vendor sentence overwrite the measurement. Do not paste an older paper’s percents onto qwen3.5:4b, mimo-v2.6:9b, or qwen3.8:27b.

Haque et al., “TinyLLM” (arXiv:2511.22138, 27 Nov 2025), evaluate small models on the Berkeley Function Calling Leaderboard. Their multi-turn tables describe an older small generation as weak at prompt-based function calling, and the PDF does not agree with itself on the exact percent. This chapter does not quote one of those percents as the score, and that paper is not a measurement of qwen3.5:4b.

A personal GitHub gist, “Local agentic AI benchmark for 2026” (author’s own test: five queries that each need three tools in order, repeated on a Mac mini and a Lunar Lake PC), states the failure mode directly: “They fail when tool B's argument depends on tool A's output. That's the on-device cliff.” Those figures are the gist author’s, from a small hand-built set, for an older generation. They are not a published leaderboard, and they are not the score for qwen3.5:4b.

Long chains are hard even for models that are not small. Patil et al. (arXiv:2509.26553) write that “success rates fall sharply as sequence length grows: GPT-5 drops from 72.5% (5 core nodes) to 15.0% (20 core nodes).” That paper’s method is their FuncBenchGen framework. It is evidence about chain length, not a reason to expect an 8 GB local model to do better.

AgentFloor (arXiv:2605.00334) did not test qwen3.5:4b, mimo-v2.6, or qwen3.8. This chapter does not paste that paper’s percents onto these tags.

One size you must not invent

The Ollama Qwen3.5 readme’s benchmark table names Qwen3.5-397B-A17B. That name was not in the downloadable size list on the library page (the largest tag opened there is 122b). A local file size for the 397B model is UNVERIFIED. Do not print a gigabyte figure for it.

An older generation is still on the library and is not the pick. Llama 4 on Ollama is llama4:16x17b at 67 GB and llama4:128x17b at 245 GB. Neither file is the pick, and 67 GB does not fit a 48 GB machine.

4. The plain recommendation

Promise less than the vendor pages promise. Use the tag in the table for that tier.

An 8 GB computer runs qwen3.5:4b. Promise a small local model with vision, tools, and thinking, and promise the publisher’s card only as the publisher’s claim. Do not promise 256K. A vendor line about tools is not a promise that a multi-step chain will finish.

A 16 GB computer runs mimo-v2.6:9b. The weight file fits more comfortably than the 8 GB file. MiMo’s agent rows are the publisher’s claim against Qwen3.5-9B, not an independent paper. Do not load the 13.1 GB file there.

About 24 GB of unified memory or VRAM is the Ollama tier for qwen3.8:27b at 18 GB. 32K fits. Do not promise 256K. The same tier’s llama.cpp file is ByteShape GPU-5 at 13.1 GB, and that file is not an Ollama tag. This tier can take a picture, because the Ollama tag lists vision. It will be slower than a hosted frontier model, and long chains stay fragile even for much larger models (the FuncBenchGen paper’s own GPT-5 figure drops from 72.5% at 5 steps’ worth of core nodes to 15.0% at 20).

A 32 GB machine uses the same qwen3.8:27b tag and has more room for context. Muse Glimmer is not the pick.

A 48 GB machine uses qwen3.8:27b-q8_0 at 30 GB. An older generation is not the pick.

A 64–80 GB machine uses qwen3.8:27b-bf16 at 56 GB. On 64 GB, stay on qwen3.8:27b-q8_0 if the operating system and the context do not fit.

A 128 GB machine uses qwen3.5:122b at 81 GB. Muse Spark Open is not a local pull.

5. Privacy upside, and the practical costs

Privacy. A model the reader has pulled, called at http://localhost:11434 with no API key, runs on that computer. Ollama’s API introduction says local requests do not need an API key, and the quickstart’s local tab says “No API key required.” LM Studio’s docs say the app “can operate entirely offline” once the model files are already on disk. That is the upside you may promise: the prompt stays on the machine while the agent stays local.

The prompt leaves if any of these happen:

Speed. On a machine without a strong GPU, generation is the slow part. Ollama’s macOS doc says Intel Macs are “x86 (CPU only).” The macOS download page says: “Large models are slow on a computer without a strong GPU.” The quickstart says: “With less VRAM, Ollama can use system RAM, but responses may be slower.” Do not invent a tokens-per-second number for qwen3.5:4b, mimo-v2.6:9b, or qwen3.8:27b. The notes measured none.

Addendum, 2026-11-14 (published measurements, not our own): the llama.cpp maintainers' llama-bench table in discussion #4167 gives Llama 2 7B Q4_0 (3.56 GiB) text generation at about 14 tokens/s on a base M1 and 21–32 on base M2 to M5. Malakhov, "Deploying LLMs on CPU-only Environments with llama.cpp" (CEUR-WS vol. 4164, 2026, doi), Table 1, gives an Intel Core i7-13700H on CPU alone (Linux) at about 15 tokens/s for an older 7B file and about 9 for Gemma-3-12B. These are similar models, not the exact files in the tiers above. Local Agent Bench is cited above only for the 17 GB fit of qwen3.8:27b at 32K. It is not a paper, and it is not evidence that a 4B model matches Qwen3.8.

Vision. The picks in the table list vision, so qwen3.5:4b, mimo-v2.6:9b, qwen3.8:27b, and qwen3.5:122b can take an image. An older text-only generation is not the pick. A text-only card cannot take a picture. DFlash2, the separate draft, is text only.

Electricity. No vendor watt or kilowatt-hour figure was found for the notes, so none is stated. The practical cost is that the CPU or GPU stays busy for the whole reply, which draws more power than an idle computer, and a slow CPU run stays busy longer. Do not turn that into a dollar amount. The cost rows below stay at $0 plus the computer the reader already has, because the notes never price the electricity.

Cost rows

One row per tier. “How smart it feels” is a judgment for this chapter, not a benchmark and not a vendor score. A 5 would mean the machine feels like a frontier agent. No tier here gets a 5. The 8 GB judgment stays low because the file is small and the card’s scores are the publisher’s claim, not a long run on 8 GB. The 16 GB judgment uses MiMo’s own table, which is the publisher’s claim. From 24 GB up, the judgment follows the Qwen3.8 card’s coding and agent rows, again as the publisher’s claim, plus room for the file. The 128 GB row uses the 122B card’s comparison the same way. The monthly cost is not an electricity estimate. The notes give no watt or kilowatt-hour figure, so the bill for the software is $0 plus the computer the reader already has. Power use is real and unpriced. The catch badge on every row is Hardware dependent.

Option Real monthly cost Credit card or hardware How smart it feels (1-5) Catch
8 GB computer, qwen3.5:4b (Q4_K_M 3.3 GB) $0 plus the computer you already have Hardware you already have. No subscription is priced in the notes. 2, a judgment: a small local model with vision and tools. The card’s scores are the publisher’s claim. Not a frontier agent. Hardware dependent. Do not promise 256K. The quickstart download that wants about 8 GB free is not this row.
16 GB computer, mimo-v2.6:9b (6.6–8.7 GB) $0 plus the computer you already have Hardware you already have. No subscription is priced in the notes. 3, a judgment: a more comfortable file. MiMo’s agent rows are the publisher’s claim, not an independent paper. Hardware dependent. Do not load the 13.1 GB llama.cpp file on this computer.
About 24 GB, qwen3.8:27b (18 GB) $0 plus the computer you already have Hardware you already have. No subscription is priced in the notes. 4, a judgment: vision, tools, and room for a 32K context (17 GB in Local Agent Bench). Not a 5. Hardware dependent. Do not promise 256K. ByteShape GPU-5 is the llama.cpp file, not an Ollama tag. Long chains stay fragile.
32 GB, same qwen3.8:27b tag $0 plus the computer you already have Hardware you already have. No subscription is priced in the notes. 4, a judgment: the same model, with more room for context. Not a 5. Hardware dependent. Muse Glimmer is not the pick. No open Muse exists under 30B.
48 GB, qwen3.8:27b-q8_0 (30 GB) $0 plus the computer you already have Hardware you already have. No subscription is priced in the notes. 4, a judgment: the same model at a larger quant. The card’s score is not a quant-specific delta. Not a 5. Hardware dependent. An older generation is not the pick.
64–80 GB, qwen3.8:27b-bf16 (56 GB) $0 plus the computer you already have Hardware you already have. No subscription is priced in the notes. 4, a judgment: the same model in a 56 GB file. Not a 5. Hardware dependent. On 64 GB, stay on qwen3.8:27b-q8_0 if the operating system and the context do not fit.
128 GB and above, qwen3.5:122b (81 GB) $0 plus the computer you already have Hardware you already have. No subscription is priced in the notes. 4, a judgment: the 81 GB tag. The comparison with GPT-OSS-120B is Qwen’s column, the publisher’s claim. Not a 5. Hardware dependent. Muse Spark Open is not a local pull.

Sources