How do you turn a local model into a server your apps can use?

A chat window is only the start. Once a model runs on your own hardware, you probably want other programs to use it: a coding agent, a script that summarizes files, an internal tool for your team. To do that, you run the model as a server, a program that waits for requests on a network address and sends answers back.

Nearly all of these servers speak the same language. They copy the request format of OpenAI's API, so software written for a hosted model can usually point at your machine by changing one setting. The comparison of Ollama and LM Studio covers the desktop apps. This article covers the serving side, and how to pick an engine.

Three engines

A worked example: three ways to start

With Ollama, the server is already running in the background once it is installed, and the API answers on port 11434.

With llama.cpp, you name a model and a context size. Here 32,768 tokens are shared by two slots:

llama-server -hf <user>/<model>:<tag> -c 32768 --parallel 2 --port 8080

With vLLM, on a Linux machine with a suitable graphics card:

uv pip install vllm --torch-backend=auto
vllm serve Qwen/Qwen2.5-1.5B-Instruct --host 127.0.0.1 --max-model-len 8192 --gpu-memory-utilization 0.8

Any of them can then be called the same way, here from Python with the standard OpenAI client:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="local")
reply = client.chat.completions.create(
    model="local",
    messages=[{"role": "user", "content": "Summarize this note in two lines: ..."}],
)
print(reply.choices[0].message.content)

Change the port to match your engine. On Ollama and vLLM, model must be the name the server lists at /v1/models.

Settings that bite

Memory. vLLM takes 92% of the graphics memory by default so lower --gpu-memory-utilization if anything else shares the card. Its memory guide also recommends limiting --max-model-len when you run short.

Context multiplied by users. In Ollama, required memory scales with the number of parallel requests times the context length. Four parallel requests at 32,000 tokens cost about the same as one at 128,000. A model that fit fine for one person can fail for four, and the memory calculator shows how to plan for this.

Queues. When too many requests arrive, Ollama answers with a 503 error, and OLLAMA_MAX_QUEUE sets how long the line can get. Your app should retry rather than fail.

What batching buys you

A language model can work on several requests at once, reading the same weights for all of them. Continuous batching adds new requests to the batch as old ones finish, so the machine stays busy. llama-server and vLLM both offer it.

LMSYS measured what this means in practice on a DGX Spark, using another server called SGLang. On Llama 3.1 8B, one request at a time produced 20.5 tokens per second. With 32 requests at once, total output reached 368 tokens per second. That is about 18 times more work from the same chip, though each user then sees about 11.5 tokens per second, which is slower than one user alone. Batching is a benefit for a shared service, not for a single chat.

Keeping it private

Local means private only as long as the server stays local. Ollama and llama-server listen only to your own machine by default, and vLLM's guide describes localhost as its default. Name the address explicitly, as in the command above, then check it from another device.

• Ollama: setting OLLAMA_HOST to 0.0.0.0 exposes it to your network.
• llama-server: --host changes the address, and --api-key turns on a key. There is no key by default.
• vLLM: --api-key or the VLLM_API_KEY variable does the same.

If others need access, use a VPN or a reverse proxy with its own login rather than opening the port on your router. Chat logs stay on your disk too, as the cloud or local guide explains.

Here is my setup: [engine, model, graphics memory, number of expected users, typical prompt length]. Suggest values for context length and parallel requests that fit my memory, and show the arithmetic. Then list what I should check to confirm the server is reachable only by the people I intend.

Do you need a server at all if you work alone?

You probably already have one. Ollama and LM Studio both run a local API, so a coding agent or a script can use them without any extra work. There is no reason to install vLLM for a single user on a home computer: the advantage of batching appears only when several requests arrive at the same time.

Move to vLLM when you have a real queue: a team sharing a model, or a program sending many requests in parallel. Before then, the simplest setup that keeps your data on your machine is the right one. If you do grow, your apps will not need to change much, since all three accept the same request format.