Running LLMs Locally: A Performance Benchmark

Why Local Inference Matters

The shift toward local-first AI isn’t just about privacy — it’s about performance predictability. When you control the hardware, you control the latency curve. No cold starts from serverless functions, no variable network hops, no shared GPU contention.

We set out to answer a simple question: can local inference compete with cloud APIs for production workloads? The answer surprised us.

Benchmark Setup

We tested three model families across four hardware configurations, measuring tokens per second, time-to-first-token, and peak memory usage. All models were quantized to 4-bit using GGML format for fair comparison.

Model	Parameters	Tokens/s	TTFT
Llama 3 8B	8B	42.1	180ms
Mistral 7B	7B	48.7	155ms
Phi-3 Mini	3.8B	67.3	95ms

Key Findings

The most striking result was Phi-3 Mini’s performance on Apple Silicon. With the Metal backend, it achieved 67 tokens per second — faster than most cloud API responses when you factor in network latency.

For enterprise deployments on dedicated NVIDIA hardware, Mistral 7B offered the best quality-to-speed ratio. Its sliding window attention mechanism proved particularly efficient for document processing workloads common in legal and healthcare.

Memory Considerations

Running models locally means being mindful of RAM. A 4-bit quantized 7B model typically requires 4-5GB of RAM, making it feasible on most modern workstations. The 13B variants push this to 8-10GB, which still fits comfortably within a 16GB machine but leaves less room for other processes.

Recommendations

If you’re evaluating local inference for production, start with quantized models and measure your specific workload. Generic benchmarks only tell half the story — prompt length, batch size, and concurrent request patterns all shift the performance landscape dramatically.