Running LLMs Locally with Ollama: Privacy, Speed, and Zero Cost

Not every AI use case needs to send data to a cloud API. For privacy-sensitive applications, offline requirements, or high-volume workloads, running LLMs locally with Ollama is often the right choice.

What Is Ollama?

Ollama is an open-source tool that makes running LLMs locally as simple as running Docker containers. With one command, you can download and run state-of-the-art models.

```bash

# Install

curl -fsSL https://ollama.ai/install.sh | sh

# Run a model

ollama run llama3.3

# Pull a specific model

ollama pull mistral

ollama pull qwen2.5:72b

```

Best Local Models in 2026

| Model | Size | Best For |

|-------|------|---------|

| Llama 3.3 70B | 40GB | General purpose, near GPT-4 quality |

| Qwen 2.5 72B | 40GB | Coding and reasoning |

| Mistral 24B | 14GB | Fast, good quality |

| Phi-4 | 8GB | Small devices, edge deployment |

| DeepSeek-R1 | Various | Complex reasoning |

| Gemma 3 27B | 16GB | Google's open model |

Integrating with Your Application

Ollama provides an OpenAI-compatible API, so switching is trivial:

```typescript

import OpenAI from "openai";

const client = new OpenAI({

baseURL: "http://localhost:11434/v1",

apiKey: "ollama", // required but unused

});

const response = await client.chat.completions.create({

model: "llama3.3",

messages: [{ role: "user", content: "Summarise this document..." }],

});

```

When to Use Local LLMs

Use local when:

Use cloud APIs when:

Hardware Requirements

| Model Size | Minimum VRAM | Recommended |

|-----------|--------------|-------------|

| 7B | 8GB | 12GB |

| 13B | 12GB | 16GB |

| 34B | 24GB | 40GB |

| 70B | 40GB | 80GB |

For CPU-only inference, expect 5-10x slower speeds.

Ollama in Production

For production deployments, run Ollama as a service with multiple instances behind a load balancer. Use quantised models (Q4_K_M or Q5_K_M) for the best quality/performance balance.

```bash

# Run as a service with custom host

OLLAMA_HOST=0.0.0.0:11434 ollama serve

```

Local LLMs are increasingly viable for production. Talk to us about whether local deployment is right for your use case.