Local LLM (Ollama)

KaaS works with any OpenAI-compatible API, including Ollama for fully local, private LLM inference. This guide walks through setting up Ollama as the LLM backend for KaaS.

Prerequisites

• A machine with at least 8 GB RAM (16 GB+ recommended for larger models)
• KaaS installed via Docker or CLI

Step 1: Install Ollama

macOS:

brew install ollama

Linux:

curl -fsSL https://ollama.ai/install.sh | sh

Verify the installation:

ollama --version

Step 2: Start Ollama

ollama serve

By default, Ollama listens on http://localhost:11434. Keep this running in the background (or use systemctl enable ollama on Linux for auto-start).

Step 3: Pull a Model

Choose and download a model. For knowledge compilation tasks, 8B+ parameter models are recommended:

# Recommended for quality/speed balance
ollama pull llama3

# Alternatives
ollama pull mistral
ollama pull qwen2
ollama pull gemma2

Verify the model is available:

ollama list

Step 4: Configure KaaS

Point KaaS at the Ollama endpoint using environment variables.

Docker:

docker run -d --name kaas \
  -p 8080:8080 \
  -v ./data:/app/data \
  -e LLM_BASE_URL=http://host.docker.internal:11434/v1 \
  -e LLM_MODEL=llama3 \
  -e LLM_API_KEY=ollama \
  kaas

CLI:

export LLM_BASE_URL="http://localhost:11434/v1"
export LLM_MODEL="llama3"
export LLM_API_KEY="ollama"    # Ollama ignores this, but KaaS requires a non-empty value
kaas serve

Or set them in etc/kaas.toml:

[llm]
api_key = "ollama"
base_url = "http://localhost:11434/v1"
model = "llama3"

WARNING

When running KaaS in Docker, use host.docker.internal (macOS/Windows) or 172.17.0.1 (Linux) instead of localhost to reach Ollama on the host machine.

Step 5: Verify the Connection

After starting KaaS, check that it can reach Ollama:

# Quick test — call Ollama's OpenAI-compatible endpoint directly
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "llama3", "messages": [{"role": "user", "content": "hello"}]}'

If you get a streamed response, Ollama is ready. Open http://localhost:8080 and submit content to verify end-to-end compilation works.

Performance Considerations

Model Size	RAM Required	Compilation Speed	Quality
7-8B	8 GB	Fast	Good for most use cases
13-14B	16 GB	Moderate	Better extraction accuracy
70B+	48 GB+	Slow	Best quality, requires GPU

Recommendations:

• Minimum viable: llama3 (8B) — good balance of speed and quality.
• Better quality: llama3:70b or qwen2:72b — significantly slower without a GPU.
• GPU acceleration: Ollama automatically uses GPU if available (NVIDIA, Apple Silicon). No extra config needed.
• Summarize model: For large corpora, you can set a smaller/faster model for summarization tasks:

export LLM_SUMMARIZE_MODEL="llama3"   # Use smaller model for summaries
export LLM_MODEL="qwen2:72b"          # Use larger model for main compilation

Troubleshooting

Symptom	Cause	Fix
Connection refused	Ollama not running	Run ollama serve
Model not found	Model not pulled	Run ollama pull <model>
Very slow responses	Model too large for RAM	Use a smaller model or add GPU
Docker can't reach Ollama	Wrong hostname	Use host.docker.internal or 172.17.0.1