What You'll Need
Raspberry Pi 5 (8GB) — strongly recommended, or Pi 4 (4GB minimum); 64GB+ microSD card; active cooling; Ubuntu Server 24 LTS or Raspberry Pi OS 64-bit. Internet connection required only for initial setup.
Why Offline-First?
Most AI applications today require cloud connectivity — sending your data to someone else's servers, paying per-token fees, and hoping your internet connection stays stable. This guide takes a different approach: everything runs locally, permanently, without external dependencies.
The stack you'll build:
- Ollama — Local LLM inference engine (qwen2.5, qwen3, phi4-mini, etc.)
- ChromaDB — Vector database for semantic search and embeddings
- RAG Pipeline — Retrieval-augmented generation for document Q&A
- Python API — Simple Flask/FastAPI bridge to connect components
Documents
PDFs
Text Files
↓
↓
ChromaDB
↔
nomic-embed-text
↓
↓
Phase 1: System Preparation
Start with a fresh Ubuntu Server 24 LTS or Raspberry Pi OS 64-bit. Ensure your system is up to date and has the necessary build tools.
sudo apt-get update && sudo apt-get upgrade -y
Reading package lists... Done
Building dependency tree... Done
sudo apt-get install -y python3-pip python3-venv git curl
✓ Dependencies installed
Memory Considerations
LLMs are memory-hungry. For Raspberry Pi 5, use the 8GB model. Pi 4 users should stick to smaller models (3B parameter) and enable swap space. With a Hailo-8L NPU, you can offload inference and run larger models — see the Hailo setup guide.
Phase 2: Install Ollama
Ollama makes running local LLMs trivial. One command installs the service, and model management is handled automatically.
Phase 2 — Ollama Installation
curl -fsSL https://ollama.com/install.sh | sh
>>> Installing ollama...
Downloading ollama...
✓ Ollama installed to /usr/local/bin
ollama --version
ollama version 0.6.4
Verify Ollama is running and test with a small model:
ollama pull qwen2.5-coder:3b
pulling manifest...
pulling 6f482eff78d0... 100%
✓ Model downloaded (~1.9GB)
ollama run qwen2.5-coder:3b
>>> Write a Python function to calculate fibonacci
def fibonacci(n):
if n <= 1:
return n
return fibonacci(n-1) + fibonacci(n-2)
Recommended Models for Pi
Model size directly impacts inference speed. Here are my recommendations:
- qwen2.5-coder:3b — Fast coding assistant, good for autocomplete and quick queries
- qwen3:4b — General purpose with strong reasoning; latest Qwen generation
- phi4-mini:3.8b — Microsoft's efficient model, excellent reasoning for size
- qwen2.5:7b — Higher quality, requires 8GB Pi and patience
- nomic-embed-text — Embedding model for RAG (required)
Phase 3: Install ChromaDB
ChromaDB is a vector database optimized for embeddings and semantic search. It stores document chunks as high-dimensional vectors, enabling fast similarity search.
mkdir -p ~/ai-stack && cd ~/ai-stack
python3 -m venv venv
source venv/bin/activate
pip install chromadb sentence-transformers flask
Collecting chromadb...
✓ ChromaDB installed
Phase 4: Build the RAG Pipeline
Now for the core component — a RAG (Retrieval-Augmented Generation) system that:
- Loads documents (PDFs, text files, etc.)
- Splits them into chunks
- Generates embeddings via Ollama
- Stores in ChromaDB
- Retrieves relevant chunks for queries
- Sends context + query to LLM for response
Create the RAG module at ~/ai-stack/rag_engine.py:
rag_engine.py
import
chromadb
import
ollama
from
pathlib
import
Path
from
typing
import
List
class
RAGEngine
:
"""Local RAG pipeline using ChromaDB + Ollama."""
def
__init__
(self, collection_name=
"documents"
):
self.client = chromadb.PersistentClient(path=
"./chroma_db"
)
self.collection = self.client.get_or_create_collection(
name=collection_name,
metadata={
"hnsw:space"
:
"cosine"
}
)
def
embed_text
(self, text: str) -> List[float]:
"""Generate embeddings via Ollama's nomic-embed-text."""
response = ollama.embed(
model=
"nomic-embed-text"
,
input=text
)
return
response[
"embeddings"
][0]
def
add_documents
(self, texts: List[str], ids: List[str] = None):
"""Index documents into ChromaDB."""
if
ids
is
None:
ids = [f
"doc_{i}"
for
i
in
range(len(texts))]
embeddings = [self.embed_text(t)
for
t
in
texts]
self.collection.add(
embeddings=embeddings,
documents=texts,
ids=ids
)
def
query
(self, question: str, n_results: int = 3) -> dict:
"""Retrieve relevant documents for a question."""
query_embedding = self.embed_text(question)
return
self.collection.query(
query_embeddings=[query_embedding],
n_results=n_results
)
def
generate_answer
(self, question: str, model: str =
"qwen2.5-coder:3b"
) -> str:
"""Full RAG pipeline: retrieve context, generate answer."""
results = self.query(question)
context =
"\n\n"
.join(results[
"documents"
][0])
prompt = f
"""Answer based on the following context:
Context:
{context}
Question: {question}
Answer:"""
response = ollama.generate(
model=model,
prompt=prompt
)
return
response[
"response"
]
Now create a simple loader utility for text files:
document_loader.py
from
pathlib
import
Path
def
load_text_files
(directory: str, chunk_size: int = 500) -> list:
"""Load and chunk text files from a directory."""
chunks = []
paths = []
for
file_path
in
Path(directory).glob(
"*.txt"
):
text = file_path.read_text()
for
i
in
range(0, len(text), chunk_size):
chunk = text[i:i + chunk_size]
chunks.append(chunk)
paths.append(f
"{file_path.name}_{i}"
)
return
chunks, paths
Quick feedback
Did this guide help?
Your answers shape what we write next.
Phase 5: Create the API Server
A simple Flask API provides a clean interface to your RAG stack. Create api_server.py:
api_server.py
from
flask
import
Flask, request, jsonify
from
rag_engine
import
RAGEngine
app = Flask(__name__)
rag = RAGEngine()
@app.route(
"/health"
, methods=[
"GET"
])
def
health
():
return
jsonify({
"status"
:
"ok"
,
"engine"
:
"local"
})
@app.route(
"/ingest"
, methods=[
"POST"
])
def
ingest
():
"""Add documents to the knowledge base."""
data = request.get_json()
texts = data.get(
"texts"
, [])
ids = data.get(
"ids"
)
rag.add_documents(texts, ids)
return
jsonify({
"indexed"
: len(texts)})
@app.route(
"/query"
, methods=[
"POST"
])
def
query
():
"""Query the knowledge base."""
data = request.get_json()
question = data.get(
"question"
)
model = data.get(
"model"
,
"qwen2.5-coder:3b"
)
if
not
question:
return
jsonify({
"error"
:
"question required"
}), 400
answer = rag.generate_answer(question, model)
return
jsonify({
"question"
: question,
"answer"
: answer,
"model"
: model
})
if
__name__ ==
"__main__"
:
app.run(host=
"0.0.0.0"
, port=5000, debug=False)
Phase 6: Test the Stack
Let's verify everything works. First, ensure Ollama is running (it starts automatically), then launch your API:
cd ~/ai-stack && source venv/bin/activate
python api_server.py
* Running on http://0.0.0.0:5000
In another terminal, test the endpoints:
curl http://localhost:5000/health
{"engine": "local", "status": "ok"}
curl -X POST http://localhost:5000/ingest \\\\
-H "Content-Type: application/json" \\\\
-d '{"texts":["Raspberry Pi 5 features a quad-core ARM Cortex-A76 CPU at 2.4GHz"],"ids":["pi5_0"]}'
{"indexed": 1}
curl -X POST http://localhost:5000/query \\\\
-H "Content-Type: application/json" \\\\
-d '{"question":"What is the CPU in Raspberry Pi 5?"}'
{"answer":"The Raspberry Pi 5 features a quad-core ARM Cortex-A76 CPU running at 2.4GHz."...}
Phase 7: Production Deployment
For a production setup, use systemd to manage services and create a startup script:
sudo tee /etc/systemd/system/ai-stack.service << 'EOF'
[Unit]
Description=Local AI Stack API
After=network.target ollama.service
[Service]
Type=simple
User=pi
WorkingDirectory=/home/pi/ai-stack
ExecStart=/home/pi/ai-stack/venv/bin/python api_server.py
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now ai-stack
Performance Optimization
Running LLMs on Pi requires tuning. Here are proven optimizations:
Memory Optimization
Enable zram for compressed swap, reducing OOM kills during model loading:
sudo apt-get install zram-tools && sudo systemctl enable zramswap
- Use smaller models — 3B parameters is the sweet spot for Pi 5
- Quantization matters — Q4_K_M models run 2-3x faster than FP16
- Batch embeddings — Send multiple texts to embed at once
- SSD storage — USB3 SSD significantly improves model load times
- Active cooling — Throttling kills performance; use a fan
Troubleshooting
Common issues and solutions:
A quick reassurance before the table: an offline AI stack involves four moving parts — Ollama, ChromaDB, Python dependencies, and your own code — and when something breaks, the error can surface in a misleading place. A ChromaDB error might actually be a disk-space issue. An Ollama timeout might mean thermal throttling, not a model problem. If this is your first time debugging a multi-service stack, you'll feel like you're chasing ghosts. That's not a reflection on you. It's just how distributed systems fail. Check the logs, check the resources, check one layer at a time.
| Out of memory |
Use smaller model, enable zram, or add USB swap |
| Slow inference |
Ensure active cooling; check for thermal throttling with vcgencmd measure_temp |
| ChromaDB errors |
Delete ./chroma_db and re-index; check disk space |
| Ollama not responding |
sudo systemctl restart ollama or check logs with journalctl -u ollama |
| Embeddings timeout |
nomic-embed-text is fast; if timing out, check Ollama is running and model is pulled |
Extending the Stack
This is a foundation. Consider these enhancements:
- PDF support — Add
PyPDF2 or pymupdf to extract text from PDFs
- Web scraping — Use
beautifulsoup4 to index web pages
- Multi-user — Add Flask-Login and user-specific collections
- Chat history — Store conversation context for multi-turn interactions
- Stream responses — Use Flask-SSE or WebSockets for streaming LLM output
- Frontend — Build a React/Vue interface to the API
Next Steps
After completing this setup:
Get the Code
The complete RAG engine code is available on GitHub as rag-chatbot. It includes PDF support, chunked loading, and a simple web interface.