Computer Vision · Media · Organization

AI-Powered Media Management

I had 12,000 photos scattered across three cloud services and no way to find anything. So I built a local photo library that automatically tags faces, sorts scenes, and lets me search with natural language — all running on my Pi, with nothing leaving my network.

⏱ 30 min 📊 Intermediate 📅 May 2026

What We're Building

A private media library that understands what's in your photos. Point it at a folder of images and it'll generate searchable tags (people, places, objects, scenes), detect and cluster faces, and let you search with natural language: "show me photos of Emma at the beach from last summer." No cloud. No Google Photos. No iCloud. Your media stays on your hardware.

The Photo Problem

My phone had 12,000 photos. My "organization strategy" was the camera roll: chronological, uncurated, impossible to search. Google Photos offered to fix this with AI tagging, but the price was uploading my entire visual life to their servers. I wasn't comfortable with that.

For anyone with personal or sensitive photos — kids, medical documents, travel shots from places that don't need to be in a training dataset — cloud-based photo management is a hard no. The good news: local computer vision models have gotten good enough that you can run the same features entirely on your own hardware. I did it on a Pi 5.

What Local Computer Vision Can Do Today

Object Detection

Identify objects in photos — cars, furniture, electronics, tools, plants. Tags like "laptop," "bicycle," or "potted plant" become searchable.

Scene Classification

Recognize environments: beach, forest, office, kitchen, concert venue. "Find photos from outdoor events" actually works.

Face Detection & Clustering

Detect faces, group similar ones together. You label each cluster once ("that's Emma") and all matching photos are tagged.

Text Recognition (OCR)

Read text in photos — signs, menus, documents, whiteboards. "Find the photo of the restaurant menu from New Orleans."

Natural Language Search

CLIP-based search: "a red bicycle leaning against a brick wall" returns relevant photos even if they were never manually tagged.

Duplicate Detection

Find near-duplicate photos and burst-mode sequences. Recover gigabytes of storage from accidental duplicates.

Hardware Considerations

Computer vision is computationally heavier than text-based LLM inference. Here's what to expect on different Pi 5 configurations:

HARDWARE IMAGE CLASSIFICATION FACE DETECTION CLIP SEARCH
Pi 5 (CPU only) ~3 sec/image · Fine for batch ~2 sec/image · Acceptable ~8 sec/image · Slow for large libraries
Pi 5 + Hailo-8L ~0.3 sec/image · Great ~2 sec/image (CPU-bound) ~8 sec/image (CPU-bound)
Pi 5 + Hailo-10H ~0.1 sec/image · Excellent ~0.3 sec/image · Great ~0.5 sec/image · Excellent

Without an NPU, Use Batching

If you're running CPU-only, don't expect real-time results. Index a few hundred photos at a time, let it run overnight, and build your library incrementally. For a 20,000-photo library on a bare Pi 5, expect the initial index to take 8-16 hours. With a Hailo-10H, the same library indexes in under 2 hours.

Step 1: Install the Vision Stack

Installing computer vision dependencies
cd ~/ai-stack && source venv/bin/activate
pip install opencv-python-headless pillow numpy
✓ Core image processing installed
pip install transformers torch torchvision --index-url https://download.pytorch.org/whl/cpu
✓ PyTorch + Transformers for vision models
pip install face-recognition sentence-transformers
✓ Face detection + CLIP embeddings ready

Step 2: The Media Indexing Engine

Create media_indexer.py — the core of your local media management system. This handles object detection, scene classification, face detection, and CLIP-based semantic tagging:

media_indexer.py — Core indexing engine
# Local Media Indexer
# Tags photos with objects, scenes, faces, and CLIP embeddings

import
os
import
json
import
numpy
as
np
from
pathlib
import
Path
from
PIL
import
Image
from
typing
import
List, Dict
from
dataclasses
import
dataclass, asdict
@dataclass
class
MediaRecord
: filepath: str filename: str width: int height: int tags: List[str]
# Auto-generated tags
scene: str
# Scene classification
faces: int
# Number of faces detected
text_content: str
# OCR-extracted text, if any

class
MediaIndexer
:
"""Local media analysis — no cloud dependencies."""

def
__init__
(self):
# Scene classification categories
self.scene_categories = [
"indoor"
,
"outdoor"
,
"nature"
,
"urban"
,
"beach"
,
"mountain"
,
"forest"
,
"garden"
,
"office"
,
"kitchen"
,
"restaurant"
,
"street"
,
"concert"
,
"sports"
,
"celebration"
,
"portrait"
, ]
# Common objects to detect
self.object_tags = [
"person"
,
"car"
,
"bicycle"
,
"dog"
,
"cat"
,
"laptop"
,
"cell phone"
,
"book"
,
"plant"
,
"food"
,
"drink"
,
"furniture"
,
"instrument"
, ] self._init_models()
def
_init_models
(self):
"""Lazy-load models to save memory when not indexing."""
self._models_loaded = False
def
_ensure_models
(self):
"""Load vision models on first use."""
if
self._models_loaded:
return

# CLIP for semantic understanding
from
transformers
import
CLIPProcessor, CLIPModel self.clip_model = CLIPModel.from_pretrained(
"openai/clip-vit-base-patch32"
) self.clip_processor = CLIPProcessor.from_pretrained(
"openai/clip-vit-base-patch32"
)
# Face detection using dlib
import
face_recognition self.face_detector = face_recognition
print(
"✓ Vision models loaded"
) self._models_loaded = True
def
_classify_scene
(self, image: Image.Image) -> str:
"""Classify the scene using CLIP zero-shot classification."""
self._ensure_models()
categories = [f"a photo of a {c} scene"
for
c
in
self.scene_categories]
inputs = self.clip_processor( text=categories, images=image, return_tensors=
"pt"
, padding=True ) outputs = self.clip_model(**inputs) scores = outputs.logits_per_image.softmax(dim=1)[0]
best_idx = scores.argmax().item()
return
self.scene_categories[best_idx]
def
_detect_faces
(self, image_path: str) -> int:
"""Count number of faces in image."""
self._ensure_models()
try
: image = self.face_detector.load_image_file(image_path) locations = self.face_detector.face_locations(image, model=
"hog"
)
return
len(locations)
except
:
return
0
def
_generate_tags
(self, image: Image.Image) -> List[str]:
"""Zero-shot object detection via CLIP."""
self._ensure_models()
prompts = [f"a photo containing a {tag}"
for
tag
in
self.object_tags]
inputs = self.clip_processor( text=prompts, images=image, return_tensors=
"pt"
, padding=True ) outputs = self.clip_model(**inputs) scores = outputs.logits_per_image.softmax(dim=1)[0]
# Keep tags with confidence above threshold
threshold = 0.15 matched = []
for
i, score
in
enumerate(scores):
if
score > threshold: matched.append(self.object_tags[i])
return
matched
def
index_image
(self, filepath: str) -> MediaRecord:
"""Generate complete metadata record for a single image."""
filename = os.path.basename(filepath) image = Image.open(filepath).convert(
"RGB"
) width, height = image.size
# Run all classifiers
tags = self._generate_tags(image) scene = self._classify_scene(image) faces = self._detect_faces(filepath)
return
MediaRecord( filepath=str(filepath), filename=filename, width=width, height=height, tags=tags, scene=scene, faces=faces, text_content=
""
)
def
index_directory
(self, directory: str, extensions: List[str] = None) -> List[MediaRecord]:
"""Walk a directory and index all images."""
if
extensions
is
None: extensions = [
".jpg"
,
".jpeg"
,
".png"
,
".webp"
,
".bmp"
]
records = [] files = [f
for
f
in
Path(directory).rglob(
"*"
)
if
f.suffix.lower()
in
extensions]
for
i, filepath
in
enumerate(files):
try
: record = self.index_image(str(filepath)) records.append(record)
if
(i + 1) % 20 == 0: print(f" {i+1}/{len(files)} images indexed...")
except
Exception
as
e: print(f" ✗ {filepath.name}: {e}")
return
records

Step 3: Run Your First Index

Start small — pick a manageable folder (200-500 photos) for your first run:

Indexing a photo folder
cd ~/ai-stack && source venv/bin/activate
python3
>>> from media_indexer import MediaIndexer
# First run downloads CLIP model (~600MB cache)
>>> indexer = MediaIndexer()
>>> records = indexer.index_directory("/home/pi/Pictures/family")
✓ Vision models loaded
20/437 images indexed...
40/437 images indexed...
437/437 images indexed.
# Save records for searching
>>> import json
>>> with open("/home/pi/Pictures/index.json", "w") as f:
... json.dump([r.__dict__ for r in records], f, indent=2)
✓ 437 photos indexed with tags, scenes, and face counts

Step 4: Build Natural Language Search

CLIP enables something remarkable: search your photos with natural language descriptions, even without any pre-existing tags. Create media_search.py:

media_search.py
# Natural Language Photo Search
from
PIL
import
Image
from
pathlib
import
Path
from
transformers
import
CLIPProcessor, CLIPModel
import
torch
class
PhotoSearch
:
"""Search your photos with plain English."""

def
__init__
(self):
print
(
"Loading CLIP model..."
) self.model = CLIPModel.from_pretrained(
"openai/clip-vit-base-patch32"
) self.processor = CLIPProcessor.from_pretrained(
"openai/clip-vit-base-patch32"
)
print
(
"✓ Ready to search"
)
def
search
(self, query: str, image_paths: list, top_k: int = 10) -> list:
"""Find top_k images matching a natural language query."""
results = []
for
path
in
image_paths: image = Image.open(path).convert(
"RGB"
) inputs = self.processor( text=[query], images=image, return_tensors=
"pt"
, padding=True )
with
torch.no_grad(): outputs = self.model(**inputs) score = outputs.logits_per_image.item()
results.append((str(path), score))
# Sort by score descending
results.sort(key=
lambda
x: x[1], reverse=True)
return
results[:top_k]

# === Example usage === # searcher = PhotoSearch() # paths = [str(p) for p in Path("./photos").glob("*.jpg")] # results = searcher.search("a cat sleeping on a windowsill", paths) # for path, score in results: # print(f"{score:.3f} {path}")

Step 5: Face Clustering

This is the feature people care about most — automatically finding all photos of the same person. The face_recognition library handles this well:

Face clustering workflow
python3
>>> import face_recognition

# Step 1: Encode every face in your library
>>> all_faces = {}
>>> from pathlib import Path
>>> for img_path in Path("./photos").glob("*.jpg"):
... image = face_recognition.load_image_file(str(img_path))
... encodings = face_recognition.face_encodings(image)
... if encodings:
... all_faces[str(img_path)] = encodings

# Step 2: Cluster similar faces with DBSCAN
# Compare each face to known clusters, group similar ones

# Step 3: Label each cluster once
>>> # "Cluster_3" → label as "Emma"
>>> # All photos in cluster 3 now searchable as "Emma"
✓ Face clustering complete — export clusters.json

Face Clustering Strategy

For a personal library (not surveillance, not law enforcement — just family photos), a simple strategy works well: encode every detected face, then use DBSCAN clustering on the encodings. Each cluster gets a UUID until you label it. You only need to label each cluster once.

Step 6: Combined Search Library

A lightweight Python class that combines tag filtering, scene filtering, face filtering, and provides library statistics. Create search_photos.py:

search_photos.py — Combined search
import
json
from
pathlib
import
Path
class
PhotoLibrary
:
"""Search your indexed photo library."""

def
__init__
(self, index_path: str):
with
open(index_path)
as
f: self.records = json.load(f) print(f"Loaded {len(self.records)} indexed photos")
def
filter_by_tags
(self, tags: list) -> list:
"""Find photos matching all given tags."""
results = []
for
r
in
self.records:
if
all(tag
in
r[
"tags"
]
for
tag
in
tags): results.append(r)
return
results
def
filter_by_scene
(self, scene: str) -> list:
"""Find photos of a specific scene type."""
return
[r
for
r
in
self.records
if
r[
"scene"
] == scene]
def
filter_by_faces
(self, min_faces: int = 1) -> list:
"""Find photos containing at least min_faces people."""
return
[r
for
r
in
self.records
if
r[
"faces"
] >= min_faces]
def
summary
(self):
"""Print library statistics."""
all_tags = {}
for
r
in
self.records:
for
tag
in
r[
"tags"
]: all_tags[tag] = all_tags.get(tag, 0) + 1
scenes = {}
for
r
in
self.records: scene = r[
"scene"
] scenes[scene] = scenes.get(scene, 0) + 1
print(f"\n=== Library Summary ===") print(f"Total photos: {len(self.records)}") print(f"\nTop tags:")
for
tag, count
in
sorted(all_tags.items(), key=
lambda
x: -x[1])[:10]: print(f" {tag} {count}") print(f"\nScenes:")
for
scene, count
in
sorted(scenes.items(), key=
lambda
x: -x[1]): print(f" {scene} {count}")
# Usage: # lib = PhotoLibrary("/home/pi/Pictures/index.json") # beach_photos = lib.filter_by_scene("beach") # group_photos = lib.filter_by_faces(min_faces=3) # animals = lib.filter_by_tags(["dog", "cat"]) # lib.summary()

Building the Web UI

All of this works from the command line, but the real experience is a web interface with thumbnail browsing and natural language search. A simple Flask app provides this — create media_server.py with a /api/search endpoint that queries your index and a lightweight HTML frontend with a search bar and photo grid. Expose it on port 5051 and access it from any device on your local network.

The key endpoints:

  • GET / — Photo browser with search bar and thumbnail grid
  • GET /api/search?q=beach+sunset — Returns matching photo records as JSON
  • GET /photo?path=/home/pi/Pictures/img.jpg — Serves the full-resolution image

Privacy: The Real Reason to Self-Host Photo AI

Photos are uniquely personal. They contain faces of children, the insides of your home, documents on your desk, locations you frequent. Uploading them to a cloud provider for AI analysis means granting that provider:

  • Access to biometric data: Face encodings of every person you photograph
  • Location history: Even without GPS metadata, scene classification reveals patterns: beaches, schools, workplaces
  • Relationship mapping: Who appears together, how often, in what contexts
  • Document exposure: OCR picks up mail on countertops, whiteboards in meeting rooms, screens in the background

Google Photos is free because your photos are the product. The AI features are subsidized by the training value of your visual data. Local computer vision gives you the same capabilities — slower, less polished, occasionally less accurate — with zero data leakage.

The Face Recognition Ethics Line

Face clustering for your own family photos, on your own hardware, is one thing. Running face recognition on photos of people without their knowledge, or building a surveillance system, is another. This guide is for managing your personal media library. The technology is the same either way — the ethics depend on how you use it.

Scaling: From Hundreds to Tens of Thousands

For serious photo libraries, you'll outgrow the simple JSON index. Here's the upgrade path:

1
SQLite for metadata

Replace JSON file with a SQLite database. Index tags, scenes, face clusters, and file paths. Adds millisecond query speed for tag-based filtering on 50K+ photos.

2
ChromaDB for CLIP embeddings

Store CLIP embeddings in ChromaDB for sub-second semantic search across your entire library. This enables "find me more photos like this one."

3
Thumbnail pre-generation

Generate 256px thumbnails at index time. Scrolling through thousands of results stays responsive without loading full-resolution images.

4
Hailo-10H for real-time search

40 TOPS makes CLIP embedding generation fast enough for interactive search. Type a query, see results update as you refine.

What This Replaces

CLOUD SERVICE LOCAL ALTERNATIVE TRADE-OFF
Google Photos AI search CLIP + ChromaDB Slower but private
Apple Photos face recognition face_recognition + DBSCAN You label the clusters
Google Lens text extraction Tesseract OCR Works offline
Paid duplicate finder apps perceptual hash (phash) No upload, no purchase

Next Steps