AI / Retrieval-Augmented Generation (Prototype)
Corpus - AI Document Intelligence
- Python
- FastAPI
- Next.js
- PostgreSQL (pgvector)
- Claude API
- RAG
Overview
Corpus is a document intelligence system that lets users ask questions across a library of documents and get grounded answers with inline, clickable citations back to the source pages.
Problem
Answering questions across many documents by hand is slow, and generic LLM answers without citations are hard to trust or verify.
Solution
A retrieval-augmented generation pipeline that combines vector and keyword search, reranks the fused results, and prompts Claude to cite-or-abstain so every answer traces back to a specific source passage.
Key Features
- Hybrid retrieval: vector search (HNSW cosine) fused with PostgreSQL full-text search (tsvector) via Reciprocal Rank Fusion
- Cross-encoder reranking (bge-reranker-base locally, or Cohere Rerank)
- Claude API streaming generation with a cite-or-abstain prompting strategy
- Multi-turn conversation history
- PDF viewer that jumps directly to the cited page
Technology Stack
- Next.js 15
- TypeScript
- Tailwind CSS
- shadcn/ui
- FastAPI (Python)
- PostgreSQL + pgvector (Nile)
- OpenAI text-embedding-3-small
- Claude API
Architecture
Next.js frontend -> FastAPI backend -> hybrid retrieval (pgvector + Postgres full-text search) -> cross-encoder reranker -> Claude API for cited generation.
Database
PostgreSQL with the pgvector extension, hosted on Nile, storing both embeddings and full-text search indexes.
API / Backend
A FastAPI backend exposing retrieval and chat endpoints consumed by the Next.js frontend.
Authentication & Authorization
Not documented yet - see the project source for the current implementation.
Security
Not documented yet - see the project source for the current implementation.
Challenges
Fusing two different retrieval signals (vector similarity and keyword search) into a single ranked result set, then re-ranking that fused set before it ever reaches the generation step.
What I Learned
How hybrid retrieval and reranking noticeably improve citation accuracy over a vector-search-only pipeline, and how to structure a prompt so the model abstains rather than answers when it can't cite a source.
Future Improvements
The project's own roadmap notes an evaluation suite as the next step, along with testing the pipeline against real APIs and databases rather than isolated unit tests.
Live Demo
No public live demo for this project yet.