Enterprise RAG Assistant

Purpose

Full-stack Retrieval-Augmented Generation application with document processing, semantic search, hybrid retrieval, and AI-powered chat with streaming responses

Architecture Decisions

• ▸Monorepo with separate frontend (Next.js) and backend (Express) services — clear separation of concerns, independent deployment, and team scalability
• ▸Supabase as unified platform for PostgreSQL, pgvector, Auth, and Storage — eliminates the need to wire together 4+ separate services and provides built-in RLS
• ▸In-process job queue over Redis/BullMQ — sufficient for single-server deployment with simpler operational complexity; can migrate to external queue later for horizontal scaling
• ▸Server-Sent Events (SSE) for chat streaming over WebSockets — simpler implementation, native HTTP compatibility, no connection management overhead for a request-response chat pattern
• ▸Hybrid retrieval (vector + keyword) as default strategy — neither pure semantic search nor pure keyword search provides adequate recall alone across technical documents

Technology Choices

Trade-offs

• ⚖In-process job queue vs Redis/BullMQ: Chose in-process for simplicity — sacrifices durability on server crash and prevents horizontal scaling of workers, but eliminates Redis dependency and operational complexity
• ⚖Supabase managed pgvector vs self-hosted: Chose managed for zero-ops — trades fine-grained index tuning control for automatic backups, connection pooling, and integrated auth
• ⚖Hybrid search vs pure vector: Chose hybrid for better recall — adds ~50ms latency from parallel keyword search but catches exact-match terms that embeddings miss (acronyms, codes, proper nouns)
• ⚖1000-token chunk size vs smaller: Larger chunks preserve more context for the LLM — trades retrieval precision (smaller chunks are more targeted) for answer quality (more surrounding context)
• ⚖SSE vs WebSockets: SSE is unidirectional and simpler — trades bidirectional capability (not needed for chat) for automatic reconnection and no connection state management

Scaling Considerations

• ↗IVFFlat index with 100 lists — performs well up to ~1M vectors; beyond that, HNSW index would be more appropriate for sub-linear search
• ↗Embedding batch processing (100/batch) with exponential backoff — respects OpenAI rate limits while maximizing throughput during document indexing
• ↗Context assembly with token budget (8000 tokens max) — prevents prompt overflow regardless of how many chunks are retrieved
• ↗Circuit breakers on OpenAI and Supabase calls — prevents cascade failures when external services degrade; 5-failure threshold with 30s/15s reset windows
• ↗Docker Compose with health checks — enables orchestrator-level restart and dependency ordering for future Kubernetes migration

Problem Statement

Enterprise documents — policy manuals, technical specifications, compliance guides, and operational procedures — are locked in PDF format across departments. Employees spend hours manually searching through hundreds of pages to find specific answers. Traditional keyword search returns too many irrelevant results, while asking a colleague means waiting hours or days for a response. The core problem is making institutional knowledge instantly queryable through natural language conversation.

Why This Approach

Fine-tuning a model on company documents was considered but rejected for three reasons: cost (re-training for every document update), freshness (fine-tuned models become stale as documents change), and hallucination control (fine-tuned models can still fabricate answers without source attribution). RAG solves all three — documents can be re-indexed in minutes, the model always grounds answers in retrieved context, and every response includes citations linking back to source chunks. This makes RAG the only viable approach for an enterprise knowledge system where accuracy and traceability are non-negotiable.

Challenges Encountered

• ◆Docker container-to-host networking — the frontend and backend containers needed proper network configuration to communicate, requiring careful Docker daemon and platform-specific networking setup
• ◆Chunk size calibration — too small (200 tokens) lost critical context between paragraphs; too large (2000 tokens) diluted retrieval precision; the hierarchical paragraph → sentence → word splitting with 1000-token max and 200-token overlap was the result of testing dozens of configurations
• ◆Follow-up question handling — "What about the second section?" is meaningless without conversation history; implemented an OpenAI-powered query rewriting step that rewrites follow-ups as standalone questions before retrieval
• ◆Streaming response persistence — the SSE stream sends tokens in real-time, but the complete answer, citations, and token usage must be persisted after the stream ends, requiring careful async coordination between the stream consumer and database writes
• ◆Embedding rate limits — indexing large document sets triggers OpenAI rate limits; solved with batch processing (100 chunks/batch), exponential backoff, and the in-process job queue with configurable concurrency

Performance Improvements

• ↑Hybrid retrieval (60% vector + 40% keyword) improved recall by catching exact-match terms (acronyms, model numbers, proper nouns) that pure vector search misses
• ↑Query rewriting eliminates the "What about it?" problem — without it, follow-up questions retrieve irrelevant chunks, degrading answer quality by 40-60%
• ↑Context deduplication and document-ordered assembly prevents the LLM from seeing the same chunk twice and maintains reading order for coherent answers
• ↑Circuit breakers prevent cascade failures — when OpenAI is slow, the system fails fast with 503 instead of accumulating timeout connections
• ↑Token-aware context assembly caps the prompt at 8000 tokens regardless of retrieval count, preventing prompt overflow and controlling OpenAI API costs