01 / Overview
What it is and the problem it solves
Querying heterogeneous corporate documentation required reducing manual search without losing the evidence needed to verify each answer.
Bachelor’s thesis and functional prototype developed with real corporate documentation. It is not presented as a RAG system currently operating in production.
02 / How it works
How the system works
- 01
Page-by-page visual extraction into hierarchical Markdown for documents with heterogeneous layouts.
- 02
Semantic chunking with section paths, pages, provenance, security level and representations tailored to lexical and vector search.
- 03
Persistent BM25 and ChromaDB indexes, parallel retrieval, RRF fusion and reranking of the strongest candidates before generation.
- 04
Numbered context and streamed answers with citations, sources, scores and visible diagnostics in a local interface.
03 / My contribution
What I did
I designed and implemented the complete pipeline: page extraction, cleaning, metadata, chunking, indexes, fusion, reranking, generation and validation.
Main decisions
- Separate offline processing from online querying so every stage can be rebuilt and audited.
- Combine BM25 and ChromaDB through Reciprocal Rank Fusion instead of relying on one retriever.
- Keep explicit provenance on every chunk and apply Jina Reranker before generation.
04 / Validation and outcome
How I checked the result
The suite covers extraction, metadata, cleaning, chunking, BM25, embeddings, hybrid retrieval and generation using simulated clients. I also prepared question batches to inspect retrieved evidence and compare BM25, embeddings, RRF and reranking. The figures demonstrate pipeline execution; they are not presented as a universal accuracy guarantee.
Project outcome
- Reproducible pipeline with persistent artifacts and selective reindexing.
- Grounded answers with identifiable sources and a local query interface.
- Automated validation of contracts, retrieval and system behavior.
05 / Lessons learned
What I learned during development
The final quality of a RAG system does not depend on the LLM alone. Poor extraction, a table split without context or weak retrieval constrains everything downstream. Designing traceability from ingestion was as important as choosing the generative model.