Hybrid RAG for corporate documentation

Corpus preparation takes place before queries. Each PDF is extracted page by page using an OpenAI model and normalised to Markdown; the system records metadata and a document fingerprint, and avoids reprocessing duplicates when configured to do so. The content is then split into chunks with section and page context, and lexical and vector indexes are prepared. At query time, both retrievers produce candidates that are fused and, when enabled, reranked before building the generator's context. The answer displays references to the passages used.

The architecture separates document preprocessing from online retrieval. This makes intermediate outputs inspectable and reusable without repeating ingestion for every question. The implemented input format is PDF: pages are represented as images for multimodal extraction by default, and sending each page as a PDF is also supported in the configuration. OpenAI also handles text cleaning and structuring. The code validates extracted pages, records issues and preserves metadata, including hashes, pages, sections and document classification.

Chunking analyses the Markdown hierarchy and recognises structures such as tables, frequently asked questions and clauses. It combines small blocks where appropriate and splits large ones, with a target of approximately 500 tokens, a maximum of 900 and an overlap of 80. Each chunk retains its original content and representations tailored to lexical and vector search, together with document references. This adapts indexing without losing the link to the source.

Retrieval combines BM25, which favours terminology matches, with text-embedding-3-large embeddings persisted in Chroma, which support semantic similarity search. A deterministic Spanish tokenizer normalises accents and preserves codes and business terms. Since the scores from the two methods are not directly comparable, Reciprocal Rank Fusion (RRF) combines their ranking positions. The jina-reranker-v3 reranker, accessed through an API, can refine the candidate list before selecting the final evidence; it can be disabled. Generation uses the OpenAI Responses API, with gpt-5-mini as the configurable default model. The context numbers chunks and their sources; if the configured minimum number of sources is not met, it skips the model call and returns an insufficient-evidence response.

Interaction is available through a CLI and a simple web interface served locally. Persistent intermediate contracts and component separation make models and parameters easier to replace or adjust. Although configurations and unit tests with simulated clients are available, this version does not provide a quantified comparison that would establish superiority over other retrieval approaches or models. These choices are presented as implementation decisions.

Variation in PDF formatting makes it difficult to preserve tables, headings and page references. The workflow extracts content page by page, normalises it using structural instructions, and records warnings and quality checks. The chunker uses headings and block types to preserve context and avoid search depending on unstructured plain text. To make answers reviewable, identifiers, sections and pages accompany chunks throughout retrieval and are carried into the displayed sources.

Lexical and semantic search produce different rankings, and classified documents require additional restrictions. RRF combines rankings without adding incompatible scores; metadata and document-level filters narrow the results. These levels are filtering metadata: the implementation does not provide authentication, per-user authorisation or access auditing.

The repository includes unit tests for processing, cleaning, chunking, BM25, embeddings, fusion, filters, reranking, generation and the interface. Several isolate the logic through simulated clients, so they check contracts and controlled behaviour rather than production answer quality. Question sets and a retrieval evaluator are also available, calculating Hit@k and MRR from expected chunks or documents. This version does not provide verified aggregate metrics with an identified dataset, split, configuration and evaluation method; consequently, it does not claim a particular quality level or a quantitative improvement for the system.

The inspected version has a functional PDF-to-answer workflow with sources, persistent indexes and local query options; it depends on external OpenAI services and, when reranking is enabled, Jina. Formats other than PDF and multi-user operation are outside the demonstrated scope. The most valuable next improvements would be to version evaluation datasets and record query-level results to compare BM25, embeddings, RRF and reranking; measure retrieval, citation fidelity and answer grounding separately; and design authentication and authorisation before handling sensitive documents in a shared environment. Validation of tables and incomplete-extraction cases should also be expanded.