Evidence-based clinical question answering using retrieval-augmented generation (RAG) and open-source large language models

Abstract Large language models (LLMs) are being investigated for clinical decision support tools, but can produce unreliable outputs, including hallucinated information, overgeneralized answers, and responses not clearly derived from authoritative medical sources. To ground responses in high-quality medical texts, researchers developed Retrieval-Augmented Generation (RAG). In this work, we show that, within a structured, single-textbook corpus, embedding model choice has significantly greater impact on RAG system reliability and performance than chunking strategy. The system compares twelve embedding models—general-purpose and biomedical-specific—under token-level and character-level chunking, prompting the LLM to answer from retrieved snippets. We evaluate retrieval effectiveness using rank-based metrics and generation quality using standard measures across 232 questions. General-purpose embeddings, particularly Jina-Embeddings-v3 and BGE-base-en-v1.5, outperformed biomedical-specific encoders in both retrieval and generation. Across both strategies, higher retrieval precision translated directly into improved lexical and semantic alignment with gold references. Under our operational definition requiring explicit evidence spans, 35.6% of answered questions lacked supporting evidence; our best configuration (BGE-base-en-v1.5, token-level) achieved the lowest rate at 26.1%—an evidence-grounding measure, not a guarantee of correctness. These findings highlight the central role of retrieval quality in clinical RAG systems, showing that open-source, evidence-driven architectures offer a transparent alternative for textbook-based clinical QA settings.