posts/llama_index_rag

[utterances-bot]

Running Large Language Models (LLMs) locally for Retrieval-Augmented-Generation (RAG) Systems with full privacy – Hans Dembinski’s blog

https://hdembinski.github.io/posts/llama_index_rag.html

[HDembinski]

Update 1: There is a new library from the authors of the popular pydantic type validation library, called PydanticAI, which looks very promising.

Update 2: This article does not discuss the size of context windows. The way to provide a LLM with input is to put the text in its context window. The earlier LLMs von 2024 typically had context windows that are 4-8k token large (a token is roughly a word), which is not a lot, so it was important to pinpoint relevant information and feed only small snippets of text into the machine. Due to advances in technology, even small LLMs are now trained with context windows as large as 128k token, which alleviates the need to locate relevant text pieces with pinpoint accuracy, making RAG easier. Unfortunately, with a 8 GB GPU, one cannot exploit such large context windows.