What Is RAG (Retrieval-Augmented Generation), and How Is It Created?
Language models such as GPT can generate texts, summaries, and analyses at an impressive level. However, their limitation is that they operate based on the data on which they were trained, meaning they do not have access to your current documents, projects, or internal materials. This is where RAG, or Retrieval-Augmented Generation, comes in: an approach that combines the power of a language model with your private knowledge base.
In many cases, it is enough to upload your files to a given language model. NotebookLM allows you to manage multiple files, while the Projects mode in ChatGPT allows multiple threads to use the same files. For simple tasks, creating a RAG is not necessary; however, where we want to use hundreds of files, more complex solutions are essential.
Thanks to RAG, you can create your own AI assistant that not only “knows” general knowledge but can also answer based on your data. It can analyze company reports, project documentation, notes, and even PDF files, generating responses based on their content.
Table of Contents
-
RAG vs. Files in a ChatGPT Thread, Files in Projects, or in NotebookLM
-
Accurate search – selecting the embedding model and the value of k
.
What Exactly Is RAG
Bridging Two Worlds
RAG is a combination of two elements:
- Retrieval — the component responsible for searching for information.
Instead of retraining the model from scratch, RAG uses a search engine (e.g., a vector database) to find text excerpts that best match the user’s question. - Augmented Generation — the generative component.
The language model (e.g., GPT-4 or Claude) receives these excerpts as context and uses them to generate a response.
As a result, RAG does not “guess” like a classic chatbot; instead, it responds based on real data sources that you control.
How it works step by step
- The user asks a question (e.g., “What were the sales results in Q3?”).
- The system searches for the most relevant excerpts from your documents.
- The language model combines this data and generates a response in natural language.
Thanks to this, AI can act as your private analyst, consultant, or office assistant.
RAG and Files in a ChatGPT Thread, Files in Projects, or in Notebook LM
What are ChatGPT projects?
Projects in ChatGPT mode allows you to assign a set of files to a specific project so that the model can use them across multiple threads. This is a convenient solution for users who want constant access to their own materials without having to upload them again. The model can analyze documents, search for information within them, and create responses based on their content, making work with larger datasets much faster and more consistent.
How Does RAG Work?
The Retrieval-Augmented Generation (RAG) system works in a similar way, but in a much more flexible manner. In this approach, data is stored in an external database (e.g., Weaviate, Milvus, Pinecone), which enables semantic-level searching. The language model then retrieves only those text fragments that are most relevant to the user's question and generates an answer based on them. This allows the knowledge base to be dynamically updated and expanded without the need to retrain the model.
Key Differences Between Projects and RAG
Data Structure and Control
In ChatGPT projects, data is stored in OpenAI’s closed environment, and the user has no direct influence over how it is indexed. In a RAG system, control is comprehensive—you can decide how data is stored, processed, and searched, as well as add your own metadata and filtering rules.
Scalability and Integrations
Projects are ideal for working with small document collections, whereas RAG can handle thousands or millions of files and integrate with external systems such as CRMs, knowledge bases, or API services.
Data Update
In projects, files must be updated manually. In RAG, this process can be automated so that the knowledge base is continuously supplemented with new content.
Model behavior customization
RAG allows you to control how context is retrieved (e.g., semantically, lexically, or by source), and even to apply filters and weights to individual documents. In ChatGPT projects, this process is completely hidden from the user.
When is it worth choosing which solution?
Situation | Better solution |
Want to quickly combine several files and use them in conversations | ChatGPT Project |
Need your own knowledge system with updates and integrations | RAG |
Do you care about privacy and control over your data | RAG |
Want to conveniently analyze several documents without configuration | ChatGPT Project | .
Why It Is Worth Building Your Own RAG System
Data currency
RAG enables the use of the latest documents without the need to train a new model. Simply add a file, and the system will immediately start using it.
Privacy and Control
Your data remains under your control—it is not sent to a public model or used in its training. This is important if you work with confidential information.
Flexibility
The system can be adapted to any context: it can serve as a customer service assistant, a tool for searching technical documentation, or a knowledge base for the team.
How to Prepare Data for RAG
Collect and organize documents
Start by gathering the materials that are to form your knowledge base. These may include:
- PDF and DOCX files,
- text notes,
- project documentation,
- instructions, reports, and emails.
It is best if documents are organized thematically and logically. Each should have a title and contain coherent content — this makes it easier for the system to divide them into fragments.
Divide documents into smaller sections
RAG operates on so-called chunks, which are text fragments typically 300–1,000 words long. Why? Because models handle shorter contexts better. If you have a lengthy report, it’s worth dividing it into chapters or sections.
Standardize the data format
Before importing, it is worth converting everything to a simple text format (e.g.
.txt
lub
.md
). Some tools can automatically process PDFs or Word documents, but plain text is always the most reliable.
How Vectorization Works
What does it mean to “turn text into vectors”
Vectorization is the process of transforming text fragments into numbers (vectors) that reflect the meaning of the content. This enables a computer to “understand” that sentences with similar meanings have close numerical values.
Example:
“How does invoicing work in the system?” and “What does the invoice issuance process involve?”
After vectorization, both sentences will have very similar representations.
Vector Databases
Such vectors are stored in specialized databases called vector databases. The most popular ones include:
- FAISS (Facebook AI Similarity Search) — a fast and free open-source library,
- Weaviate — a platform with a ready-to-use API,
- Milvus — a high-performance database for large datasets,
- Pinecone — a cloud-based solution, often used in production applications.
Building Your Own RAG — Step by Step
Step 1. Prepare the data
Collect the documents, organize them, and divide them into chunks. If you have, for example, 20 files of several pages each, the system will convert them into dozens of text “chunks” that will be easier to search through later.
Step 2. Vectorize the data
Use one of the vectorization tools, e.g.:
- LangChain — a Python framework that automates the entire process,
- LlamaIndex (formerly GPT Index) — ideal for simple integrations,
- OpenAI embeddings API— enables vector creation without having to install anything locally.
The vectorization process looks roughly like this:
- You load the document.
- You divide it into chunks.
- You process each chunk through an embedding model (e.g.
text-embedding-3-large
).
- You save the result in a vector database.
Step 3. Create the retrieval layer
This is a key element. When a user asks a question, the system must find the most relevant passages in your database. To do this, the search is performed by comparing the question vector with the vectors of your documents. The results are sorted by semantic similarity.
Step 4. Connect everything to the language model
Now you are integrating retrieval with a generative model.
The simplest way:
- you provide the retrieved excerpts as context to the model,
- the model generates a response using this knowledge.
In LangChain or LlamaIndex, this can be done with just a few lines of code. Example pseudocode:
query = "How does the reservation system work?" context = retriever.get_relevant_documents(query) response = llm.generate_answer(context + query) print(response)
How to Add New Data to the System
When you want to add new files, you don’t have to rebuild everything from scratch. Simply vectorize the new fragments and add them to the database. The existing data remains untouched, so the process is fast and scalable.
How to Test Your RAG System
Simple Functional Test
Ask questions whose answers can be found in your documents, e.g.:
“When was the new policy introduced?”
“What does the onboarding process look like?”
If the model responds accurately and cites content from your database, everything is working correctly.
Assessment of Result Quality
Worth checking:
- whether the model selects the correct passages (so-called recall),
- whether it generates answers consistent with the documents (so-called precision),
- whether it does not “make up” data when it is not in the database.
How to Maintain and Develop RAG
Regular Updates
The more frequently you add new documents, the more up-to-date the answers are. You can set up an automated process that scans the directory once a week and vectorizes new files.
Monitoring
It is a good practice to record users’ questions and whether the answer was accurate. This makes it possible to gradually improve the system, for example by adding missing documents or improving indexing.
Scaling
If the database grows, you can move from a local FAISS installation to a cloud-based solution, such as Weaviate Cloud or Pinecone. This will keep searches fast, even with thousands of documents.
Common Mistakes When Building RAG
Text Fragments That Are Too Large
If the system processes very long blocks (e.g., entire chapters), the model may lose the meaning and return imprecise answers. It is better to divide texts into smaller chunks.
Low-Quality Data
Messy, outdated, or duplicate content reduces search quality.
Make sure your data is consistent and clear.
No tests
Without testing the system, it is difficult to assess whether it is working correctly. It is worth creating several sample test questions and checking them regularly.
.
Example Applications of RAG
- Company assistant — answers questions about regulations, products, and procedures.
- Technical support system — automatically searches documentation and provides ready-made solutions.
- Tool for sales teams — instantly searches for information about offers, customers, and contact history.
- Internal knowledge base — transforms scattered files into one coherent answer system.
Potential Problems When Using RAG
1. Quality and consistency of source data: If the documents you upload are inconsistent, outdated, or poorly organized, the model may have difficulty selecting the appropriate context. As a result, the answers may be less accurate. This means that the quality of what you upload truly affects the quality of the answers.
2. Proper tuning and fine-tuning of the system: Sometimes it is necessary to experiment a little with the search parameters or with the way the model selects context itself. If these settings are not properly configured, RAG may not return the most relevant passages.
3. Performance and resources: If you have a very large database, searching may be somewhat slower, especially on less powerful hardware. Sometimes optimization is necessary, or you may need to use more powerful infrastructure if there are truly a lot of these files.
4. Update Issues: Although adding new files is fairly simple, it is important to remember to update the database regularly. If you forget to add new data, the model may rely on outdated information, which can reduce the quality of its responses.
5. Integration Complexity: At first, if someone is not accustomed to such tools, the setup itself may seem somewhat complicated. You need to understand how a vector database works and how to connect it to the model. This can be a challenge at the very beginning.
What Does Good Data Quality Mean for RAG
In RAG systems, quality does not result solely from the power of the language model. It is the outcome of four equally important elements: how data is split, the choice of embedding model, prompt quality, and metadata consistency. Each of these directly affects the precision of responses and whether the system truly understands your data.
.
Proper Chunking While Preserving Context
Good text segmentation (so-called chunking) involves dividing documents in such a way that the model does not lose the meaning of the whole. Fragments should not be too long, because the model may lose the sense, nor too short, because they will lose context.
A practical rule is: approximately 300–1000 words per fragment, with slight content overlap (so-called overlap) to preserve the continuity of the sentence and logic.
For example, if a chapter ends with the sentence “In the next step, we will discuss the payment process…”, this fragment should also begin in the subsequent chunk. This enables the system to logically combine this information in its response.
Accurate Search – Selecting an Embedding Model and the Value of k
How a RAG system finds context depends on the quality of embeddings, that is, representations of text in the form of numbers. The embedding model must be suited to the language, type of documents, and level of detail.
- For technical data, models of the type work well
text-embedding-3-large
.
- For simpler texts, a smaller model is sufficient; it operates faster and costs less.
The k parameter determines how many chunks the system retrieves from the database to prepare a response. A k value that is too low may cause important contexts to be omitted, while a value that is too high may overload the model with excessive data. Optimal values are k = 3–7 for small databases and k = 10–20 for large ones.
Transparent Prompt with Citation Instructions
Prompt is the framework of a conversation between you and the model. If it is imprecise, even the best data will not guarantee quality. A good prompt in a RAG system should:
- Clearly require that the answer be based only on the provided context.
- Require source citations, for example: “Provide the answer solely based on the context. If there is insufficient information, say: ‘no data’. At the end, list the sources in square brackets.”
- Be simple and repeatable, so that the model does not interpret the instructions differently each time.
As a result, the responses are consistent, reliable, and it is easy to verify which documents they come from.
Consistent Metadata – Document Title, Chapter, Version
Metadata is hidden information that helps understand a document’s context. Well-designed metadata is key to the quality of search and interpretation. Every document should have:
- Title – clear and descriptive, e.g. “Privacy Policy – version 2025.01”.
- Chapter or section – indicating the subject area (“Human Resources”, “Reservations”, “IT Security”).
- Document version – allowing the system to reject outdated data and prioritize the latest version.
Metadata consistency makes it possible to filter results and avoid incorrect answers, for example when two versions of the regulations differ by only one paragraph. In practice, a well-designed metadata structure means that RAG becomes not only a search engine, but also a knowledge version control system.
Summary
RAG is one of the simplest ways to transform static documents into an intelligent knowledge system. It does not require model training or extensive infrastructure — good data organization and a few open-source tools are enough.
Once you launch it, it becomes your own private ChatGPT, which knows all your materials, remembers the context, and always uses the latest information.