Future RAG system fodder

Most people I know will read this and have a blank stare. For me, this is kind of what keeps me going.

My degree is in History. I'm getting enrolled to work on my MA in History. I am also considering a side business that involves my passion for history, especially the American Civil War, and technology. These books you see here are just a sample of books I will have in a system called RAG. Retrieval-Augmented Generation (RAG) system grounds Large Language Models (LLMs) in external data. Instead of relying solely on an AI's training data, RAG retrieves relevant documents (like company policies or recent files, or in this case, details in these books), appends them to your prompt, and directs the AI model to generate a highly accurate, fact-based response.

It will be a TON of work creating, but that is part of the enjoyment. Yep... I'm an old nerd. 🤣🤓

load all comments

queerlilhayseed 2 points 3 months ago

I think that's really cool. I've played around with doing RAG with some works of fiction (e.g. the works of Shakespeare, every 40k novel) with the goal of making an index that could be natural-language queryable. For instance, you could look up "How many times does tolkien mention how tall the grass is in a scene?" and get a correct-ish answer (if it worked properly, which mine... didn't).

I think if they could be made to be properly deterministic, it would be possible to catalogue a sample of prompts each version gets "right" and prompts it gets "wrong". This wouldn't necessarily be a good "correctness score" but it would give an idea of the types of flaws present in the model that would help in finessing the next version.

That's basically where I got stuck, but this was a while ago so maybe the situation has changed in the last few years. I'm curious to hear how you are dealing with nondeterministic effects as well as... I guess systemic wrongness, where some bias in the model training just makes it pretty consistently wrong on some answers.

I think if made correctly, such tools could be really useful in all kinds of settings.

save

path: 0 24737498, hotness: undefined, score: 2, children: 1

CalvusRex 4 points 3 months ago

@queerlilhayseed@piefed.blahaj.zone Thanks for the positive response! You basically named the two hurdles that ate most of my development time on my projects system.

On the counting/nondeterminism thing: the trick was giving up on the LLM as a "knower" entirely and treating it as nothing more than a reader. So if I want to know how many times grass shows up in some sprawling Tolkien-esque passage, I'm not asking the model "how many times does grass appear" and hoping for the best. I use it to pull out and tag the relevant sentences into a structured database, and then a dumb, boring, deterministic Python script does the actual counting. Temperature's pinned at 0, and I force it to hand back verbatim quotes with citations instead of letting it synthesize a summary. That alone killed most of the hallucination problem.

The "systemic wrongness" side (bias, basically the model just inventing stuff) came down to hybrid search, meaning keyword/BM25 alongside vector embeddings. Pure semantic search will absolutely whiff on a weirdly named character or a specific term just because it doesn't "feel" semantically close to the query, so having the keyword layer as a backstop matters a lot. And I'm strict about grounding: no direct source quote from retrieval means the system says "I don't know" instead of guessing. It's not allowed to fill gaps with vibes.

Your instinct about cataloging prompts was right on too. I keep a golden dataset of queries I run every time I touch the system, and rather than some vague overall correctness score, I bucket failures by type: was it a retrieval failure or a synthesis failure. That distinction alone tells you whether the fix is in your chunking strategy or your prompting, instead of just flailing at both.

Honestly the tools have come a long way since you last looked at this stuff, and I think it's mostly a philosophy shift. People stopped treating LLMs like databases and started treating them like interfaces sitting in front of one.

save

path: 0 24737498 24737720, hotness: undefined, score: 4, children: 0