Prompt injection test caught my RAG pipeline hallucinating bad data

I ran a simple adversarial prompt through my internal document chatbot and it pulled a fake contract clause from nowhere, citing a 2023 revision that doesn't exist. The fix was adding a source verification step that cross-checks each retrieved chunk against the original PDF hash. Anyone else seen their vector DB confidently invent facts when users phrase questions a certain way?

1 comments

1 Comment

skyler_anderson6510d ago

The part where it "confidently invent[s] facts" hit close to home. I had one where someone asked about our return policy in Q3 of last year, and the bot made up a whole section about a 30 day window that never existed. We actually keep anniversary dates in the file names, so I matched it up and the source wasn't even in the corpus. The PDF hash check sounds smart, I might steal that. It's wild how a slightly different question flips the thing from reliable to full on fiction mode.