Loading…
3
Shoutout to the guy on r/LocalLLaMA who said stop tuning prompts and just fix your eval set
Spent 3 weeks last month fighting a support bot that kept making stuff up about return windows, and I rewrote the system prompt maybe 40 times with zero real gain. Then I read a comment buried in a thread that said my prompt was fine and my eval set was garbage, which stung but turned out to be dead on. I built 60 hand-labeled test cases from real chat logs and found out 70% of my failures were the retriever grabbing the wrong doc, not the model. Fixed the chunking and accuracy jumped way more than any prompt tweak ever did. Anyone else get suckered into prompt chasing before they looked at their data?
1 comments
Log in to join the discussion
Log In1 Comment
gavin_allen487d ago
What did your eval set look like before you rebuilt it? I had the same thing happen with a docs Q&A bot and it was chunking all along. My chunks were splitting tables in half so the model got "30 days from" and nothing else, then it guessed the rest. I spent two days rewriting instructions about not making things up before I actually read the retrieved text. Once I switched to splitting on headers and keeping rows together, made up answers mostly went away. My test set was like five questions I wrote from memory too, which of course all passed.
1