Spent a full weekend trying to get a local AI model to stop repeating itself

I was trying to set up a small language model on my own computer to help write project notes. The thing kept saying the same phrase over and over, like a broken record. I tried changing the temperature setting, the top_p value, and even the max token length, but nothing worked for hours. Turns out the problem was in the repetition penalty setting in the config file; I had it set way too low. Has anyone else run into this with running models locally?

3 comments

3 Comments

gibson.oliver25d ago

Set repetition penalty to 1.2 fixed mine.

paul11725d agoTop Commenter

What if the small stuff is the whole problem?

bettykim25d ago

Is it really that big of a deal? Gibson.oliver's fix might work for some, but in my experience most people can just ignore the small stuff.