I finally hit 95% accuracy on my image generator after 47 tries - but I think benchmarks are lying to us

So I spent 3 weeks tweaking a stable diffusion setup and got a 95% match rate on a custom test set of 200 photos. Everyone in the Discord was hyping it up. But then I realized my test set was full of the same boring angles and lighting. Real world use dropped to like 60%. These benchmark numbers people brag about feel fake when you actually use the thing in the wild. Anyone else notice their model scores don't match reality?

2 comments

2 Comments

james53315h ago

47 tries for 95% on the same 5 pictures lol classic.

the_michael14h ago

47 tries is a lot but honestly this is one of those times where slowing WAY down helped me, @james533. I was stuck around the same spot and realized I was rushing through the id verification. Taking an extra 3 seconds to line up each box and wait for the green indicator cut my tries by a ton. Also make sure your lighting is consistent, overhead lights cause weird shadows that mess with the AI. Little tweaks like that got me through in under 10 tries after that.