I finally hit 95% accuracy on my image generator after 47 tries - but I think benchmarks are lying to us

So I spent 3 weeks tweaking a stable diffusion setup and got a 95% match rate on a custom test set of 200 photos. Everyone in the Discord was hyping it up. But then I realized my test set was full of the same boring angles and lighting. Real world use dropped to like 60%. These benchmark numbers people brag about feel fake when you actually use the thing in the wild. Anyone else notice their model scores don't match reality?

3 comments

3 Comments

james5331mo ago

47 tries for 95% on the same 5 pictures lol classic.

the_michael1mo agoMost Upvoted

47 tries is a lot but honestly this is one of those times where slowing WAY down helped me, @james533. I was stuck around the same spot and realized I was rushing through the id verification. Taking an extra 3 seconds to line up each box and wait for the green indicator cut my tries by a ton. Also make sure your lighting is consistent, overhead lights cause weird shadows that mess with the AI. Little tweaks like that got me through in under 10 tries after that.

the_jason1mo ago

@james533 have you tried turning your phone sideways? I was doing the same thing until I realized the AI scans wider when you hold it landscape instead of portrait. Made the box alignment way easier, the corners actually fit without me having to tilt the phone weird. Also check if your camera lens is clean, smudges mess with the contrast and the AI gets confused on where the edges are.