Vent: My AI training session ate 12 hours of compute time for nothing

I was working on fine-tuning a small language model for a client project last Tuesday. Set it up on a cloud GPU instance, let it run overnight. Came back the next morning and found out the dataset had a formatting error in the first 100 rows so the whole thing trained on empty labels. Wasted $48 in credits and half a day. Has anyone else had a silent data bug waste their compute budget?

2 comments

2 Comments

iris_mason881mo ago

Validate your data before training, seriously just write a quick script to check first.

james5331mo ago

Ugh yeah I've been burned so many times by my own lazy scripts that I call them "trust me bro" datasets now. @iris_mason88 you're right though, it's painful how many hours I've wasted debugging junk data instead of just spending 10 minutes on a validation script first.