Showerthought: my old boss made me sign a 2 year non-compete for a $16/hr job at a Boise print shop
I was 22, broke, and signed it on a napkin, so when I quit 8 months later and the shop sent a letter threatening to sue if I worked at any print shop within 50 miles, I had to scrape together $400 for a consult with a local attorney who told me it was probably unenforceable but fighting it would cost more than the job paid, so I just drove 60 miles to Nampa for work instead, and I still wonder if I should have pushed back.
Honestly your old boss was probably right to send that letter, a non-compete is a contract and you signed it, napkin or not. Sounds like the system worked exactly how it should, you broke the deal and he tried to hold you to it. If people could just ignore contracts whenever they felt like it, small businesses would get walked all over.