19
That time a Databricks engineer told me to batch my API calls and I kept ignoring it
I was doing 10,000 individual calls to the GPT model for a project and wondering why it took 3 hours, then a guy I know from a meetup said to batch them into groups of 20 and my run time dropped to 11 minutes flat. Has anyone else had a simple change like this totally reshape how they work with these models?
2 comments
Log in to join the discussion
Log In2 Comments
jade31mo ago
Batching changed my entire workflow honestly. I was doing 50,000 individual calls for a text analysis thing and it was taking like 6 hours. Grouped them into batches of 30 and suddenly it was done in 15 minutes. The API rate limiting is no joke, it's basically punishing you for not grouping requests together. Now I just batch everything by default even for small stuff because it's become a habit.
8
jana_henderson521mo ago
That's a wild improvement. Going from 6 hours to 15 minutes is the kind of difference that makes you wonder why you didn't batch sooner. The API rate limiting really does force your hand on this stuff. It almost feels like they want you to batch or they'll just slow you down on purpose. I bet you saved a ton of money on compute costs too.
0