Claude 3.5 just crushed my coding benchmark in 30 seconds flat

I spent 2 hours last Friday debugging a Python script for data parsing. Handed the same broken code to Claude 3.5 Sonnet. Fixed it in under a minute. The new model dropped on June 21 and I've been testing it all week. Handles way more context than GPT-4 too. I gave it a 50 page PDF and it summarized key points without cutting out. Has anyone else noticed how much faster the output feels since Anthropic released it?

2 comments

2 Comments

elliot_grant384d ago

I feel you on that legal contract thing, man. I tried feeding Claude a 40 page lease agreement a few days ago and it straight up missed a whole section about renewal terms. Like it just skipped over the paragraph completely. The speed is nice and all but if it's going to drop important chunks of text that's a serious problem for real work. I actually had to go back and double check everything it pulled out because I lost trust after that. GPT-4 has been more consistent for me too when it comes to keeping track of all the little details across long documents. Kinda feels like Anthropic pushed this update out a bit too fast before it was ready.

mason_lee4d ago

Ngl, I had a totally different experience with the June 21 update. I threw a 30 page legal contract at it for clause extraction and it missed 3 major sections. Output speed was fine but accuracy took a hit compared to the previous version. Honestly, GPT-4 still handles my multi-step coding tasks better without losing track of the original requirements.