Fine-tuned Llama 3.1 vs GPT-4o mini for ticket routing, the difference was huge

Ran a side by side test last Friday with 500 real support tickets from our system. Fine-tuned Llama 3.1 on about 2k historical examples and it hit 91% correct category picks, GPT-4o mini got 78% with the same prompt. The custom model also ran locally so each call cost us like 0.2 cents vs 1.5 cents per API hit. Anyone else seen fine-tuning beat the big models when the task is narrow like this?

1 comments

1 Comment

the_william8d ago

Kinda makes sense when you think about it. Specialized tools beat general ones all the time. Like, I got a decent chef's knife that does everything okay, but my old paring knife is way better for peeling fruit and small stuff. Same idea here. When the job is really narrow, a model built for just that one thing is gonna beat a big general one. Plus being able to run it yourself without paying per call changes the whole math of what you can do with it. It's like the difference between hiring a handyman or calling a specialist, the specialist knows the exact job better even if the handyman knows a bit of everything.