50% of my test runs broke in production, not the training data

I ran a batch of 40 edge cases through my fine-tuned model last week. 20 of them failed when I swapped the API endpoint to the real server. The weird part? They all passed in the sandbox. Same prompts, same weights, same version. Turned out the issue was my scheduler, it wasn't sending the new system prompt with every request. I spent 3 days chasing hallucinations that were never there. Has anyone else burned time on deployment config instead of the model itself?

1 comments

1 Comment

valsullivan17h ago

Question whether the sandbox ever really matched production if the system prompt was missing all along.