TIL my simple chatbot idea needed way more training data than I thought

I wanted to make a bot that could answer questions about my town's bus schedule, figured it would take a weekend. I mean, the info is all online, right? I spent two whole weeks just cleaning and formatting the timetable data from the city's PDFs before the model could even start learning the routes. Has anyone else had a project balloon in time because of data prep?

3 comments

3 Comments

willow_ellis3mo ago

Tell me about it. Data prep always takes ten times longer than you think it will.

the_uma3mo ago

Right? @willow_ellis, what part of the prep always ends up being the biggest time sink for you?

emeryfox1mo ago

Wait, is nobody else annoyed by how the raw data always comes in different formats from different sources, even when they say it's standardized? That's the part that kills me every time, trying to wrangle everything into one clean layout.