The manager who said it's not job-related is crazy... woodworking shows you can figure things out and finish projects. That's useful anywhere, even an office.
Is it really that deep though? Like yeah it shows you can build stuff but so does putting together ikea furniture. Sometimes hiring folks just read way too much into tiny details. Maybe one guy had a cool birdhouse once and got nostalgic, the other just wanted to stick to the script. Seems like a coin toss more than some big life lesson.