Here's how I look at it.
When I was in college, it was during the whole throwback jersey phase (Early '00s)that seemed to be permeating around (rap videos, celebs wearing, etc...). My roommates and I were definitely into it and, on more than one occasion, we went out rockin' jerseys. I had absolutely no problem with it back then. You aren't going to see me at a bar or out on the town in one now, though.
However, I still rock jerseys sometimes if I'm headed to a game at Fenway or Gillette. (Retired the Pierce tanktop from the Garden). Beyond that, I might rock something if we're having a house party at my place or invited in a Beirut tourney (its a sport, i won't argue this.

)
Lastly, I'm probably not going to buy someone's jersey whose significantly younger than I am. It just feels weird to me.