Like some others have said, it looks like a UK culture thing because it's completely normal and very usual for people of all ages to wear football shirts as casual clothes. Personally, I believe it's completely fine, and actually what I find it odd (and rather snobbish) is to think it's odd to wear them. Also, I think they can actually look good and be a nice fashion choice, but for that to happen I believe there are 2 factors: the shirt needs to not be too tight for one's body (which applies to any other clothes too), and the design needs to be "nice" (although this is obviously subjective), which to me means not too garish and as few and small commercial logos as possible (if possible, none). For example, some of the Roma home kits from the mid 10s would look good on any reasonably fit person.