People can be shallow. Shallow people judge their own clothes in terms like "I wouldn't be caught dead in that" or "this is me". Shallow people also think everyone else is shallow. So, whether you're wearing something they would or wouldn't wear, they'll project their judgment of your clothes onto you personally. Too bad for them.