I really don't know what to say :(?

So today i was in P.E and then we were playing tag football so we had jerseys on. They were really weird, but I turn to my friend/team mate and say, "Haha, these are sooo attractive!" He agreed and we both started luaghing together. Then this other boy comes up to me and says, "Oh yeah, they're just as attractive as those huge t-shirts you wear!" The only thing I could do is comment on his semi-uni-brow. But I have a serious self image problem and I wear baggie shirts because I don't like my and I also think they are comfortable. :( What should I say to him, because it really hurt me. He isn't very popular, I am sorta, but not really.