I'm sort of in the same situation as you, except that I'm a girl.
I really don't like people coming up to me and invading my personal space. That is, if they sit unnecessarily close, attempt to hug me, or just hover around within say 30 cm. I always knew this was a self-confidence issue, and now I'm gradually becoming more open, and consequently more comfortable with physical contact.