I've had this same problem in the past, so I know how hard it is. How I cut this off was to just forcibly change my thoughts whenever I thought someone was talking about me. Keep telling yourself that you don't care what they think, and soon you will begin to think this way. Worrying about what others think is a complete waste of time, and it will never accomplish anything.

Also, I'm curious, how old are you and what brought you to Japan?