Well if you ever played the game Red Steel for the Wii, the japanese Yakuza call you a "stupid Gaijin" when they are a japanese Yakuza weilding a Katana, and speaking english with an accent, in California, I believe they are the gaijin's.

The word Gaijin isn't used just for a non-japanese man living in Japan, it is just a Japanese word for saying Outsider, it could be used for a person that joins a Clan match and he is in neither clan. So you don't really need to use the word Gaijin just for a non-japanese man, its just a term for outsiders in general of any race or group.