Tell him about your feelings of loyalty. If need be, neither of you should make a move.
Whatever you do, talk about it.
I've been in love with a girl, and two of my friends went out with her. The first one talked with me longly about that, and I hold no grudge against him.
The other one just seduced her without even talking to me about it, when all my other friends warned him that i had a crush on her. I'm really having a hard time even facing him now.
And as the blunt proverb says: bros before hoes.