I have a friend who looks damn good in a fedora (he's your typical skinny, well-dressed gayboy). I tried in on once, and I also thought I could pull it off. Probabaly because I was wearing clothes that went well with the look (white button-down shirt; black vest; black pinstriped pants).
I feel that it's topical to mention that I'm a girl here.
It's a matter of making the rest of your outfit work with the hat. If it looks like you put a bit of thought into it, you're golden.