Ricers indeed. -O2 or -O3 should be enough for everyone.
I have to write MPI code for a cluster (this thing specifically: http://www.vpac.org/content/systems_and_support/facility/linux_cluster.php), so obviously I'm looking for every little speedup possible. Compiler switches are a simple way to squeak out some further gains, so I've done a fair bit of testing on them.
At least for GCC 3.2.*, switches other than -O2 (and maybe -march) vanish into the noise. They're just not worth considering unless you're doing something unusual. Of course, if it's speed you need, you don't use GCC to begin with. You'd use Intel or IBM.
I could really use a -fomfg-fast-speed though.