This is really two separate issues. Or three, depending on how you count.

First, a bug: Firefox digs around for a favicon even if one is supplied in a <link> tag. That's obviously wrong, but is simple to fix.

Second, an opinion: loading a favicon if none is specified. Some people seem to oppose this on principle, but that seems unnecessarily dogmatic. It may not be in any official standards, but it is a de facto standard, so you might as well do it, because the users benefit.

Third, a bad behabiour: Firefox keeps requesting the favicon on each page hit if one doesn't exist. This is obviously a side effect of the cache implementation: a 404 is not cached. This makes sense for normal pages, but the favicon handling should be changed to rememeber 404s and not keep re-requesting.