Yeah I also noticed that setting the cache size doesn't really work. Too big and they are problems. Too small and it's a waste. My solution is to empty the cache regularly. But sometimes you click the Empty button and nothing happens, the cache is still there. In those cases I go to the \cache directory and delete all files myself.

Another source of problem is that file that grows very fast in Mozilla\Firefox\Profiles\[number]\downloads.rdf.
When it goes over around the 1,500,000 bytes mark then Firefox gets slower and slower to display.
To empty it: Tools > Downloads > Clean up.

One file you can safely delete is XUL.mfl, Firefox will recreate it on its next run. Sometimes deleting it solve problems.