Pfff, don't forget that Quake 3 is eight years old. 256MB will handle it just fine if you're not running other things in the background. That being said, more is always better... Even with just those four programs you mentioned (and don't forget the Finder is running too), you're probably hitting Virtual Memory quite a bit. (What IRC client are you using?)

Curious: Are you using a VGA monitor with the VGA adaptor? There's been some complaints that the mini's video output is really dark on some VGA monitors, though all the causes of it aren't known yet...
http://appleturns.com/scene/?id=5184