HariesDesign.com - The Joomla site has a checklist on security that is excellent. It covers installation, setup, and ongoing administrative matters. I would say it is required reading for anyone who has a Joomla website.

A lot of the suggestions on this page may seem a bit technical for some people, but they are worth persisting with. However, there are some suggestions even the least technically-minded person can use.

• use the latest, stable version of Joomla - keeping in mind that it sometimes takes a little while for the plugins you may need to catch up (for example, this could be an issue with Joomla 1.5).
• be careful in your choice of web host. Web hosts themselves can introduce security vulnerabilities, that can be easily avoided. There is link to a list of recommended hosts on the page above.
• delete all the left over installation files - particularly if you’re not using fantastico via cpanel to install joomla
  move your configuration file outside the Public_html folder

• change the default user name for your admin user - simple but effective!
• PHP5 is more secure, but the ability to use this will depend on whether your host is using it.
• Don’t use PHP safe mode
• Set the Register Globals Emulation off
• Delete all design templates you’re not using for your site.