Free Custom Search Engine with SPHINX
One of the things that you only find out, if you start to handle a few million files is, that MySQL is pretty cool, but if you do not 100% know, what you do, it can get pretty slow on bad SQL statements. But there is help:
http://enterprise.phpmagazine.net/2006/11/setup_an_open_source_search_en.html
We are implementing sphinx right now and it looks GREAT! That baby is fast and it takes custom datafeeds right out of the database to index and search.
You have to check-out and compile the sources right now, but I think next week we are ready to switch to sphinx for all searches on tradebit, which should speed us up like we need it for the new files coming soon.
