Hi all,

There has been some interest on this list recently about search 
engines to put on your site : I am still searching for the answer that 
suits me so I thought I would just think aloud a bit


a) google's sitesearch looks OK but the indexing frequency is up to 
them

b) freefind is nice and simple to setup, you can control the indexing 
frequency and maybe (I forget) what is indexed

c) htdig  http://www.htdig.org/ .. you can see an example of it at work
at http://finzi.psych.upenn.edu/search.html
- the drawback is that it has to be installed on the server and I think 
*nix only 

d) there was a nice article on 'the complete webmaster' by Doug 
Steinwand called 'frame search' which was not site search per se (it 
used Yahoo to do a search of the web), but did talk about presenting 
the results of the search nicely (using frames)  - the presentation 
issue is one I am interested in

e) in a previous post to this list I talked about limiting search engines 
to a specific url (with the url: or similar syntax) and when that 
worked and did not .. maybe that technique (ie setting up the search
so that it only searched your site) plus Doug Steinwand's 
framesearch plus a judicious use of robots.txt could give an elegant 
answer

f) there is mnoGoSearch in PHP which looks very powerful (it can 
search PDF etc) .. this again uses an index. I cannot test this 
because my host's build of PHP does not seem to include this. 

g) my hosting service www.olm.net offers "SFE search" which is 
apparently freeware - this uses indexing to a flat file (I think you 
need to reindex the site from time to time), and seems to work OK, 
although there is not a lot of control over which parts of the site are 
to be indexed

h)
David's PHP scripts offers a 'sitesearch' at 
 http://php.warpedweb.net/index.php?action=download&file=search

I have not tried it but the philosophy seems similar to the above - ie 
index the site to a flat file or possibly a MySQL database


Conclusions?

Assuming I don't want to search databases (just 'text' pages) and 
that my site is not 'too large' I still have the following concerns

1. control over what is indexed/searched and what is not

2. Is there really a need for indexing?. If the site is not too large, 
and assuming PHP or somesuch is available, why not just run a 
grep? That has the attraction of not requiring a third party to 
maintain the index (google, freefind), not requiring you to maintain 
or update the index... Is this "just in time" search going to be too 
slow? I suspect not. But maybe it won't be sophisticated enough to 
give 'relevance scores' a la Google (which presumably uses 
stemming - at least to the extent of ignoring plurals )..   Meaning 
based search would be nice (see www.oingo.com) but again this 
would require the index being buiilt and kept by a third party

3. Appearance of results. Yes there is a degree of customization 
available with eg freefind .. but I'd like something better. The idea of 
re-presenting the results of an external search engine appeals.



Maybe it's best to stick to the power search engines (Google), either 
using sitesearch or setting up a search form where the url: keyword 
is invisibly appended to the search query. Control over what is 
indexed can be partially achieved by robots.txt and very relevant 
keyword tags on each page. Indexing frequency I guess would be 
up to them, except I do recall that you can suggest a revisit 
frequency with a meta something.


well, these are my musings, for what they are worth

• Web Designer Courses
• Off the Shelf Training Courses

HWG hwg-techniques mailing list archives, maintained by Webmasters @ IWA