The Mudcat Café TM
Thread #40666   Message #583900
Posted By: Jim Dixon
01-Nov-01 - 11:15 AM
Thread Name: BS: Just when you thought Google had it all
Subject: RE: BS: Just when you thought Google had it all
Wow! Bill D, thanks for the info, and thanks to Masato, too. I thought I was getting to be an expert, at finding things, but I see I've got a lot to learn.

The article clarified some things for me, for example, how a spider works. Now I understand why not all Mudcat threads are accessible to Google, and why some of them are. It turns out, only the ones that a non-member can get to by following links from the home page can be indexed. The ones that you need a membership for, or the ones you need to use a search box or filter for, can't be found through Google, or any other outside search engine, apparently. And the same is true for any other web site constructed the way Mudcat is.

The lesson to be learned is this: any important information that we want to be accessible to the outside world (and we do want that, don't we?) ought to be linked-to in one of the PermaThreads.

The question that I still don't know the answer to is this: How deep does a spider go when searching a web site? Evidently it doesn't necessarily follow every link, or index every page that it finds.

I just did an experiment. I used Google to search for "ZACK, THE MORMON ENGINEER" and it found it on www.mudcat.org/titles-z.cfm. But when I searched for "way back in seventy three" -- a phrase that occurs within the song -- it didn't find it. Evidently Google's spider didn't follow the link from the song title to the song itself in DigiTrad. Ah, but I see the link actually goes to http://www.mudcat.org/!!-supersearch99.cfm?MaxHits=1&Command=search&NumLines=4&file=fall99&request=%5BZACK,+THE+MORMON+ENGINEER%5D

In other words, it actually executes a search! Very interesting …