BS: Just when you thought Google had it all

Subject: Just when you thought Google had it all
From: Bill D
Date: 31 Oct 01 - 07:11 PM

I found this page and my poor mind was boggled even more! I knew there was more...but....Well, something to do on those cold nights..*grin*...it may take me all Winter to look at everything!

Subject: RE: BS: Just when you thought Google had it all
From: Mark Cohen
Date: 31 Oct 01 - 07:18 PM

Gee, thanks, Bill, I was wondering what to do with my spare 7 minutes a day... :-)

Aloha,
Mark

Subject: RE: BS: Just when you thought Google had it all
From: wysiwyg
Date: 31 Oct 01 - 11:14 PM

Masato.... guess what's in there...

~S~

Subject: RE: BS: Just when you thought Google had it all
From: GUEST,.gargoyle
Date: 01 Nov 01 - 12:09 AM

It is always refreshing to see "newbies" discover the "ancient of days" -

Welcome to a "new-world" that has been there for YOU govt. sponsored for the past decade.

SHEESH.....Keep It "secret" the insiders don't want the rest to know!

Subject: RE: BS: Just when you thought Google had it all
From: masato sakurai
Date: 01 Nov 01 - 04:29 AM

Thanks, Bill D. What I knew about invisible webs is the links listed in The Major Search Engines (Scroll down and down...).

~Masato

Subject: RE: BS: Just when you thought Google had it all
From: McGrath of Harlow
Date: 01 Nov 01 - 06:06 AM

Think there are any songs in there?

Subject: RE: BS: Just when you thought Google had it all
From: Jim Dixon
Date: 01 Nov 01 - 11:15 AM

Wow! Bill D, thanks for the info, and thanks to Masato, too. I thought I was getting to be an expert, at finding things, but I see I've got a lot to learn.

The article clarified some things for me, for example, how a spider works. Now I understand why not all Mudcat threads are accessible to Google, and why some of them are. It turns out, only the ones that a non-member can get to by following links from the home page can be indexed. The ones that you need a membership for, or the ones you need to use a search box or filter for, can't be found through Google, or any other outside search engine, apparently. And the same is true for any other web site constructed the way Mudcat is.

The lesson to be learned is this: any important information that we want to be accessible to the outside world (and we do want that, don't we?) ought to be linked-to in one of the PermaThreads.

The question that I still don't know the answer to is this: How deep does a spider go when searching a web site? Evidently it doesn't necessarily follow every link, or index every page that it finds.

I just did an experiment. I used Google to search for "ZACK, THE MORMON ENGINEER" and it found it on www.mudcat.org/titles-z.cfm. But when I searched for "way back in seventy three" -- a phrase that occurs within the song -- it didn't find it. Evidently Google's spider didn't follow the link from the song title to the song itself in DigiTrad. Ah, but I see the link actually goes to http://www.mudcat.org/!!-supersearch99.cfm?MaxHits=1&Command=search&NumLines=4&file=fall99&request=%5BZACK,+THE+MORMON+ENGINEER%5D

In other words, it actually executes a search! Very interesting …

Subject: RE: BS: Just when you thought Google had it all
From: catspaw49
Date: 01 Nov 01 - 11:26 AM

Freakin' Amazing........................

Thanks Bill

Spaw

Subject: RE: BS: Just when you thought Google had it all
From: katlaughing
Date: 01 Nov 01 - 11:58 AM

Thanks, Bill and Masato!

Subject: RE: BS: Just when you thought Google had it all
From: MMario
Date: 01 Nov 01 - 12:12 PM

Jim - that is because in the first case your search term was part of the link itself - and thus indexed - but in the second case it was part of the text on the linked page . it is really the site's and the links that are indexed - not the content.

Subject: RE: BS: Just when you thought Google had it all
From: Mrrzy
Date: 01 Nov 01 - 01:10 PM

Wow! Fascinating! yes, there is definitely a song here... the Search Engine blues, to the tune of a good train song, no?

Subject: RE: BS: Just when you thought Google had it all
From: wildlone
Date: 01 Nov 01 - 02:52 PM

There are a few songs in there, I put lyrics into the infomine search engine and it turned up 60 hits inc the digitrad but also some I had never seen before.
dave

Subject: RE: BS: Just when you thought Google had it all
From: Jim Dixon
Date: 01 Nov 01 - 03:53 PM

MMario: I've found dozens, maybe hundreds of songs at various places on the Internet by using Google and searching for a phrase from the song (in quotes) just as I did it in the experiment described above. But usually I don't restrict my search to Mudcat. (In this experiment, I restricted it by going to Google's "Advanced Search" page and putting "www.mudcat.org" in the "Domain" box.)

In fact, without the restriction, Google DOES find "way back in seventy three" but not at Mudcat. It finds it at http://sniff.numachi.com/~rickheit/dtrad/lookup.cgi?ti=ZACKMORM&tt=OSUSANNA, which is another site that hosts the Digital Tradition.

I don't really understand the meaning of all those parameters, but it shows that the Google spider somehow searches and indexes the lyrics at Yet Another Digital Tradition Page, but it doesn't index ours. And I don't see any obvious difference between the way their site is set up and the way Mudcat is set up. Can anyone explain the difference?

Subject: RE: BS: Just when you thought Google had it all
From: Fibula Mattock
Date: 02 Nov 01 - 04:51 AM

Jim, it may be because Mudcat is done in Cold Fusion - a database interface of sorts where pages are created "on the fly" and aren't indexed or indexable in the same way as normal HTML (that's just a guess though - I'd have to check to see how it affects search engines).
Google works by page rank - each page is given a "weight" depending on how many people link to it - the more a page is linked to (again weighted, depending on what sites link to it), the higher it'll appear in the search results. Interestingly, Google caches the whole HTML source for the page - most other search engines only index metatags and links and titles etc. That's why, if a page no longer exists, Google can still sometimes provide you with an old page from the source they have cached. Contrarily, however, depending on when they update their caches, you could get a search result for an old page and when you click on the link there's a new updated page without the info you want.