I realise that the usual Google 'deepcrawl' as we knew it is supposed to be 'gone' but I noticed something very familiar today.

A lot of old, existing, already indexed web pages being fetched by Googlebot since early this morning (07 September 2003 server time).

Perhaps the 'deepcrawl' does still happen afterall, it's just not so obvious anymore...

Interesting detective work JDS and yes I'm sure the deepcrawl as we know it still happens, but its so submerged in everything else that it takes a keen eye to spot it.

Could you possibly post some of the I.P.'s?

Rob

I didn't really do any 'detective' work - just happened to find the who's online users numbers abnormally high (for a Sunday) and decided to see what all the fuss was about...

If you happen to read this post and Googlebot is still at it, you can just hover the mouse over the IP / HOST column to see the IP numbers (see link in previous paragraph). It's not easy to see a distinct pattern here (with the IPs) but then again I didn't investigate further than just to observe this and report it here.

You've got multiple bogeys (bots) all over the forum right now...

I've had a huge crawl by Googlebot today on a magnitude I've never seen before. Either that or its completely stuck in my forum.

Rob

It may be because it's fetching every available link / query string off your forum pages (meaning duplicate content) which shows heavy activity but results ultimately in very few INDEXED pages.

This is why I have restricted googlebot from fetching showthread.php, forumdisplay.php and other useless pages here for this forum. Less traffic from Googlebot but the ones indexed are found no other way...

Have you guys seen a recent pattern where GBot seems to want the robots.txt file after every single file fetch? Something like;

robots.txt
file1.htm
robots.txt
file2.htm
robots.txt
file3.htm

etc

No Div, I have not noticed this since I still use GIDTrackbot to analyse Googlebot traffic and that script doesn't track how many times Googlebot fetches the robots.txt file.

Yep! Google's looking like it's deep-crawling again; exactly one month from the last sighting here on the site. Right on schedule as well... it's happening since late 05/10/2003 according to my GIDTrackbot report.

That's funny... Googlebot won't go near my site!! Maybe you pass the word along? hee hee

Josh