How to search a huge text file for data?

JdS · #1 27-Nov-2002, 06:47

I have been trying for days to get this to work but obviously PHP's filesystem is not one of my fortes.

The large text file I want to 'search' is the website's access log file which is quite huge.

So far I have tried:

PHP Code:


<?php

//  I hope to isolate all instances of visits by search engine spiders
//  for example: googlebot / zyborg / slurp / etc
$fd = fopen( 'access.log', 'r' );
while( !feof($fd) )
{
  $buffer = fgets( $fd, 4096 );
  if( strstr(strtolower($buffer), 'googlebot') )
  {
    echo $buffer."<br />\n";
  }
}

fclose ($fd);

?>

Up to this point, I am already stuck; with even 1 day's worth of logs, this piece of code will time-out.

If you run the Apache Web Server on your win32 PC, you can find the access log (and view the sample data) file usually at:

Code:

c:\program files\apache group\apache\logs\access.log

unless, you've changed this path in httpd.conf....

pcxgamer · #2 27-Nov-2002, 15:33

Is the file opened ok? As it is now, you aren't doing any error checking whatsoever. I'd change the open line to
$fd = fopen('/path/to/access_log', 'r') or die("Couldn't open the file.");

just a thought let me know if this help or not.

JdS · #3 28-Nov-2002, 09:41

nope, it's not the matter of error-checking... the file being searched is 47MB (give or take 3 bytes)!

so it's no wonder it times out...

and i think i got a work-around but it works a bit retarded

i'll tell you more when I send off an email to Scott of www.Vilitas.com warning him that I want to try out this script sometime in the near future.

pcxgamer · #4 28-Nov-2002, 09:45

Oh ok, Let me know.

JdS · #5 29-Nov-2002, 05:46

Here's GIDGoogle™ ver. 0.0.1

-

http://topsites.gidhelp.com/get_google.php

What it does is to grab any googlebot activity off your Apache logs... and it doesn't matter how large your log file is! Cool, I think.

Once I get the script into a class and some other features added, I will offer it as a download too...

pcxgamer · #6 29-Nov-2002, 06:01

Great job, Let me know when your going to offer it as a download I could use something like on my server at maxipoint I'm gettting ready to submit my site to them.

jrobbio · #7 26-May-2003, 15:02

Did you ever create a download for this?

Rob

JdS · #8 27-May-2003, 10:27

No, it was seriously flawed. So I went the route of GIDTrackbot™!

Also, figuring out Linux these days has certainly opened up many new possibilities to get the same results with much less processing overhead... one day I will figure out a new way to get this data out (but only off a linux server).

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Rate This Thread
Rate This Thread:

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Manipulating text Files	k209310	C++ Forum	0	17-Nov-2003 11:23
CD Buring Failed	skanth2000	Computer Hardware Forum	1	15-Nov-2003 04:52
Search Engine Positioning 101 and 201 "How To" Tips...	000	Search Engine Optimization Forum	0	29-May-2003 11:34
[class] 404 search function code	jrobbio	MySQL / PHP Forum	6	22-Apr-2003 10:32
How Do i get php to find out the file type of a file for me?	viperman95833	MySQL / PHP Forum	2	08-Mar-2003 10:48