-- error
Originally Posted By: suni123
Hi
Log file content fetcher :
[Thu, 09 Oct 2014 18:10:15 +0330] Downloading list of
urls...
[Thu, 09 Oct 2014 18:10:15 +0330] Fetch url list to download
time 0.0156
[Thu, 09 Oct 2014 18:10:15 +0330] So not checking scheduler.
[Thu, 09 Oct 2014 18:10:15 +0330] Current to crawl try
again count:0
[Thu, 09 Oct 2014 18:10:15 +0330] Current to crawl
count:100
[Thu, 09 Oct 2014 18:10:15 +0330] MAIN LOOP CASE 4 -- WEB
SCHEDULER
[Thu, 09 Oct 2014 18:10:15 +0330] End Name Server Check
[Thu, 09 Oct 2014 18:10:15 +0330] Done curl exec
[Thu, 09 Oct 2014 18:10:15 +0330] Set curl options for
single page request
[Thu, 09 Oct 2014 18:10:15 +0330] Init curl request of a
single page
[Thu, 09 Oct 2014 18:10:15 +0330] http://localhost/ to see
if active crawl time has changed.
[Thu, 09 Oct 2014 18:10:15 +0330] Checking name server:
[Thu, 09 Oct 2014 18:10:15 +0330] Switching archive...
[Thu, 09 Oct 2014 18:10:15 +0330] New name:
C:/xampp/htdocs/cache/0-Archive1412515961
[Thu, 09 Oct 2014 18:10:15 +0330] MAIN LOOP CASE 1 -- SWITCH
CRAWL OR NO CURRENT CRAWL
[Thu, 09 Oct 2014 18:10:15 +0330] End Name Server Check
[Thu, 09 Oct 2014 18:10:15 +0330] Dumping 95 from old
fetch to try to make a clean re-start.
[Thu, 09 Oct 2014 18:10:15 +0330] Fetch on crawl 1412515961
was not halted properly.
[Thu, 09 Oct 2014 18:10:15 +0330] Done curl exec
[Thu, 09 Oct 2014 18:10:15 +0330] Set curl options for
single page request
[Thu, 09 Oct 2014 18:10:15 +0330] Init curl request of a
single page
[Thu, 09 Oct 2014 18:10:15 +0330] http://localhost/ to see
if should start crawling
[Thu, 09 Oct 2014 18:10:15 +0330] Checking name server:
[Thu, 09 Oct 2014 18:10:15 +0330] In Fetch Loop
Initialize logger..
[Thu, 09 Oct 2014 18:10:15 +0330]
[Thu, 09 Oct 2014 17:16:17 +0330] Init Get Pages 0.0156
[Thu, 09 Oct 2014 17:16:17 +0330] Downloading list of
urls...
[Thu, 09 Oct 2014 17:16:17 +0330] Fetch url list to download
time 0
[Thu, 09 Oct 2014 17:16:17 +0330] Time to check Scheduler
0.0936
[Thu, 09 Oct 2014 17:16:17 +0330]
http://!!!.!!.!!!.!!!/?c=fetch&a=schedu ... 2862377&se
ssion=7f9b69bf6b42a3ee2660af3f414d2d35&robot_instance=0-loca
lhost-1411380798&machine_uri=/&crawl_time=1412515961&check_c
rawl_time=1412862376
[Thu, 09 Oct 2014 17:16:17 +0330] Making request:
[Thu, 09 Oct 2014 17:16:17 +0330] Done curl exec
[Thu, 09 Oct 2014 17:16:17 +0330] Set curl options for
single page request
[Thu, 09 Oct 2014 17:16:17 +0330] Init curl request of a
single page
[Thu, 09 Oct 2014 17:16:17 +0330] Checking
http://!!!.!!.!!!.!!!/ for a new schedule.
[Thu, 09 Oct 2014 17:16:17 +0330] MAIN LOOP CASE 4 -- WEB
SCHEDULER
[Thu, 09 Oct 2014 17:16:17 +0330] End Name Server Check
[Thu, 09 Oct 2014 17:16:17 +0330] Done curl exec
[Thu, 09 Oct 2014 17:16:16 +0330] Set curl options for
single page request
[Thu, 09 Oct 2014 17:16:16 +0330] Init curl request of a
single page
[Thu, 09 Oct 2014 17:16:16 +0330] http://localhost/ to see
if active crawl time has changed.
[Thu, 09 Oct 2014 17:16:16 +0330] Checking name server:
[Thu, 09 Oct 2014 17:16:13 +0330] Ensure minimum loop time
by sleeping...3
[Thu, 09 Oct 2014 17:16:13 +0330] Update Server Time
1.279202
[Thu, 09 Oct 2014 17:16:13 +0330] ... Current
Memory:6698744
[Thu, 09 Oct 2014 17:16:13 +0330] Updated Queue Server, sent
approximately 1247020 bytes:
[Thu, 09 Oct 2014 17:16:13 +0330] This fetcher peak memory
usage: 27581576
[Thu, 09 Oct 2014 17:16:13 +0330] Web Server peak memory
usage: 8498248
[Thu, 09 Oct 2014 17:16:13 +0330] Queue Server's crawl time
is: 1412515961
[Thu, 09 Oct 2014 17:16:13 +0330] Queue Server info response
code: 1
[Thu, 09 Oct 2014 17:16:13 +0330] ... Data upload complete
[Thu, 09 Oct 2014 17:16:13 +0330] Messages from Fetch
Controller:
[Thu, 09 Oct 2014 17:16:13 +0330] Done curl exec
[Thu, 09 Oct 2014 17:16:13 +0330] Set curl options for
single page request
[Thu, 09 Oct 2014 17:16:13 +0330] Init curl request of a
single page
[Thu, 09 Oct 2014 17:16:13 +0330] ...sending about 1247020
bytes.
[Thu, 09 Oct 2014 17:16:13 +0330] Sending Queue Server Part
1 of 1...
[Thu, 09 Oct 2014 17:16:13 +0330] ...
[Thu, 09 Oct 2014 17:16:13 +0330] ...1218908 bytes of index
data
[Thu, 09 Oct 2014 17:16:13 +0330] ...Finish Compressing seen
URLs.
[Thu, 09 Oct 2014 17:16:13 +0330] Saving index shard ..
wrote doc map. Done save
[Thu, 09 Oct 2014 17:16:13 +0330] Saving index shard ..
packed header
[Thu, 09 Oct 2014 17:16:13 +0330] Saving index shard .. make
prefixes
[Thu, 09 Oct 2014 17:16:13 +0330] Saving index shard .. done
merge postings to string
[Thu, 09 Oct 2014 17:16:13 +0330] ..Done Merge Index Posting
Final Copy
[Thu, 09 Oct 2014 17:16:13 +0330] ..Merge Index Posting
Final Copy
[Thu, 09 Oct 2014 17:16:13 +0330] Merge index shard postings
to string to save memory.
[Thu, 09 Oct 2014 17:16:13 +0330] Saving Mini Inverted
Index...
[Thu, 09 Oct 2014 17:16:13 +0330] Build mini inverted
index time 0.748801
For me, it happens approximately every 10 minutes.
Where the problem can be
'''Originally Posted By: suni123'''
Hi <br>Log file content fetcher :<br><br>[Thu, 09 Oct 2014 18:10:15 +0330] Downloading list of<br>urls...<br>[Thu, 09 Oct 2014 18:10:15 +0330] Fetch url list to download<br>time 0.0156<br>[Thu, 09 Oct 2014 18:10:15 +0330] So not checking scheduler.<br>[Thu, 09 Oct 2014 18:10:15 +0330] Current to crawl try<br>again count:0<br>[Thu, 09 Oct 2014 18:10:15 +0330] Current to crawl<br>count:100<br>[Thu, 09 Oct 2014 18:10:15 +0330] MAIN LOOP CASE 4 -- WEB<br>SCHEDULER<br>[Thu, 09 Oct 2014 18:10:15 +0330] End Name Server Check<br>[Thu, 09 Oct 2014 18:10:15 +0330] Done curl exec<br>[Thu, 09 Oct 2014 18:10:15 +0330] Set curl options for<br>single page request<br>[Thu, 09 Oct 2014 18:10:15 +0330] Init curl request of a<br>single page<br>[Thu, 09 Oct 2014 18:10:15 +0330] http://localhost/ to see<br>if active crawl time has changed.<br>[Thu, 09 Oct 2014 18:10:15 +0330] Checking name server:<br>[Thu, 09 Oct 2014 18:10:15 +0330] Switching archive...<br>[Thu, 09 Oct 2014 18:10:15 +0330] New name:<br>C:/xampp/htdocs/cache/0-Archive1412515961<br>[Thu, 09 Oct 2014 18:10:15 +0330] MAIN LOOP CASE 1 -- SWITCH<br>CRAWL OR NO CURRENT CRAWL<br>[Thu, 09 Oct 2014 18:10:15 +0330] End Name Server Check<br>[Thu, 09 Oct 2014 18:10:15 +0330] Dumping 95 from old<br>fetch to try to make a clean re-start.<br>[Thu, 09 Oct 2014 18:10:15 +0330] Fetch on crawl 1412515961<br>was not halted properly.<br>[Thu, 09 Oct 2014 18:10:15 +0330] Done curl exec<br>[Thu, 09 Oct 2014 18:10:15 +0330] Set curl options for<br>single page request<br>[Thu, 09 Oct 2014 18:10:15 +0330] Init curl request of a<br>single page<br>[Thu, 09 Oct 2014 18:10:15 +0330] http://localhost/ to see<br>if should start crawling<br>[Thu, 09 Oct 2014 18:10:15 +0330] Checking name server:<br>[Thu, 09 Oct 2014 18:10:15 +0330] In Fetch Loop<br>Initialize logger..<br><br>[Thu, 09 Oct 2014 18:10:15 +0330] <br>[Thu, 09 Oct 2014 17:16:17 +0330] Init Get Pages 0.0156<br>[Thu, 09 Oct 2014 17:16:17 +0330] Downloading list of<br>urls...<br>[Thu, 09 Oct 2014 17:16:17 +0330] Fetch url list to download<br>time 0<br>[Thu, 09 Oct 2014 17:16:17 +0330] Time to check Scheduler<br>0.0936<br>[Thu, 09 Oct 2014 17:16:17 +0330]<br>http://!!!.!!.!!!.!!!/?c=fetch&a=schedu ... 2862377&se<br>ssion=7f9b69bf6b42a3ee2660af3f414d2d35&robot_instance=0-loca<br>lhost-1411380798&machine_uri=/&crawl_time=1412515961&check_c<br>rawl_time=1412862376<br>[Thu, 09 Oct 2014 17:16:17 +0330] Making request: <br>[Thu, 09 Oct 2014 17:16:17 +0330] Done curl exec<br>[Thu, 09 Oct 2014 17:16:17 +0330] Set curl options for<br>single page request<br>[Thu, 09 Oct 2014 17:16:17 +0330] Init curl request of a<br>single page<br>[Thu, 09 Oct 2014 17:16:17 +0330] Checking <br>http://!!!.!!.!!!.!!!/ for a new schedule.<br>[Thu, 09 Oct 2014 17:16:17 +0330] MAIN LOOP CASE 4 -- WEB<br>SCHEDULER<br>[Thu, 09 Oct 2014 17:16:17 +0330] End Name Server Check<br>[Thu, 09 Oct 2014 17:16:17 +0330] Done curl exec<br>[Thu, 09 Oct 2014 17:16:16 +0330] Set curl options for<br>single page request<br>[Thu, 09 Oct 2014 17:16:16 +0330] Init curl request of a<br>single page<br>[Thu, 09 Oct 2014 17:16:16 +0330] http://localhost/ to see<br>if active crawl time has changed.<br>[Thu, 09 Oct 2014 17:16:16 +0330] Checking name server:<br>[Thu, 09 Oct 2014 17:16:13 +0330] Ensure minimum loop time<br>by sleeping...3<br>[Thu, 09 Oct 2014 17:16:13 +0330] Update Server Time<br>1.279202<br>[Thu, 09 Oct 2014 17:16:13 +0330] ... Current<br>Memory:6698744<br>[Thu, 09 Oct 2014 17:16:13 +0330] Updated Queue Server, sent<br>approximately 1247020 bytes:<br>[Thu, 09 Oct 2014 17:16:13 +0330] This fetcher peak memory<br>usage: 27581576<br>[Thu, 09 Oct 2014 17:16:13 +0330] Web Server peak memory<br>usage: 8498248<br>[Thu, 09 Oct 2014 17:16:13 +0330] Queue Server's crawl time<br>is: 1412515961<br>[Thu, 09 Oct 2014 17:16:13 +0330] Queue Server info response<br>code: 1<br><br>[Thu, 09 Oct 2014 17:16:13 +0330] ... Data upload complete<br>[Thu, 09 Oct 2014 17:16:13 +0330] Messages from Fetch<br>Controller:<br>[Thu, 09 Oct 2014 17:16:13 +0330] Done curl exec<br>[Thu, 09 Oct 2014 17:16:13 +0330] Set curl options for<br>single page request<br>[Thu, 09 Oct 2014 17:16:13 +0330] Init curl request of a<br>single page<br>[Thu, 09 Oct 2014 17:16:13 +0330] ...sending about 1247020<br>bytes.<br>[Thu, 09 Oct 2014 17:16:13 +0330] Sending Queue Server Part<br>1 of 1...<br>[Thu, 09 Oct 2014 17:16:13 +0330] ...<br>[Thu, 09 Oct 2014 17:16:13 +0330] ...1218908 bytes of index<br>data<br>[Thu, 09 Oct 2014 17:16:13 +0330] ...Finish Compressing seen<br>URLs.<br>[Thu, 09 Oct 2014 17:16:13 +0330] Saving index shard ..<br>wrote doc map. Done save<br>[Thu, 09 Oct 2014 17:16:13 +0330] Saving index shard ..<br>packed header<br>[Thu, 09 Oct 2014 17:16:13 +0330] Saving index shard .. make<br>prefixes<br>[Thu, 09 Oct 2014 17:16:13 +0330] Saving index shard .. done<br>merge postings to string<br>[Thu, 09 Oct 2014 17:16:13 +0330] ..Done Merge Index Posting<br>Final Copy<br>[Thu, 09 Oct 2014 17:16:13 +0330] ..Merge Index Posting<br>Final Copy<br>[Thu, 09 Oct 2014 17:16:13 +0330] Merge index shard postings<br>to string to save memory.<br>[Thu, 09 Oct 2014 17:16:13 +0330] Saving Mini Inverted<br>Index...<br>[Thu, 09 Oct 2014 17:16:13 +0330] Build mini inverted<br>index time 0.748801<br><br>For me, it happens approximately every 10 minutes. <br>Where the problem can be