Yioop - PHP Search Engine

Fetchers and Queue Server Turn Off By Themselves.

Originally Posted By: m179

Hello,

I seem to be having a very odd issue with Yioop right now and the issue persists after full reinstalls on both machines (machine 1: web server and machine 2: fetchers (16) and queue server).

And just to put it on the record, both machines have the same "name server" and "server key" in the advanced settings in Admin Panel.

So here's what happens, I turn on all my fetchers (hosted on machine 2) and the queue server along with the name server on machine 1 and they stay on for a bit.
Then as soon as I start a crawl (which doesn't even start) it immediately kills the fetchers/queue server on machine 2 or puts them into what I call a "ghost" state.
Here is a screenshot showing that i manually turned off fetchers 8-16 but yet 0-7 are in the ghost state (logs are not recent either, very old from when it did work) and the queue server turned off as soon as I pressed start crawl which never starts, just stays idle. http://puu.sh/6yTOo/6ba439a115.png

NOTE: Just noticed, that if I turn on the fetchers and just wait about 2min. they turn off (ghost phase) by themselves, no logs. The queue server goes off at about that same time, or until I try to begin a crawl. So maybe this is a timeout of sorts since it's not really starting the fetchers on the other node?

Please assist,
Thanks,
Michael

'''Originally Posted By: m179''' Hello, I seem to be having a very odd issue with Yioop right now and the issue persists after full reinstalls on both machines (machine 1: web server and machine 2: fetchers (16) and queue server). And just to put it on the record, both machines have the same "name server" and "server key" in the advanced settings in Admin Panel. So here's what happens, I turn on all my fetchers (hosted on machine 2) and the queue server along with the name server on machine 1 and they stay on for a bit. Then as soon as I start a crawl (which doesn't even start) it immediately kills the fetchers/queue server on machine 2 or puts them into what I call a "ghost" state. Here is a screenshot showing that i manually turned off fetchers 8-16 but yet 0-7 are in the ghost state (logs are not recent either, very old from when it did work) and the queue server turned off as soon as I pressed start crawl which never starts, just stays idle. http://puu.sh/6yTOo/6ba439a115.png NOTE: Just noticed, that if I turn on the fetchers and just wait about 2min. they turn off (ghost phase) by themselves, no logs. The queue server goes off at about that same time, or until I try to begin a crawl. So maybe this is a timeout of sorts since it's not really starting the fetchers on the other node? Please assist, Thanks, Michael

-- Fetchers and Queue Server Turn Off By Themselves

If the logs are blanks that means the machines never turned on and it is taking a while for Yioop to pick up on it.
So either the message from the name server to the machine in question wasn't sent properly, or the
function call to start a process on the machine in question didn't work. The code for the former is in
models/parallel_model.php (execMachines) and for the latter lib/crawl_daemon.php (start). To test what's
going on you can add a print statement to exec machines to print out the query it makes to the machine
that it is trying to start a process on. Then you can take that query and append it to that machines url
and try it in a browser to see what it responds with. This would catch any notices or errors from that
machine and also might show you if you have entered a url for a machine wrong.

To test the start/stop processes from the command line, you can run
php fetcher.php start 0
This should start 0-fetcher on whatever machine. Do a
php uax | grep php
to see if the process got started and it is not stopping itself. If there is an issue, then in lib/crawl_daemon.php's
start function add a print statement to print $at_job just before the pclose(popen line. Try typing the
line you get directly and see it if it gives any errors.

Once you have debugged stuff make sure to remove the print statements you added.

Let me know how it goes best. Also, what version of Yioop are you using?

If the logs are blanks that means the machines never turned on and it is taking a while for Yioop to pick up on it. So either the message from the name server to the machine in question wasn't sent properly, or the function call to start a process on the machine in question didn't work. The code for the former is in models/parallel_model.php (execMachines) and for the latter lib/crawl_daemon.php (start). To test what's going on you can add a print statement to exec machines to print out the query it makes to the machine that it is trying to start a process on. Then you can take that query and append it to that machines url and try it in a browser to see what it responds with. This would catch any notices or errors from that machine and also might show you if you have entered a url for a machine wrong. To test the start/stop processes from the command line, you can run php fetcher.php start 0 This should start 0-fetcher on whatever machine. Do a php uax | grep php to see if the process got started and it is not stopping itself. If there is an issue, then in lib/crawl_daemon.php's start function add a print statement to print $at_job just before the pclose(popen line. Try typing the line you get directly and see it if it gives any errors. Once you have debugged stuff make sure to remove the print statements you added. Let me know how it goes best. Also, what version of Yioop are you using?

-- Fetchers and Queue Server Turn Off By Themselves

Originally Posted By: m179

Thanks for the reply I'll take a look into what you suggested.
It is the latest version of Yioop.
Is there a specific Port we should open for Yioop to send the nameserver requests because it looks like that is what's getting lost and not received?

Also just noticed that from the machine2 itself, I can't start the fetchers/Queue server, they don't even turn on; just stay off.

For the nameserver URL and server Key, does it matter which machine I set as the 'name server'?

Michael

'''Originally Posted By: m179''' Thanks for the reply I'll take a look into what you suggested. It is the latest version of Yioop. Is there a specific Port we should open for Yioop to send the nameserver requests because it looks like that is what's getting lost and not received? Also just noticed that from the machine2 itself, I can't start the fetchers/Queue server, they don't even turn on; just stay off. For the nameserver URL and server Key, does it matter which machine I set as the 'name server'? Michael

-- Fetchers and Queue Server Turn Off By Themselves

By default communication is on port 80. All machines in your set-up should use the same server key.
It shouldn't matter which machine you choose as the name server.

By default communication is on port 80. All machines in your set-up should use the same server key. It shouldn't matter which machine you choose as the name server.

-- Fetchers and Queue Server Turn Off By Themselves

Originally Posted By: m179

cpollett wrote:
By default communication is on port 80. All machines in your set-up should use the same server key.
It shouldn't matter which machine you choose as the name server.

So we changed our setup a bit. Now we have only one machine which hosts both the fetchers, the queue server, and the web server for requests.
And instead of starting the queue server and 16 fetchers from the Web UI, we did it from php-cli.
Is it normal that under "Manage machines" it doesn't show that the fetchers/queue server is on when in fact it really is but we turned it on manually?

'''Originally Posted By: m179''' cpollett wrote: By default communication is on port 80. All machines in your set-up should use the same server key. It shouldn't matter which machine you choose as the name server. So we changed our setup a bit. Now we have only one machine which hosts both the fetchers, the queue server, and the web server for requests. And instead of starting the queue server and 16 fetchers from the Web UI, we did it from php-cli. Is it normal that under "Manage machines" it doesn't show that the fetchers/queue server is on when in fact it really is but we turned it on manually?

-- Fetchers and Queue Server Turn Off By Themselves

The indicators in Manage Machines only work if you started the machines using manage machines.
If you start fromt eh cli that don't turn on.

Best,
Chris

The indicators in Manage Machines only work if you started the machines using manage machines. If you start fromt eh cli that don't turn on. Best, Chris