'''Originally Posted By: msan''' FreeBSD 9.1-RELEASE #0 r243825<br>Apache/2<br>GD Version bundled (2.0.34 compatible)<br>json version 1.2.1 (enabled)<br>SQLite3 module version 0.7-dev<br>SQLite Library 3.7.7.1<br>Php 5.3.21<br>Yioop 0.98<br><br>Hello to everybody,<br><br>while setting up machine for first crawl, I've noticed that fetcher initialize with yellow flag (clicking on), while server queue even clicking few times is still red-flagged. On server side both, fetcher and queue generates log files, but are not available from admin panel (panel says logs are empty).<br><br>I started both, fetcher and queue from SSH console and they were correctly started (no errors):<br><br>[root@vps /home/admin/domains/site.com/public_html/dev/bin]# php fetcher.php start<br>Starting 0-fetcher...<br>[root@vps /home/admin/domains/site.com/public_html/dev/bin]# php queue_server.php start<br>Starting Indexer-queue_server...<br>Starting Scheduler-queue_server...<br> <br><br>I've checked with &quot;ps&quot; from command line and this is result:<br><br>30919 1 S 0:00.09 php /home/admin/domains/site.com/public_html/dev/bin/fetcher.php child 0<br>30923 1 S 0:00.09 php /home/admin/domains/site.com/public_html/dev/bin/queue_server.php child none<br>30925 1 S 0:00.11 php /home/admin/domains/site.com/public_html/dev/bin/queue_server.php child none<br><br>from 0-fetcher.log:<br><br>[Thu, 19 Dec 2013 12:10:30 -0800] MAIN LOOP CASE 5 -- NO CURRENT CRAWL<br>[Thu, 19 Dec 2013 12:10:30 -0800] No data. Sleeping...<br>[Thu, 19 Dec 2013 12:10:45 -0800] Checking name server:<br>[Thu, 19 Dec 2013 12:10:45 -0800] http://someurl/ to see if should start crawling<br>[Thu, 19 Dec 2013 12:10:45 -0800] Init curl request of a single page<br>[Thu, 19 Dec 2013 12:10:45 -0800] Set curl options for single page request<br>[Thu, 19 Dec 2013 12:10:45 -0800] Done curl exec<br>[Thu, 19 Dec 2013 12:10:45 -0800] End Name Server Check<br><br>from queue_server.log:<br><br>Initialize logger..<br>[Thu, 19 Dec 2013 12:00:38 -0800] Scheduler logging started.<br>[Thu, 19 Dec 2013 12:00:38 -0800] Remove old messages..<br>[Thu, 19 Dec 2013 12:00:38 -0800] In queue loop!! Scheduler<br>[Thu, 19 Dec 2013 12:00:38 -0800] Scheduler peak memory usage so far: 7032880!!<br>[Thu, 19 Dec 2013 12:00:38 -0800] Scheduler is waiting for start message<br><br>[Thu, 19 Dec 2013 12:00:43 -0800] Indexer peak memory usage so far: 7037728!!<br>[Thu, 19 Dec 2013 12:00:43 -0800] Indexer is waiting for start message<br><br>[Thu, 19 Dec 2013 12:00:43 -0800] Scheduler peak memory usage so far: 7033136!!<br>[Thu, 19 Dec 2013 12:00:43 -0800] Scheduler is waiting for start message<br><br>in the Seekquarry Test all tests passed, except:<br><br>emptyBTreeNoSaveTestCase1/3 Tests Passed<br>FAILED: Case Test 0 assertEqual Empty B-Tree not saved saved to disk<br>FAILED: Case Test 2 assertEqual Empty B-Tree not saved saved to disk<br><br>Thanks for help

-- Machine don't initialize

It's kinda weird the btree tests don't pass. Maybe there is some permissions issue on the folder where it writes stuff?

-- Machine don't initialize

'''Originally Posted By: msan''' Hi Chris,<br><br>working directory is chmoded to 777 since installation.<br><br>A new error from console:<br><br>PHP Fatal error: Call to undefined function bzcompress() in /home/admin/domains/site.com/public_html/dev/lib/utility.php on line 693<br><br>Thanks

-- Machine don't initialize

Can you do an ls -l just to verify the permission of things since the install?<br>bzcompress should be in all recent versions of PHP. Do you have two<br>versions of PHP floating around on your system maybe?

-- Machine don't initialize

'''Originally Posted By: msan''' -rw-r--r-- 1 admin admin 27 Dec 19 19:10 .gitignore<br>-rw-r--r-- 1 admin admin 2892 Dec 19 19:10 INSTALL<br>-rw-r--r-- 1 admin admin 35385 Dec 19 19:10 LICENSE<br>-rw-r--r-- 1 admin admin 2253 Dec 19 19:10 README<br>drwxr-xr-x 2 admin admin 512 Dec 19 19:10 bin<br>-rw-r--r-- 1 admin admin 1271 Dec 19 19:10 blog.php<br>-rw-r--r-- 1 admin admin 1264 Dec 19 19:10 bot.php<br>drwxr-xr-x 2 admin admin 512 Dec 17 22:55 cgi-bin<br>drwxrwxrwx 2 admin admin 512 Dec 19 22:16 configs<br>drwxr-xr-x 2 admin admin 512 Dec 19 19:10 controllers<br>drwxr-xr-x 2 admin admin 512 Dec 20 14:03 css<br>drwxr-xr-x 2 admin admin 512 Dec 19 20:20 data<br>-rw-r--r-- 1 admin admin 2140 Dec 19 19:10 error.php<br>drwxr-xr-x 2 admin admin 512 Dec 19 19:10 examples<br>-rw-r--r-- 1 admin admin 1150 Dec 19 19:10 favicon.ico<br>-rw-r--r-- 1 admin admin 5600 Dec 19 19:10 index.php<br>drwxr-xr-x 8 admin admin 1536 Dec 19 19:10 lib<br>drwxr-xr-x 24 admin admin 512 Dec 19 19:10 locale<br>drwxr-xr-x 3 admin admin 512 Dec 19 19:11 models<br>-rw-r--r-- 1 admin admin 20 Dec 19 21:16 phpinfo.php<br>-rw-r--r-- 1 admin admin 1345 Dec 19 19:10 privacy.php<br>drwxr-xr-x 2 admin admin 512 Dec 19 19:11 resources<br>-rw-r--r-- 1 admin admin 189 Dec 19 19:10 robots.txt<br>drwxr-xr-x 2 admin admin 512 Dec 19 19:11 scripts<br>drwxr-xr-x 3 admin admin 1024 Dec 19 19:11 tests<br>drwxr-xr-x 5 admin admin 512 Dec 19 19:11 views<br>drwxrwxrwx 15 admin admin 512 Dec 20 14:30 wd<br>-rw-r--r-- 1 admin admin 3684 Dec 19 19:10 yioopbar.xml<br><br>&quot;wd&quot; is the working directory, here following its ls-l<br><br>[root@vps /home/admin/domains/site.com/public_html/dev/wd]# ls -l<br>total 64<br>drwxr-xr-x 2 root admin 2560 Dec 20 09:55 0-temp<br>drwxrwxrwx 2 apache admin 512 Dec 19 22:15 app<br>drwxrwxrwx 2 apache admin 512 Dec 19 22:15 archives<br>drwxrwxrwx 3 apache admin 512 Dec 20 16:28 cache<br>drwxrwxrwx 2 apache admin 512 Dec 19 22:15 classifiers<br>-rw-r--r-- 1 apache admin 3041 Dec 20 16:25 crawl.ini<br>drwxrwxrwx 2 apache admin 512 Dec 20 16:29 data<br>drwxrwxrwx 2 apache admin 512 Dec 19 22:15 feeds<br>drwxrwxrwx 24 apache admin 512 Dec 19 22:15 locale<br>drwxrwxrwx 2 apache admin 512 Dec 20 01:08 log<br>drwxrwxrwx 2 apache admin 512 Dec 19 22:15 prepare<br>-rwxrwxrwx 1 apache admin 2258 Dec 20 16:24 profile.php<br>-rw-r--r-- 1 apache admin 104 Dec 20 11:34 robot_table.txt<br>drwxrwxrwx 4 apache admin 1024 Dec 20 16:46 schedules<br>drwxrwxrwx 2 apache admin 512 Dec 19 22:15 search_filters<br>drwxrwxrwx 2 apache admin 512 Dec 20 14:15 temp<br><br>Can be the b-tree test failure connected to &quot;Call to undefined function bzcompress()&quot;? There is only one version of PHP runnung - 5.3.21<br><br>After that, machine initialized, I've made a first crawl. At the second crawl, indexing stopped at certain point and then through console I got that error message of call to undefined function.<br><br>Thanks

-- Machine don't initialize

bzcompress should only be called during an archive crawl of a bz2 compressed archive so<br>I am a little surprised it is even called. Maybe could you tell me a little about the<br>options you used to set up your crawl? I don't think bzcompress is connected to the btree issue which<br>seems to be a permissions issue. I notice for example that the 0-temp folder in your work<br>directory is only writable by root.

-- Machine don't initialize

'''Originally Posted By: msan''' Hi Cris,<br><br>I'll try to change the 0-Temp permission by creating a new folder a then renaming it back to 0-Temp and deleting the one writable only by root. Lets see if it works.<br><br>About crawl config, I'll paste here below the crawl.ini file which I suppose is the configuration file for the crawler, otherwise please tell me what exactly should I give you.<br><br>Code: ; crawl.ini<br>;<br>; crawl configuration file<br>;<br>[general]<br>crawl_order = 'ad';<br>crawl_type = 'ax';<br>crawl_index = '2';<br>arc_dir = &quot;MIX&quot;;<br>arc_type = &quot;MixArchiveBundle&quot;;<br>page_recrawl_frequency = '-1';<br>page_range_request = '50000';<br>max_description_len = '2000';<br>cache_pages = false;<br>restrict_sites_by_url = true;<br><br>[indexed_file_types]<br>extensions[] = 'unknown';<br>extensions[] = 'csv';<br>extensions[] = 'tab';<br>extensions[] = 'tsv';<br>extensions[] = 'epub';<br>extensions[] = 'asp';<br>extensions[] = 'aspx';<br>extensions[] = 'cgi';<br>extensions[] = 'cfm';<br>extensions[] = 'cfml';<br>extensions[] = 'do';<br>extensions[] = 'htm';<br>extensions[] = 'html';<br>extensions[] = 'jsp';<br>extensions[] = 'php';<br>extensions[] = 'pl';<br>extensions[] = 'py';<br>extensions[] = 'shtml';<br>extensions[] = 'jpeg';<br>extensions[] = 'rss';<br>extensions[] = 'svg';<br>extensions[] = 'xml';<br><br>[active_classifiers]<br><br>[active_rankers]<br><br>[allowed_sites]<br>url[] = 'http://www.airnav.com/';<br>url[] = 'http://www.airsouth.com.au/';<br>url[] = 'http://airsouthinc.com/www.airsouthinc.com/Home.html';<br>url[] = 'http://www.globalair.com/';<br>url[] = 'http://www.jetlease.com/html/';<br>url[] = 'http://www.laneaviation.com/';<br>url[] = 'http://www.rcaviationsales.com/';<br>url[] = 'http://www.robertsairsouth.net/';<br>url[] = 'http://www.southeastaircraftsales.com/';<br>url[] = 'http://www.spitfireaviation.com/';<br>url[] = 'http://www.worldjet.com/';<br>url[] = 'http://www.howardaircraft.com/';<br><br>[disallowed_sites]<br>url[] = 'domain:arxiv.org';<br>url[] = 'domain:ask.com';<br>url[] = 'domain:yelp.com';<br>url[] = 'domain:clixsense.com';<br>url[] = 'domain:google.com';<br>url[] = 'domain:yahoo.com';<br>url[] = 'domain:bing.com';<br><br>[seed_sites]<br>url[] = 'http://www.airnav.com/';<br>url[] = 'http://www.airsouth.com.au/';<br>url[] = 'http://airsouthinc.com/www.airsouthinc.com/Home.html';<br>url[] = 'http://www.globalair.com/';<br>url[] = 'http://www.jetlease.com/html/';<br>url[] = 'http://www.laneaviation.com/';<br>url[] = 'http://www.rcaviationsales.com/';<br>url[] = 'http://www.robertsairsouth.net/';<br>url[] = 'http://www.southeastaircraftsales.com/';<br>url[] = 'http://www.spitfireaviation.com/';<br>url[] = 'http://www.worldjet.com/';<br>url[] = 'http://www.howardaircraft.com/';<br><br>[page_rules]<br><br>[indexing_plugins]<br><br>

-- Machine don't initialize

'''Originally Posted By: msan''' According to B-Tree test, I've reset the 0-temp folder owner, but result is still the same. <br><br>1/3 Tests Passed<br>FAILED: Case Test 0 assertEqual Empty B-Tree not saved saved to disk<br>FAILED: Case Test 2 assertEqual Empty B-Tree not saved saved to disk<br><br>Code: [root@vps /home/admin/domains/somesite.com/public_html/dev/wd]# ls -l<br>total 64<br>drwxrwxrwx 2 admin admin 2560 Dec 30 22:35 0-temp<br>drwxrwxrwx 2 admin admin 512 Dec 22 16:52 app<br>drwxrwxrwx 2 admin admin 512 Dec 22 16:52 archives<br>drwxrwxrwx 5 admin admin 512 Dec 22 20:19 cache<br>drwxrwxrwx 2 admin admin 512 Dec 22 16:52 classifiers<br>-rw-r--r-- 1 admin admin 3016 Dec 30 22:55 crawl.ini<br>drwxrwxrwx 2 admin admin 512 Dec 22 20:02 data<br>drwxrwxrwx 2 admin admin 512 Dec 22 16:52 feeds<br>drwxrwxrwx 24 admin admin 512 Dec 22 16:52 locale<br>drwxrwxrwx 2 admin admin 512 Dec 22 16:58 log<br>drwxrwxrwx 2 admin admin 512 Dec 22 16:52 prepare<br>-rwxrwxrwx 1 admin admin 2258 Dec 22 20:01 profile.php<br>-rw-r--r-- 1 admin admin 104 Dec 22 18:23 robot_table.txt<br>drwxrwxrwx 6 admin admin 512 Dec 22 20:21 schedules<br>drwxrwxrwx 2 admin admin 512 Dec 22 16:52 search_filters<br>drwxrwxrwx 2 admin admin 512 Dec 22 16:53 temp<br><br><br><br>Now I see there many subfolders in the working directory set to root user... I think my new year will begin chown-ing subfolders

-- Machine don't initialize

'''Originally Posted By: msan''' Hi Chris,<br><br>I've setup right owner permissions, but the btree test still do not pass.<br><br>I think the issue should be around here:<br><br>Code: function emptyBTreeNoSaveTestCase()<br> {<br> $all_files = glob(BTREE_TEST_DIR.'/*.txt');<br> $this-&gt;assertEqual(0, count($all_files), 'Empty B-Tree not saved <br> saved to disk');<br> $this-&gt;test_objects['FILE1']-&gt;insert(array(1, 1));<br> $all_files = glob(BTREE_TEST_DIR.'/*.txt');<br> $this-&gt;assertEqual(2, count($all_files), 'Non-empty B-Tree successfully <br> saved to disk');<br> $this-&gt;test_objects['FILE1']-&gt;remove(1);<br> $all_files = glob(BTREE_TEST_DIR.'/*.txt');<br> $this-&gt;assertEqual(0, count($all_files), 'Empty B-Tree not saved <br> saved to disk');<br> }<br><br><br>Should be test file be written here &quot;tests/test_files&quot; (folder permissions are set to 777) , there are some other test files like test.epub, pdf, pptx and xsls<br><br>Can be this caused I'm on a subdomain, so the test cannot find the right path?

-- Machine don't initialize

BTREE_TEST_DIR is WORK_DIRECTORY.&quot;/btree_test&quot;. So if you can't create a directory in WORK_DIRECTORY you might have a problem? <br><br>Maybe B-trees are only used for handling e-tag and expires http headers so they aren't crucial for Yioop to do crawling. You can define<br>USE_ETAG_EXPIRES to be false in your local_config.php file if you want to turn them off.