Project

General

Profile

Bug #8206

Full Text Search Xapian

Added by Giuliano Mandotti about 3 years ago. Updated about 3 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
Plugins
Target version:
Start date:
20/04/2016
Due date:
% Done:

50%

Estimated time:
Complexity:
Normal
SCRUM pts - complexity:
?

Description

configuration:
chamilo 1.10.4 install and configured without errors
os: CentOS 5.6
Xapian installed, configured, activated ok

warning and fatal error as seen in screenshot attached, related to :
require_once(xapian.php) failed to open stream: no such file....in ...../xapian/XapianIndexer.class.php line 8
i've already seen a bug similar to this already resolved in 1.10.2 version (https://support.chamilo.org/issues/8013), talking about a symlink to xapian.php file not more explained.
the problem appear after saving the course creation (add_course)
if i go to other pages i see the course created.


Files

History

#1

Updated by Yannick Warnier about 3 years ago

We have already sent a series of fixes to 1.10.6 (not released yet) about that. I suppose this should wait until then (unless you can get the latest code from Git, but that implies a lot of fiddling around with the command line to update the libraries).

#2

Updated by Giuliano Mandotti about 3 years ago

thanks Yannick!
when probably 1.10.6 do you think it will be released?
can you give me some words of advice to fiddling the code, as my client, you can imagine, really need xapian working immediately?

#3

Updated by Yannick Warnier about 3 years ago

Hi Giuliano,

1.10.6 should be out next week. No guarantee though (we do that on our spare time).
You can find how to put a Chamilo in place from the "live" code here: https://github.com/chamilo/chamilo-lms/blob/1.10.x/README.md
Please proceed with caution and check everything in a separate server on folder if you do that for the first time, as it's very likely it could damage your production environment and then generate a lot of stress. Train doing that once or twice and then upgrade your customer's portal.

#4

Updated by Yannick Warnier about 3 years ago

  • Target version set to 1.10.6
#5

Updated by Yannick Warnier about 3 years ago

  • Status changed from New to Needs testing
#6

Updated by Giuliano Mandotti about 3 years ago

ok, it is in testing phase. thanks

#7

Updated by Yannick Warnier about 3 years ago

  • Category set to Plugins
  • % Done changed from 0 to 80
#8

Updated by Yannick Warnier about 3 years ago

  • Status changed from Needs testing to Assigned
  • Assignee set to Julio Montoya
  • % Done changed from 80 to 50

Julio, there are issues with the Xapian indexing feature. Once enabled, when you add a document (for example a PDF) to a course and launch a search (you can find a search box at the bottom of the left menu in the user_portal.php page), you get an error:

( ! ) Fatal error: No matching function for overloaded 'new_Query' in /usr/share/php/xapian.php on line 2607
Call Stack
#    Time    Memory    Function    Location
1    0.0000    237936    {main}( )    .../index.php:0
2    0.1048    3806960    require( '/var/www/chamilo110x/main/newscorm/lp_list_search.php' )    .../index.php:13
3    0.2363    7061896    chamilo_query_query( )    .../lp_list_search.php:124
4    0.2363    7062416    xapian_query( )    .../ChamiloQuery.php:22
5    0.2364    7063528    XapianQuery->__construct( )    .../XapianQuery.php:44
6    0.2364    7063768    new_query ( )    .../xapian.php:2607

I started a discussion with Olly Betts from Xapian here: http://lists.tartarus.org/pipermail/xapian-discuss/2016-April/009367.html and, as he pointed out, we seem to be using the Xapian library in a weird way. Could you have a quick look at it and tell me how complex you think it would be to fix?
I know it's not your code, but I also know you wrote a few fixes in the past. Hopefully, you can remember more or less how it works (you have to install the php5-xapian extension, then install a few commands to enable conversion of texts, then set a specific field "Title", and then go upload a document in the documents tool of any course with the "Simple" uploader.

This should give you all the basic pieces to be able to see the error.

Apparently, the issue is in the way we call the Xapian functions, considering we apparently "JOIN" pieces of a Xapian query in the wrong way.
If you think this will take too long (i.e. more than 4h), let's postpone this to 1.11.x. There's no real hurry to get it fixed, but it's a broken feature in 1.10 so far (although I initially thought we had fixed everything, it seems we only fixed the installer).

#9

Updated by Yannick Warnier about 3 years ago

  • Status changed from Assigned to New
  • Assignee deleted (Julio Montoya)
  • Target version changed from 1.10.6 to 1.11.0

The indexing code will be considered not-functional in 1.10 then, until we get time to fix everything in there.
Sorry Giuliano.

#10

Updated by Giuliano Mandotti about 3 years ago

thanks Yannick, i really understand. it looked just one problem but as you discovered others, this require more time on it, and for the project is not functional.
for my client it is.
we had also many problem to enable xapian through plesk, and other little problem but i'm waiting open ticket as 1.10.6 is coming so it is useless open issue for 10.4 when maybe 10.6 will resolve them
i maybe know your answer,and maybe you have already done much discussion and reasoning before the choice for xapian, but do you think it is possible to use without great effort another search engine with chamilo like apache solr?

#11

Updated by Yannick Warnier about 3 years ago

The choice of Xapian is mostly justified by the avoidance of a dependency on Java, which is now Oracle technology, is crippled by security flaws, is very difficult to maintain on a server and is usually quite slow. For all of these reasons, I decided (years ago) to go with Xapian. This worked for us for years, but we have left it a bit unwatched for some time, hence the current state.

Implementing another Search engine would not be super-difficult, but it would bring new dependencies that I don't think would be any easier to install with Plesk :-)

#12

Updated by Giuliano Mandotti about 3 years ago

yes, that's what i thought about your decision and agree.
now xapian too is creating some problem in installing it where plesk is the software manager.
we have to take care of writing docs to install chamilo where plesk is active, because it adds other steps not enough documented and causing instability to the server and to chamilo platform too.we can discuss about that via mail.
in the chamilo install i'm doing now we've encountered many issues due to plesk libs,path,permissions and chamilo to is returning many errors other than i've submitted in this ticket.

Here the issues:
1) config: server: Centos 7.2, plesk 12.5, chamilo 1.10.4,but installed as upgrade from 10.2 because during install process failed the db connection test where with 10.2 upgraded no problems (strange, surely some permission,but the requirements were all satisfied, and just for test we tried to install 4 example prestashop and everything gone well immediately)
2) inside the platform we cannot see, in admin/system status the Mysql info, it is empty where the other section display everything right. surely some permission
3)xapian: after many difficult and the help of a plesk specialist we've installed xapian 1.2.23 for php 5.6.21

Fatal error: require_once(): Failed opening required 'xapian.php' (include_path='.:/opt/plesk/php/5.6/share/pear') in /var/www/vhosts/chamilodomain/main/inc/lib/search/xapian/XapianQuery.php on line 10
and yesterday another issue: unable to connect to search DB

for this fatal error we have tried to do the symlink to xapian.php as i saw in an old chamilo ticket but it doesn't work

now i'll continue facing these errors, but i have the sensation that more we use xapian more errors will appear.
i know your big effort so i just tell you this to share info for the community and i'll do my best to contribute in resolve these issues.

thanks Yannick

#13

Updated by Julio Montoya about 3 years ago

Talking about new search engines ... nowadays people are using ElasticSearch.
They are open source (apache 2 license)

https://www.elastic.co/products/elasticsearch

And it looks easy to install:
https://www.elastic.co/downloads/elasticsearch

It could be easy to implement in Chamilo 2.x because there are a bundle for that and its uses Doctrine.

I didn't try it myself but it looks way more easy than xapian (because of Doctrine entities).

#14

Updated by Giuliano Mandotti about 3 years ago

Thank you Julio. that's a great opportunity for chamilo and for all people using chamilo as a collaboration platform. they are growing quickly and they need more stable and simple (to install) search engine than the current xapian version.
i'm studying it more in deep to give my contribution.
for xapian: now have installed on centos 7.2, php 5.6.21, xapian 1.2.23 without plesk 12.5 (which is creating more issues), chamilo see xapian installed in platform but still not work giving error: failed to connect to searchdb.
maybe some permission (but the folder and its parent are writable)? or some xapian settings?
or both?
i've asked to xapian forum which says that it surely not a xapian error.
as you've looked at xapian more than other chamilo-people, do you have some ideas to search for possible errors?
is there a separate command for creating the database before running
Chamilo in some way or before starting xapian plugin?
thank you Julio for your great job and effort

#15

Updated by Giuliano Mandotti about 3 years ago

connection issue resolved. it was a not updated permission settings.
now it works fine except that search function give always result empty, even if sure something should be found.

#16

Updated by Yannick Warnier about 3 years ago

  • Target version changed from 1.11.0 to 2.0

Moving to 2.0. Our deadline for 1.11 is 15/06 for a beta and 2.0 is for September/October, so a reasonable wait for a feature that is used very little (so far). But I agree on the importance to provide a nice search tool for our users in general. I just don't agree to force a dependency on Java.

#17

Updated by Giuliano Mandotti about 3 years ago

now the platform snapshot is: centos 7.2, xapian 1.2.23, chamilo 1.10.6, no plesk installed, php 5.6.21
the error is in xapian.php:
Fatal error: No matching function for overloaded 'new_Query' in /usr/local/share/php5/xapian.php on line 2623

i've found a very old similar issue (2006-2007) resolved with a patch, that now is part of the right code.
it seems that the construct doesnt like the 1-parameter query case.

i understand the moving to 2.0, i also understand and agree with you not to force a dependency on java, that's why i'm starting studying elastic search, as Julio suggested to be part of chamilo, maybe in the next 2.0 version
and yes it is very important nowaday to provide a nice search tool for all user in general

#18

Updated by Yannick Warnier about 3 years ago

Giuliano Mandotti wrote:

Fatal error: No matching function for overloaded 'new_Query' in /usr/local/share/php5/xapian.php on line 2623

i've found a very old similar issue (2006-2007) resolved with a patch, that now is part of the right code.
it seems that the construct doesnt like the 1-parameter query case.

You've now got to the exact same point as we have. This is where we have to modify the code. We've tried it (with very little time to do so) and when resolving the 1st param issue, we don't get results anymore :-) So yes, you're just there. If you get a solution, we'll review it and integrate it. Otherwise we just need more time.

#19

Updated by Giuliano Mandotti about 3 years ago

ok Yannick, now i'm testing some solutions; one now is working but not completely as the search is functioning but only in the My courses Page, not in the course/search page and it only returns the document containing the result. no number page info, etc.
i'm working on it and let you know

i've discovered and resolved locally 2 little bugs in 1.10.6: one aesthetical in a breadcrumb and another aestethical but with logical impact in event messages management. i'm just subscribed to github/chamilo to send my correction as i have time
thanks

Also available in: Atom PDF