Comments on: Starting with Zend_Search_Lucene http://dev.juokaz.com/php/starting-with-zend_search_lucene Random ideas, scripts and facts Mon, 29 Mar 2010 18:47:16 +0000 http://wordpress.org/?v=2.9.2 hourly 1 By: Patrick http://dev.juokaz.com/php/starting-with-zend_search_lucene/comment-page-1#comment-1293 Patrick Mon, 20 Apr 2009 15:42:01 +0000 http://dev.juokaz.com/?p=344#comment-1293 What is the limit of records for the Zend implementation of Lucene? A time ago I inserted 10.000 records coming from a Mysql database (key/value combo's) with value being a TEXT data type in Mysql. When I did a $index->find("value:'test'"); I got too many records back, even resulting in scores of 4.08383+39E (or something like that) ??? If I keep my records below 1000 everything seems to work fine... (Debian Etch GNU/Linux, PHP v5.2, Zend Framework v1.7.5) Regards, Patrick What is the limit of records for the Zend implementation of Lucene?
A time ago I inserted 10.000 records coming from a Mysql database (key/value combo’s) with value being a TEXT data type in Mysql.
When I did a $index->find(“value:’test’”); I got too many records back, even resulting in scores of 4.08383+39E (or something like that) ???
If I keep my records below 1000 everything seems to work fine… (Debian Etch GNU/Linux, PHP v5.2, Zend Framework v1.7.5)

Regards,

Patrick

]]> By: willian http://dev.juokaz.com/php/starting-with-zend_search_lucene/comment-page-1#comment-676 willian Fri, 03 Apr 2009 19:56:28 +0000 http://dev.juokaz.com/?p=344#comment-676 hi, i'm trying to just delete a doc that i added. i have 20 documents added. lucene says that the doc was deleted but when i search i have a result like this: "contains 20 documents", but it must be 19. do you have any idea why its happening. thanks, willian hi,

i’m trying to just delete a doc that i added. i have 20 documents added. lucene says that the doc was deleted but when i search i have a result like this: “contains 20 documents”, but it must be 19. do you have any idea why its happening.

thanks,
willian

]]> By: Juozas http://dev.juokaz.com/php/starting-with-zend_search_lucene/comment-page-1#comment-315 Juozas Fri, 13 Mar 2009 18:52:41 +0000 http://dev.juokaz.com/?p=344#comment-315 <a href="http://framework.zend.com/code/rdiff/Zend_Framework?csid=14304&u&N" rel="nofollow">Patch</a> shows added limits for max terms per query and prefix length. Probably next week I will try to test it with new version, but from source it looks that "* and * and *" issue should have been fixed. Patch shows added limits for max terms per query and prefix length. Probably next week I will try to test it with new version, but from source it looks that “* and * and *” issue should have been fixed.

]]> By: robo47 http://dev.juokaz.com/php/starting-with-zend_search_lucene/comment-page-1#comment-314 robo47 Fri, 13 Mar 2009 18:46:54 +0000 http://dev.juokaz.com/?p=344#comment-314 Seems something gets fixed in ZF 1.7.7 Release http://framework.zend.com/issues/browse/ZF-3321 http://framework.zend.com/code/changelog/Zend_Framework/?cs=14304 Didn't look what exactly got changed, but ticket got marked as fixed. Seems something gets fixed in ZF 1.7.7 Release

http://framework.zend.com/issues/browse/ZF-3321
http://framework.zend.com/code/changelog/Zend_Framework/?cs=14304

Didn’t look what exactly got changed, but ticket got marked as fixed.

]]> By: Juozas Kaziukenas’ Blog: Starting with Zend_Search_Lucene : Dragonfly Networks http://dev.juokaz.com/php/starting-with-zend_search_lucene/comment-page-1#comment-305 Juozas Kaziukenas’ Blog: Starting with Zend_Search_Lucene : Dragonfly Networks Fri, 13 Mar 2009 04:52:29 +0000 http://dev.juokaz.com/?p=344#comment-305 [...] Kaziukenas has a recent post to his blog introducing one of the many useful components of the Zend Framework - [...] [...] Kaziukenas has a recent post to his blog introducing one of the many useful components of the Zend Framework – [...]

]]> By: Juozas http://dev.juokaz.com/php/starting-with-zend_search_lucene/comment-page-1#comment-301 Juozas Thu, 12 Mar 2009 19:23:44 +0000 http://dev.juokaz.com/?p=344#comment-301 4 billion results clearly will kill Zend_Lucene :)) But all normal solutions (Lucene, Sphinx) should work fine, I guess. 4 billion results clearly will kill Zend_Lucene :)) But all normal solutions (Lucene, Sphinx) should work fine, I guess.

]]> By: selected http://dev.juokaz.com/php/starting-with-zend_search_lucene/comment-page-1#comment-300 selected Thu, 12 Mar 2009 19:21:16 +0000 http://dev.juokaz.com/?p=344#comment-300 it is fine as long as u use it for something small (like 6000 products or a blog like this). In my case (4 billion a4 documents) i would have to hung my self with zend implementation of lucene. it is fine as long as u use it for something small (like 6000 products or a blog like this). In my case (4 billion a4 documents) i would have to hung my self with zend implementation of lucene.

]]> By: robo47 http://dev.juokaz.com/php/starting-with-zend_search_lucene/comment-page-1#comment-299 robo47 Thu, 12 Mar 2009 18:57:12 +0000 http://dev.juokaz.com/?p=344#comment-299 Without the ability to let the user write complete querys lucene looses a lot. All the nice power the Lucene Implementation gives is away. I currently only filter out * and ~, so most things the query language provides is still usable. After finding out about the * AND * ... AND * -problem myself i made a lot of test-querys with the zend implementation and the only dangerous querys i was able to create contained * or ~ . It's only some basic blacklisting but better than nothing and additional I have implemented a search-query-log which includes execution-time + memory-usage of the search, so if anything bad happens, I can analyze the querys and probably find a way to filter them. Would be nice to find a solution for this, for example a way to give the query a time limit which is checked in the search-process and allows to throw an exception after the time is exceeded, because if the script dies because of memory_limit or max_execution_time, there are only 2 ways this is shown: white page or if display_errors is on, an error to the user ... both ways aren't something i want to choose. Without the ability to let the user write complete querys lucene looses a lot. All the nice power the Lucene Implementation gives is away.
I currently only filter out * and ~, so most things the query language provides is still usable.
After finding out about the * AND * … AND * -problem myself i made a lot of test-querys with the zend implementation and the only dangerous querys i was able to create contained * or ~ .
It’s only some basic blacklisting but better than nothing and additional I have implemented a search-query-log which includes execution-time + memory-usage of the search, so if anything bad happens, I can analyze the querys and probably find a way to filter them.
Would be nice to find a solution for this, for example a way to give the query a time limit which is checked in the search-process and allows to throw an exception after the time is exceeded, because if the script dies because of memory_limit or max_execution_time, there are only 2 ways this is shown: white page or if display_errors is on, an error to the user … both ways aren’t something i want to choose.

]]> By: Juozas http://dev.juokaz.com/php/starting-with-zend_search_lucene/comment-page-1#comment-297 Juozas Thu, 12 Mar 2009 17:34:58 +0000 http://dev.juokaz.com/?p=344#comment-297 Oh, I see. I read many (some) complaints about memory/speed issues. Now I need to add "query injections" to list what can cause problems. What about letting users to submit only keywords, not actual queries and then create them? It should work, but you limit yourself a lot :( Oh, I see.
I read many (some) complaints about memory/speed issues. Now I need to add “query injections” to list what can cause problems.

What about letting users to submit only keywords, not actual queries and then create them? It should work, but you limit yourself a lot :(

]]> By: robo47 http://dev.juokaz.com/php/starting-with-zend_search_lucene/comment-page-1#comment-296 robo47 Thu, 12 Mar 2009 17:28:01 +0000 http://dev.juokaz.com/?p=344#comment-296 But currently it is important to filter data which you pass to the query-methode, because some things can easily get the script to reach memory_limit or max_execution_time. especially querys like * AND * AND * .... AND * will use LOTS of memory and can run for minutes or longer. the proximity search also offers some danger to long running querys when giving 2 equal words and a big number. thread in a german forum: http://www.zfforum.de/showthread.php?p=29697 A related open Bug in the tracker: http://framework.zend.com/issues/browse/ZF-3321 But currently it is important to filter data which you pass to the query-methode, because some things can easily get the script to reach memory_limit or max_execution_time.

especially querys like * AND * AND * …. AND * will use LOTS of memory and can run for minutes or longer. the proximity search also offers some danger to long running querys when giving 2 equal words and a big number.

thread in a german forum: http://www.zfforum.de/showthread.php?p=29697
A related open Bug in the tracker: http://framework.zend.com/issues/browse/ZF-3321

]]>