| Summary | Mnemo2 locks up when encountering large notes |
| Queue | Mnemo |
| Queue Version | 2.1.1 |
| Type | Bug |
| State | Resolved |
| Priority | 2. Medium |
| Owners | |
| Requester | jwinius (at) umrk (dot) to |
| Created | 11/08/2007 (185 days ago) |
| Due | |
| Updated | 11/17/2007 (176 days ago) |
| Assigned | 11/09/2007 (184 days ago) |
| Resolved | 11/17/2007 (176 days ago) |
| Attachments | |
| Milestone | |
| Patch |
> Which one, 2, or 2.2-alpha?
2.2-alpha. I could never load the entire list with 2.
> Hardware probably isn't the issue here, it's download time. Make sure
> you've enabled page compression, etc.
I've installed the deflate module in Apache2 and turned off the debug mode for both Horde3 and Apache2, but that didn't seem to make any difference. I'll keep an eye on this issue, though.
State ⇒ Resolved
Which one, 2, or 2.2-alpha? Hardware probably isn't the issue here, it's download time. Make sure you've enabled page compression, etc.Yes, swapping in Mnemo 2.2-alpha fixes the problem for me. Thanks!
It would be nice, though, if there was simply a way to disable the previews -- something which I'm sure would also help the performance when many notes are involved. I've got over 300, and while my old system takes only one second to show the entire list, Mnemo2 on Horde3 takes almost four seconds despite newer, faster hardware.
Mnemo 2.2-alpha cuts previews to not be the whole note body. If that is indeed your problem, the alpha (which is compatible with your current Horde version) will fix it. Please test that and let us know.After some more experimentation I've made an important discovery. Although the problem does have to do with the size of the notes, it now appears not to be a problem with the size of the individual notes. Rather, it seems to be caused by the total size of all of the notes.
Not wanting to wait for a solution, I wrote a perl script to process my horde2 database dump and print out the sizes of the individual mnemo notes. My intention was to identify the notes that were too large and chop them up into smaller ones so that I could get mnemo2 to work again. However, the printout showed that a number of other notes, that were being successfully read by mnemo2, were much larger than the 34.6K size limit I mentioned earlier -- they were up to 77K in size.
Another test showed that if I started with a known non-working situation -- leaving that larger 34.6K note intact on the end -- and then deleted a few smaller notes (instead of making the 34.6K note smaller), mnemo2 would suddenly start to work again.
I calculated the total number of bytes and found that if all of the notes amounted to 1,252,799 bytes or less, mnemo2 would work. However, if the total was 1,266,774 bytes or more, mnemo2 would not work. The actual limit is somewhere in between.
I'm guessing, but could it be that this bug has something to do with the mnemo2 preview function?
> Can you please be a little more specific than "lock up"? Do you get PHP errors (either in
> the generated page, or in your server logs? Do you get apache segfaulting? Do you get
> javascript errors? Is the server unresponsive, or the browser? Etc...
Only mnemo (including the icon bar above) becomes unresponsive. This is what the Horde3 log says when I start up mnemo (debug mode):
=== Begin Horde3 log ===
Nov 10 13:11:03 HORDE [debug] [] SQL Query by Prefs_sql::retrieve(): SELECT pref_scope, pref_name, pref_value FROM horde_prefs WHERE pref_uid = ? AND (pref_scope = ? OR pref_scope = 'horde') ORDER BY pref_scope [on line 137 of "/usr/share/horde3/lib/Horde/Prefs/sql.php"]
Nov 10 13:11:03 HORDE [debug] [mnemo] Hook _horde_hook_share_init in application horde not called. [on line 1462 of "/usr/share/horde3/lib/Horde.php"]
Nov 10 13:11:03 HORDE [debug] [mnemo] SQL Query by DataTree_sql::getByAttributes(): SELECT c.datatree_id, c.datatree_name FROM horde_datatree c LEFT JOIN horde_datatree_attributes a1 ON a1.datatree_id = c.datatree_id WHERE c.group_uid = ? AND (a1.attribute_name = ? AND a1.attribute_key = ?) GROUP BY c.datatree_id, c.datatree_name, c.datatree_order ORDER BY c.datatree_order, c.datatree_name, c.datatree_id [on line 1111 of "/usr/share/horde3/lib/Horde/DataTree/sql.php"]
Nov 10 13:11:03 HORDE [debug] [mnemo] SQL Query by DataTree_sql::getByAttributes(): SELECT c.datatree_id, c.datatree_name FROM horde_datatree c LEFT JOIN horde_datatree_attributes a1 ON a1.datatree_id = c.datatree_id WHERE c.group_uid = ? AND ((a1.attribute_name = ? AND a1.attribute_value = ?) OR (a1.attribute_name = ? AND a1.attribute_key = ? AND CASE WHEN CAST(a1.attribute_value AS VARCHAR) ~ '^-\?[0-9]+$' THEN (CAST(a1.attribute_value AS INTEGER) \& ?) <> 0 ELSE FALSE END) OR (a1.attribute_name = ? AND CASE WHEN CAST(a1.attribute_value AS VARCHAR) ~ '^-\?[0-9]+$' THEN (CAST(a1.attribute_value AS INTEGER) \& ?) <> 0 ELSE FALSE END) OR (a1.attribute_name = ? AND CASE WHEN CAST(a1.attribute_value AS VARCHAR) ~ '^-\?[0-9]+$' THEN (CAST(a1.attribute_value AS INTEGER) \& ?) <> 0 ELSE FALSE END)) GROUP BY c.datatree_id, c.datatree_name, c.datatree_order ORDER BY c.datatree_order, c.datatree_name, c.datatree_id [on line 1111 of "/usr/share/horde3/lib/Horde/DataTree/sql.php"]
Nov 10 13:11:03 HORDE [debug] [mnemo] SQL Query by DataTree_sql::_loadById(): SELECT datatree_id, datatree_parents FROM horde_datatree WHERE datatree_id IN (5) AND group_uid = 'horde.shares.mnemo' ORDER BY datatree_id [on line 342 of "/usr/share/horde3/lib/Horde/DataTree/sql.php"]
Nov 10 13:11:03 HORDE [debug] [mnemo] SQL Query by DataTree_sql::_loadById(): SELECT datatree_id, datatree_name, datatree_parents, datatree_order FROM horde_datatree WHERE datatree_id IN (?, ?) AND group_uid = ? ORDER BY datatree_id [on line 375 of "/usr/share/horde3/lib/Horde/DataTree/sql.php"]
Nov 10 13:11:03 HORDE [debug] [mnemo] SQL Query by DataTree_sql::getAttributes(): SELECT datatree_id, attribute_name AS name, attribute_key AS "key", attribute_value AS value FROM horde_datatree_attributes WHERE datatree_id IN (5) [on line 992 of "/usr/share/horde3/lib/Horde/DataTree/sql.php"]
Nov 10 13:11:03 HORDE [debug] [mnemo] Mnemo_Driver_sql::retrieve(): SELECT * FROM mnemo_memos WHERE memo_owner = ? [on line 344 of "/usr/share/horde3/mnemo/lib/Driver/sql.php"]
Nov 10 13:11:04 HORDE [debug] [mnemo] SQL Query by DataTree_sql::getByAttributes(): SELECT c.datatree_id, c.datatree_name FROM horde_datatree c LEFT JOIN horde_datatree_attributes a1 ON a1.datatree_id = c.datatree_id WHERE c.group_uid = ? AND (a1.attribute_name = ? AND a1.attribute_key = ?) GROUP BY c.datatree_id, c.datatree_name, c.datatree_order ORDER BY c.datatree_order, c.datatree_name, c.datatree_id [on line 1111 of "/usr/share/horde3/lib/Horde/DataTree/sql.php"]
Nov 10 13:11:04 HORDE [debug] [mnemo] SQL Query by DataTree_sql::getByAttributes(): SELECT c.datatree_id, c.datatree_name FROM horde_datatree c LEFT JOIN horde_datatree_attributes a1 ON a1.datatree_id = c.datatree_id WHERE c.group_uid = ? AND ((a1.attribute_name = ? AND a1.attribute_value = ?) OR (a1.attribute_name = ? AND a1.attribute_key = ? AND CASE WHEN CAST(a1.attribute_value AS VARCHAR) ~ '^-\?[0-9]+$' THEN (CAST(a1.attribute_value AS INTEGER) \& ?) <> 0 ELSE FALSE END) OR (a1.attribute_name = ? AND CASE WHEN CAST(a1.attribute_value AS VARCHAR) ~ '^-\?[0-9]+$' THEN (CAST(a1.attribute_value AS INTEGER) \& ?) <> 0 ELSE FALSE END) OR (a1.attribute_name = ? AND CASE WHEN CAST(a1.attribute_value AS VARCHAR) ~ '^-\?[0-9]+$' THEN (CAST(a1.attribute_value AS INTEGER) \& ?) <> 0 ELSE FALSE END)) GROUP BY c.datatree_id, c.datatree_name, c.datatree_order ORDER BY c.datatree_order, c.datatree_name, c.datatree_id [on line 1111 of "/usr/share/horde3/lib/Horde/DataTree/sql.php"]
Nov 10 13:11:04 HORDE [debug] [mnemo] SQL Query by DataTree_sql::getAttributes(): SELECT datatree_id, attribute_name AS name, attribute_key AS "key", attribute_value AS value FROM horde_datatree_attributes WHERE datatree_id IN (5) [on line 992 of "/usr/share/horde3/lib/Horde/DataTree/sql.php"]
Nov 10 13:11:04 HORDE [debug] [mnemo] SQL Query by DataTree_sql::_buildParentIds(): SELECT datatree_id, datatree_parents FROM horde_datatree WHERE datatree_name = ? AND group_uid = ? ORDER BY datatree_id [on line 261 of "/usr/share/horde3/lib/Horde/DataTree/sql.php"]
Nov 10 13:11:04 HORDE [debug] [mnemo] SQL Query by DataTree_sql::_load(): SELECT c.datatree_id, c.datatree_name, c.datatree_parents, c.datatree_order FROM horde_datatree c WHERE c.group_uid = 'horde.perms' [on line 130 of "/usr/share/horde3/lib/Horde/DataTree/sql.php"]
Nov 10 13:11:04 HORDE [debug] [mnemo] SQL Query by DataTree_sql::getByAttributes(): SELECT c.datatree_id, c.datatree_name FROM horde_datatree c LEFT JOIN horde_datatree_attributes a1 ON a1.datatree_id = c.datatree_id WHERE c.group_uid = ? AND (a1.attribute_name = ? AND a1.attribute_key = ?) GROUP BY c.datatree_id, c.datatree_name, c.datatree_order ORDER BY c.datatree_order, c.datatree_name, c.datatree_id [on line 1111 of "/usr/share/horde3/lib/Horde/DataTree/sql.php"]
Nov 10 13:11:04 HORDE [debug] [mnemo] SQL Query by DataTree_sql::getByAttributes(): SELECT c.datatree_id, c.datatree_name FROM horde_datatree c LEFT JOIN horde_datatree_attributes a1 ON a1.datatree_id = c.datatree_id WHERE c.group_uid = ? AND ((a1.attribute_name = ? AND a1.attribute_value = ?) OR (a1.attribute_name = ? AND a1.attribute_key = ? AND CASE WHEN CAST(a1.attribute_value AS VARCHAR) ~ '^-\?[0-9]+$' THEN (CAST(a1.attribute_value AS INTEGER) \& ?) <> 0 ELSE FALSE END) OR (a1.attribute_name = ? AND CASE WHEN CAST(a1.attribute_value AS VARCHAR) ~ '^-\?[0-9]+$' THEN (CAST(a1.attribute_value AS INTEGER) \& ?) <> 0 ELSE FALSE END) OR (a1.attribute_name = ? AND CASE WHEN CAST(a1.attribute_value AS VARCHAR) ~ '^-\?[0-9]+$' THEN (CAST(a1.attribute_value AS INTEGER) \& ?) <> 0 ELSE FALSE END)) GROUP BY c.datatree_id, c.datatree_name, c.datatree_order ORDER BY c.datatree_order, c.datatree_name, c.datatree_id [on line 1111 of "/usr/share/horde3/lib/Horde/DataTree/sql.php"]
Nov 10 13:11:04 HORDE [debug] [mnemo] SQL Query by DataTree_sql::getAttributes(): SELECT datatree_id, attribute_name AS name, attribute_key AS "key", attribute_value AS value FROM horde_datatree_attributes WHERE datatree_id IN (5) [on line 992 of "/usr/share/horde3/lib/Horde/DataTree/sql.php"]
=== End Horde3 log ===
The apache2 log say a lot less (debug mode):
=== Begin Apache2 log ===
atheris.umrk.to - - [10/Nov/2007:13:17:53 +0100] "GET /horde3/mnemo/?Horde=06118e37f6e6c0f6e81582eefafb19b7 HTTP/1.1" 200 - "https://webmail.umrk.to/horde3/services/portal/sidebar.php?Horde=06118e37f6e6c0f6e81582eefafb19b7" "Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.8 (like Gecko) (Debian)"
=== End Apache2 log ===
Otherwise, I have not found any particular errors associated with this problem. Although the Horde3 iconbar above becomes unresponsive, the sidebar continues to work, so it is not a problem to switch to another Horde3 application. The rest of the browser and all other open pages are also totally unaffected.
State ⇒ Feedback
Priority ⇒ 2. Medium
Can you please be a little more specific than "lock up"? Do you get PHP errors (either in the generated page, or in your server logs? Do you get apache segfaulting? Do you get javascript errors? Is the server unresponsive, or the browser? Etc...State ⇒ Unconfirmed
Summary ⇒ Mnemo2 locks up when encountering large notes
Queue ⇒ Mnemo
Priority ⇒ 3. High
Type ⇒ Bug
While migrating a horde2 database (including mnemo v1.1) to horde3, I notice that mnemo2 locks up if any of the notes contain more than about 33.6 KB of text. If there is even one note in several hundred that is slightly too large, the list of notes will not be generated and the icons across the top of the page will either stop working (Konqueror), or the entire icon bar will disappear (Firefox).
In such cases I have found that if I use phppgadmin to edit the offending text field and reduce it slightly in size, mnemo2 will suddenly start to work again. It will even allow me to save extra text in the same note again, after which it will once again cease to function as a result.
Curiously, if I use Konqueror the limit for mnemo2 is a text size of 34606 bytes, with 34607 causing the application to lock up. However, when using Firefox (Iceweasel) the maximum supported size was 34485 bytes, while for both Mozilla (Iceape) and Epiphany it was 34489 bytes.
My horde2 system is still running on a Debian sarge machine with postgresql v7.4.7 and apache-ssl v1.3.33. So far, I am not aware that mnemo v1.1 has any such text size limitations. My new horde3 system is on Debian etch with postgresql v7.4.17 and apache2 v2.2.3. The browsers are on a workstation running Debian sid.