Sun, 05 Apr 2026 19:21:54 +0000 https://bugs.horde.org/ticket/6972 mailbox listing speed-up (IMAP_Tree rewrite) After our recent horde/imp upgrade at umich, we've seen login times rise from ~5-10 seconds to ~10-30 seconds. The bulk of this time is spent building the mailbox list in IMP_Tree (imp/lib/IMAP/Tree.php). We've made some changes which dramatically reduce the IMP_Tree startup time and would love some feedback and basic sanity checking before we move forward with them. On an unloaded test server in our environment, imap_getmailboxes and imap_list commands can take up to 1.7 seconds each against our Cyrus system. First, IMP_Tree calls imap_getmailboxes for each namespace with a % appended to get one level deep of subfolders. Second, IMP_Tree::_initSubscribed executes imap_list for each namespace with an * appended to get all sublevels of folders. Last, it recursively crawls each folder found during the first set of imap_getmailboxes and calls imap_getmailboxes on each folder with % again until all branches have been plumbed. Our improvements are premised on the fact that one level searches (%) and searches which return full trees of results (*) actually take the same time to return from Cyrus. So we are bypassing all other searches and starting off with one single full (*) search from the root of the user's mailbox. One quirk we had to side step then was that % searches will return objects for folders which have children that can't be directly accessed themselves. The * search will not. So we just iterate through the php data structure and detect parentless children and fake up an object for them. Then, if tree_view is turned on, we separate non-personal mailboxes into IMPTREE_OTHER_KEY and IMPTREE_SHARED_KEY. Since the list of folders at this point has all the children, we bypass the recursive calls to expand. Tue, 24 Jun 2008 19:49:09 +0000 https://bugs.horde.org/ticket/6972#t46775 Works okay for me so far. Wed, 25 Jun 2008 04:18:44 +0000 https://bugs.horde.org/ticket/6972#t46788 I've been working on this for about a week now. The patch has the right idea, but there is a boatload of code that needs to be taken out (10 KB so far) and there are several bugs not covered by this patch. Additionally, I am looking at not storing polled, subscribed, and unsubscribed information in separate lists to reduce session size even more (the data will be rebuilt from the tree). Wed, 25 Jun 2008 04:38:44 +0000 https://bugs.horde.org/ticket/6972#t46789 Changes have been made in CVS for this ticket: http://cvs.horde.org/diff.php/imp/lib/IMAP/Tree.php?r1=1.187&r2=1.188&ty=u Wed, 25 Jun 2008 18:20:06 +0000 https://bugs.horde.org/ticket/6972#t46812 First attempt has been committed to HEAD. Now testing is needed. Those using a non-HEAD version can download the file from: http://cvs.horde.org/co.php/imp/lib/IMAP/Tree.php?r=1.188&p=1 and directly replace the existing imp/lib/IMAP/Tree.php Wed, 25 Jun 2008 18:21:55 +0000 https://bugs.horde.org/ticket/6972#t46813 Changes have been made in CVS for this ticket: http://cvs.horde.org/diff.php/imp/config/servers.php.dist?r1=1.82&r2=1.83&ty=u http://cvs.horde.org/diff.php/imp/lib/Session.php?r1=1.168&r2=1.169&ty=u Wed, 25 Jun 2008 18:52:36 +0000 https://bugs.horde.org/ticket/6972#t46816 Changes have been made in CVS for this ticket: http://cvs.horde.org/diff.php/imp/lib/IMAP/Tree.php?r1=1.188&r2=1.189&ty=u Wed, 25 Jun 2008 19:29:47 +0000 https://bugs.horde.org/ticket/6972#t46817 It started using folder subscriptions even though I have them turned off. but once I cleared my dovecot .subscriptions file (had some old folders in it) the current code is working well for me. I'm a pretty small test case though. Thu, 26 Jun 2008 04:33:56 +0000 https://bugs.horde.org/ticket/6972#t46824 > It started using folder subscriptions even though I have them turned > off. but once I cleared my dovecot .subscriptions file (had some old > folders in it) the current code is working well for me. I'm a pretty > small test case though. I use dovecot and I saw this too - imap_lsub on dovecot appears to return the list of mailboxes in the subscribed folder regardless of whether they are present on the server or not. Seems like this is broken IMAP behavior, but maybe I'll ask Timo (dovecot dev) if this is correct or not. To workaround, we should probably get the list of folders from imap_list() and cross-reference with the list of folders from imap_lsub(). Thu, 26 Jun 2008 18:21:22 +0000 https://bugs.horde.org/ticket/6972#t46896 I'm not actually too worried about that, though that would be nice - what's more curious is that I don't use subscriptions (they're turned off in my prefs) and IMP started to use them. Thu, 26 Jun 2008 18:27:14 +0000 https://bugs.horde.org/ticket/6972#t46898 > I'm not actually too worried about that, though that would be nice - > what's more curious is that I don't use subscriptions (they're turned > off in my prefs) and IMP started to use them. I don't see the subscriptions issue with the latest version - maybe it was fixed in the last few days. As for the other issue, dovecot is exhibiting correct behavior. From RFC 3501 [6.3.9]: The returned untagged LSUB response MAY contain different mailbox flags from a LIST untagged response. If this should happen, the flags in the untagged LIST are considered more authoritative. ... The server MUST NOT unilaterally remove an existing mailbox name from the subscription list even if a mailbox by that name no longer exists. So if viewing only subscribed folders, we do need to cross-reference with imap_list() results to get an accurate representation of the current subscribed folders. Thu, 26 Jun 2008 19:34:36 +0000 https://bugs.horde.org/ticket/6972#t46900 I have prefs set to display namespaces separately and use subscriptions. Now, if there is a shared folder from another user, the parent node saying "Other Users' Folders" is displayed (empty), even if I didn't subscribe to any of those other users' folders. Additionally, the user itself is clickable, though I'm not sure if this is an issue with the IMAP server or IMP, i.e. I have the hierarchy: Other User's Folders username foldername if the user "username" shared his folder "foldername". And both "username" and "foldername" appear as clickable, valid mailboxes. Thu, 26 Jun 2008 21:18:41 +0000 https://bugs.horde.org/ticket/6972#t46905 > I have prefs set to display namespaces separately and use > subscriptions. Now, if there is a shared folder from another user, > the parent node saying "Other Users' Folders" is displayed (empty), > even if I didn't subscribe to any of those other users' folders. I thought this was a feature? Or did we decide in the past that if a public/other namespace is empty, we would only show if subscriptions were off? We do need to show this namespace somewhere, even if it is empty, if simply to allow the user to create subfolders in this namespace. > Additionally, the user itself is clickable, though I'm not sure if > this is an issue with the IMAP server or IMP, i.e. I have the > hierarchy: > > Other User's Folders > username > foldername > > if the user "username" shared his folder "foldername". And both > "username" and "foldername" appear as clickable, valid mailboxes. Can you check the output from _getList() and see if "namespace.username.folder" is being returned from the IMAP server, or if both "namespace.username.folder" and "namespace.username" is being returned. Obviously, if it is the former than IMP_Tree is buggy, and if it is the latter then the IMAP server is buggy. Thu, 26 Jun 2008 21:59:42 +0000 https://bugs.horde.org/ticket/6972#t46907 >> I have prefs set to display namespaces separately and use >> subscriptions. Now, if there is a shared folder from another user, >> the parent node saying "Other Users' Folders" is displayed (empty), >> even if I didn't subscribe to any of those other users' folders. > > I thought this was a feature? Or did we decide in the past that if a > public/other namespace is empty, we would only show if subscriptions > were off? We do need to show this namespace somewhere, even if it is > empty, if simply to allow the user to create subfolders in this > namespace. At least for the shared folders namespace, you can't create any folder directly under it anyway. And we don't show the Public Folders node either, if there aren't any folders below it. >> Additionally, the user itself is clickable, though I'm not sure if >> this is an issue with the IMAP server or IMP, i.e. I have the >> hierarchy: >> >> Other User's Folders >> username >> foldername >> >> if the user "username" shared his folder "foldername". And both >> "username" and "foldername" appear as clickable, valid mailboxes. > > Can you check the output from _getList() and see if > "namespace.username.folder" is being returned from the IMAP server, > or if both "namespace.username.folder" and "namespace.username" is > being returned. Obviously, if it is the former than IMP_Tree is > buggy, and if it is the latter then the IMAP server is buggy. It only contains "namespace.username.folder". Thu, 26 Jun 2008 22:33:26 +0000 https://bugs.horde.org/ticket/6972#t46909 Changes have been made in CVS for this ticket: http://cvs.horde.org/diff.php/imp/lib/IMAP/Tree.php?r1=1.192&r2=1.193&ty=u Fri, 27 Jun 2008 00:56:49 +0000 https://bugs.horde.org/ticket/6972#t46911 With my latest changes, everything seems to be working (at least for me). Fri, 27 Jun 2008 02:15:16 +0000 https://bugs.horde.org/ticket/6972#t46917 Not for me, the issue with the empty parent node still exists. I noticed though, that this doesn't happen from the start, but only after first showing all folders, and then going back to only showing subscribed folders (in the folders navigator). Fri, 27 Jun 2008 07:09:40 +0000 https://bugs.horde.org/ticket/6972#t46920 > With my latest changes, everything seems to be working (at least for me). Before the patch,we were able to see/use shared folders with the 'imap_config' option. now, we don't see shared folders any more with or without the imap_config option. We use courier-imap-4.3.0. $servers['imap'] = array( 'name' => 'Courier IMAP Server', 'server' => '127.0.0.1', 'hordeauth' => true, 'protocol' => 'imap/notls', 'port' => 143, 'maildomain' => 'mbpgroup.com', 'smtphost' => '127.0.0.1', 'smtpport' => 25, 'realm' => '', 'preferred' => '', 'quota' => array( 'driver' => 'imap', 'params' => array(), ), 'acl' => array( 'driver' => 'rfc2086', /* ), 'imap_config' => array( 'children' => true, 'namespace' => array( '' => array( 'name' => '', 'delimiter' => '.', 'type' => 'personal', 'hidden' => false ), '#shared.' => array( 'name' => '#shared.', 'delimiter' => '.', 'type' => 'personal', 'hidden' => false ), ), 'search_charset' => array( 'UTF-8' => true ) */ ) ); Fri, 27 Jun 2008 08:04:43 +0000 https://bugs.horde.org/ticket/6972#t46921 changing the pref value of tree_view gives me this in http's logs : [Fri Jun 27 10:28:52 2008] [error] [client 192.168.100.188] PHP Notice: Undefined variable: name in /var/www/html/horde/imp/lib/IMAP/Tree.php on line 483, referer: http://192.168.1.22/horde/services/prefs.php?app=imp&group=display [Fri Jun 27 10:28:52 2008] [error] [client 192.168.100.188] PHP Notice: Undefined index: in /var/www/html/horde/imp/lib/IMAP/Tree.php on line 483, referer: http://192.168.1.22/horde/services/prefs.php?app=imp&group=display [Fri Jun 27 10:28:52 2008] [error] [client 192.168.100.188] PHP Notice: Undefined variable: name in /var/www/html/horde/imp/lib/IMAP/Tree.php on line 483, referer: http://192.168.1.22/horde/services/prefs.php?app=imp&group=display Fri, 27 Jun 2008 08:31:18 +0000 https://bugs.horde.org/ticket/6972#t46923 > changing the pref value of tree_view gives me this in http's logs : > [Fri Jun 27 10:28:52 2008] [error] [client 192.168.100.188] PHP > Notice: Undefined variable: name in > /var/www/html/horde/imp/lib/IMAP/Tree.php on line 483, referer: > http://192.168.1.22/horde/services/prefs.php?app=imp&group=display There is something really wrong with your namespace detection. Put in: print_r($this->_namespaces); above line 469 ("foreach ($this->_namespaces as $key => $val) {") and report the output. Fri, 27 Jun 2008 16:48:57 +0000 https://bugs.horde.org/ticket/6972#t46943 Changes have been made in CVS for this ticket: http://cvs.horde.org/diff.php/imp/lib/IMAP/Tree.php?r1=1.193&r2=1.194&ty=u Fri, 27 Jun 2008 17:02:58 +0000 https://bugs.horde.org/ticket/6972#t46946 > Not for me, the issue with the empty parent node still exists. I > noticed though, that this doesn't happen from the start, but only > after first showing all folders, and then going back to only showing > subscribed folders (in the folders navigator). Works for me still. I tried playing around with a few more things so maybe my last commit will help. Fri, 27 Jun 2008 17:03:35 +0000 https://bugs.horde.org/ticket/6972#t46947 Great! It fixed both, the linking, and the empty node problem. Fri, 27 Jun 2008 21:32:22 +0000 https://bugs.horde.org/ticket/6972#t46950 This code seems to be working well for everyone. I think we should keep it in HEAD a bit longer and if we see no further major issues, merge to FW_3 for 4.2.2. Or maybe this is such an upgrade over the current code that it would be worth it to merge to 4.2.1, reasoning that any bugs in it are less critical than the bugs in the current code? Mon, 30 Jun 2008 18:22:02 +0000 https://bugs.horde.org/ticket/6972#t47022 I vote for merging it into 4.2.1. We'll be backporting it if it isn't. So.. at this point, these changes effect lib/IMAP/Tree.php, lib/Session.php and config/servers.php? Mon, 30 Jun 2008 18:39:34 +0000 https://bugs.horde.org/ticket/6972#t47024 I agree on merging it - it seems to have cleared up 4-5 bugs so far. Mon, 30 Jun 2008 18:43:07 +0000 https://bugs.horde.org/ticket/6972#t47025 > So.. at this point, these changes effect lib/IMAP/Tree.php, > lib/Session.php and config/servers.php? Really, only lib/IMAP/Tree.php is needed. The other 2 files remove a no-longer needed configuration variable. Mon, 30 Jun 2008 18:59:44 +0000 https://bugs.horde.org/ticket/6972#t47030 Changes have been made in CVS for this ticket: http://cvs.horde.org/diff.php/imp/config/servers.php.dist?r1=1.50.2.21&r2=1.50.2.22&ty=u http://cvs.horde.org/diff.php/imp/docs/CHANGES?r1=1.699.2.319&r2=1.699.2.320&ty=u http://cvs.horde.org/diff.php/imp/lib/IMAP/Tree.php?r1=1.25.2.51&r2=1.25.2.52&ty=u http://cvs.horde.org/diff.php/imp/lib/Session.php?r1=1.74.2.41&r2=1.74.2.42&ty=u Mon, 30 Jun 2008 20:11:40 +0000 https://bugs.horde.org/ticket/6972#t47045 On further reflection, agreed that merging to 4.2.1 makes the most sense. Mon, 30 Jun 2008 20:12:27 +0000 https://bugs.horde.org/ticket/6972#t47047