6.0.0-git
2019-04-24

[#11983] limit synced mailboxes by user preferences
Summary limit synced mailboxes by user preferences
Queue Synchronization
Queue Version Git master
Type Enhancement
State Rejected
Priority 1. Low
Owners
Requester jmozdzen (at) nde (dot) ag
Created 2013-01-20 (2285 days ago)
Due
Updated 2014-04-07 (1843 days ago)
Assigned
Resolved 2014-04-07 (1843 days ago)
Milestone 5.2
Patch Yes

History
2014-04-07 16:32:33 Michael Rubinsky Comment #17
State ⇒ Rejected
Reply to this comment
On reviewing this some more, I'm going to reject this. Exchange has a 
limit of 1000 folders by default, if a client cannot handle up to this 
many folders without falling apart, it's a client issue. This approach 
is just too fraught with the possibility of errors in other parts of 
the code (like searching, message moving etc...).
2013-06-12 18:54:45 Michael Rubinsky State ⇒ Accepted
Milestone ⇒ 5.2
 
2013-02-04 17:47:56 jmozdzen (at) nde (dot) ag Comment #16
New Attachment: Driver.php.diff Download
Reply to this comment

[Show Quoted Text - 10 lines]
I've tried to incorporate your comments - is this new patch sufficient 
for approval?

Concerning the array of special mailboxes - is my approach 
(initializing the member to "null" in the c'tor and verifying against 
that value for lazy loading in "_getMailFolders()") correct?

Let me know if I should further improve this patch.
2013-01-23 21:15:09 jmozdzen (at) nde (dot) ag Comment #15 Reply to this comment
1) You need to review http://www.horde.org/apps/horde/docs/CODING_STANDARDS
2) _getMailFolder() still calls getSpecialMailboxes() and compares 
the names anyway. You should build the map of the special mailboxes 
regardless if it is $inbox_only or not, store it locally in the 
object and use it in getMailFolder() as well.
I'll take your comments and accordingly will re-work the code - but 
this may take a day or two (not that I believe 5.1 is already lurking 
behind the next corner ;) )

That functional change really helped to reduce the load on my device 
to a reasonable level.
2013-01-23 21:12:14 jmozdzen (at) nde (dot) ag Comment #14 Reply to this comment
Today I noticed that my events etc. don't get synced to my remote
device.[...].
No, that's correct. Changes to the folders returned from 
_getMailFolders() would have no effect on synching non-email folders.
Yes, my fault: although I don't recall having changed it, the 
configuration for an add'l calendar to sync was gone - resulting in an 
almost empty calendar on the remote device. Once I re-added that 
calendar in my kronolith sync preferences and fully re-established the 
association (I really had to delete and re-add the account on the 
remote), both sets of events were synced.
2013-01-23 01:51:00 Michael Rubinsky Milestone ⇒ 5.1
 
2013-01-23 01:42:31 Michael Rubinsky Comment #13 Reply to this comment
Feel free to submit a patch and I'll review.
So here's my attempt to a patch to 
/usr/share/php5/PEAR/Horde/Core/ActiveSync/Driver.php
1) You need to review http://www.horde.org/apps/horde/docs/CODING_STANDARDS
2) _getMailFolder() still calls getSpecialMailboxes() and compares the 
names anyway. You should build the map of the special mailboxes 
regardless if it is $inbox_only or not, store it locally in the object 
and use it in getMailFolder() as well.
2013-01-23 01:24:38 Michael Rubinsky Comment #12 Reply to this comment
Tests here work fine so far.
Today I noticed that my events etc. don't get synced to my remote 
device. I had thought that this was handled by _getFolders(), which 
constructs a combined array of collections and mailnfolders, but 
obviously that is insufficient.
No, that's correct. Changes to the folders returned from 
_getMailFolders() would have no effect on synching non-email folders.
2013-01-23 00:57:34 jmozdzen (at) nde (dot) ag Comment #11 Reply to this comment
Tests here work fine so far.
Today I noticed that my events etc. don't get synced to my remote 
device. I had thought that this was handled by _getFolders(), which 
constructs a combined array of collections and mailnfolders, but 
obviously that is insufficient.

2013-01-21 17:18:07 jmozdzen (at) nde (dot) ag Comment #10
New Attachment: inboxonly.tar Download
Reply to this comment
Feel free to submit a patch and I'll review.
So here's my attempt to a patch to 
/usr/share/php5/PEAR/Horde/Core/ActiveSync/Driver.php and a 
corresponding prefs.d file (who's contents ought to be merged into 
prefs.php in case your accepting the patch?)

Tests here work fine so far.


2013-01-21 16:51:58 Michael Rubinsky Comment #9 Reply to this comment
If I had to call _getMailFolder() for every folder returned by 
_imap->getMailboxes(), wouldn't this get too expensive? My list may 
well be longer than 1k entries... each tme creating an instance of 
Horde_ActiveSync_Message_Folder, calling getSpecialMailboxes() etc, 
and then dumping the result in all but a handful of cases.
Well, getSpecialMailboxes() already caches the list so we only hit the 
imap server once.
Purely functionally speaking, the quickest approach would be to call 
_imap->getSpecialMailboxes() only once, re-map to an "folder 
name"-based array, then check every "_imap->getMailboxes()" result 
against that new array. Does that sound acceptable to you?
Feel free to submit a patch and I'll review.
2013-01-21 16:33:04 jmozdzen (at) nde (dot) ag Comment #8 Reply to this comment
Isn't "Horde_Core_ActiveSync_Driver::_getMailFolders()" part of "the
library code" as well? [...]
The framework libraries are designed to be used as stand alone 
libraries - outside of a Horde environment. Core is a special 
library that contains the core libraries needed for a Horde 
environment.
Thanks, that explains it!
Secondly, is there a proper way to detect which folders returned by
"_imap->getMailboxes()" are "special" mailboxes? [...]
This is already done in 
Horde_Core_ActiveSync_Driver::_getMailFolder(). You just need to 
check the type property of the returned folder object.
If I had to call _getMailFolder() for every folder returned by 
_imap->getMailboxes(), wouldn't this get too expensive? My list may 
well be longer than 1k entries... each tme creating an instance of 
Horde_ActiveSync_Message_Folder, calling getSpecialMailboxes() etc, 
and then dumping the result in all but a handful of cases.

Purely functionally speaking, the quickest approach would be to call 
_imap->getSpecialMailboxes() only once, re-map to an "folder 
name"-based array, then check every "_imap->getMailboxes()" result 
against that new array. Does that sound acceptable to you?
2013-01-21 14:44:49 Michael Rubinsky Comment #7 Reply to this comment

[Show Quoted Text - 11 lines]
The framework libraries are designed to be used as stand alone 
libraries - outside of a Horde environment. Core is a special library 
that contains the core libraries needed for a Horde environment.
Secondly, is there a proper way to detect which folders returned by 
"_imap->getMailboxes()" are "special" mailboxes? I believe that even 
if I set preference to only see "INBOX", I'd need at least "Trash" 
and "Sent" as well, and probably "Drafts" (and "Templates"?) too. 
I've seen no specific attributes for these folders and remember that 
some clients I used were able to select which folder to use for 
these functions.
This is already done in 
Horde_Core_ActiveSync_Driver::_getMailFolder(). You just need to check 
the type property of the returned folder object.

2013-01-21 11:41:00 jmozdzen (at) nde (dot) ag Comment #6 Reply to this comment

[Show Quoted Text - 9 lines]
I've move the code to Horde_Core_ActiveSync_Driver::_getMailFolders(), 
which works as expected (at the current stage of development). But two 
questions remain:

Isn't "Horde_Core_ActiveSync_Driver::_getMailFolders()" part of "the 
library code" as well? For both places I've implemented this so far, 
accessing the preferences worked as expected, probably because the 
routines were called in the context of a logged-in user. If this may 
cause problems for other types of invocation (without user context), 
I'd need some architectural guidance.

Secondly, is there a proper way to detect which folders returned by 
"_imap->getMailboxes()" are "special" mailboxes? I believe that even 
if I set preference to only see "INBOX", I'd need at least "Trash" and 
"Sent" as well, and probably "Drafts" (and "Templates"?) too. I've 
seen no specific attributes for these folders and remember that some 
clients I used were able to select which folder to use for these 
functions.
I've seen code to somewhat handle this in IMP - would the "proper" way 
be to use the same code as IMP does, to determine which folder is a 
special folder? I'll then have to dig through IMP to find out if this 
is available to me within the ActiveSync context or if I'll have to 
somehow duplicate it (yuck ;) ).
2013-01-20 21:33:27 Michael Rubinsky Comment #5 Reply to this comment

[Show Quoted Text - 36 lines]
That would introduce a whole other level of messiness since folders 
are more dynamic than address books/calendars. Not that it couldn't be 
done. The reason we do this for calendars/address books is because EAS 
only supports a single one of each of those. The pref allows the user 
to decide which sources to combine into the single data source that is 
sent to the client. We don't have to worry about 
subscriptions/namespaces etc... with those sources.

If we do this, I think I'd do it as a INBOX and Special Folders only 
or all subscribed folders.
That's when I thought that a client-side subscription model, 
implemented on the ActiveSync part (rather than the final mobile 
client) ought to help... and the long time it seems to take to 
process the results of a folder sync, I decided to code a filter 
option for that call. But of course, there may be (and your reply 
says there *are*) side effects I couldn't foresee.
I'd need more testing to see exactly how this would affect search and such.
Do you see a different, better place to implement such a filter? I'd 
be willing to give this another try.
You can't use the prefs object inside the library code. This would 
have to be implemented inside Horde_Core_ActiveSync_Driver:: probably 
in _getMailFolders() or it's own filtering method.
BTW: After running this filter on my server/account, processing time 
and data transfer volume have already decreased. But I see many 
instances of supposedly "filtered" folders in the ActiveSync debug 
log, there still is much work left to actually reduce the handling 
to only those folders I actually want to see on the remote :/
This might be left over from data not updated in the syncCache of 
folder state yet. Once the device requests a new FOLDERSYNC it should 
only request data on those folders. In other words, when it receives a 
FOLDERSYNC response that indicates that folder FOO is no longer 
available, it thinks that folder has been deleted on the server, and 
should not issue any more requests for it.

Update the patch with the above changes and I'll look at adding it in 
Horde 5.1.
2013-01-20 21:28:45 jmozdzen (at) nde (dot) ag Comment #4 Reply to this comment
Ha - seems you're right again:
On the other hand, my remote device won't let me access
folders not under my INBOX at all, and I definitely don't need all
the sub-folders as available through the web interface.
Curious as to what client this is. I've never seen a client limit 
folders to those only under INBOX.
I just noticed that during a development error, I actually had some 
"not under INBOX" folders in the list - and my WebOS client would show 
them. It must have been more of a problem of handling the large number 
of folders, the displayed list is rather bad to handle and either the 
"shared folders" where at the cut off at end of the list or lost 
somewhere in the too long (scrollable) list of folders.

That list handling (within the WebOS mail client) is a PITA when 
having many folders - reason enough to limit the list of displayed 
folders. I couldn't find an option for this within that client, so I'm 
still after the ActiveSync filtering solution ;)
2013-01-20 21:12:46 jmozdzen (at) nde (dot) ag Comment #3 Reply to this comment
I have a fairly large mailbox structure, plus access to various
shared folders. When I synchronize with a remote device, I therefore
generate tons of traffic and load, especially on the "small" remote
device.
Currently, it sends all "subscribed" folders in the FOLDERSYNC 
request. The rationale is that if you have subscribed to the folder, 
you are interested enough in it to want to see it on the device.
When on the road, I usually only need to see if I have new mail and 
optionally reply to it. Some other folders will be required, too, like 
"Sent" and "Trash".
Note that sending the folder via FOLDERSYNC does not necessarily 
mean it pushes that folder, just that it is available on the device 
for viewing. On most of the clients I test, the email is lazy loaded 
into folders other than the INBOX - it only polls the server when 
you view the folder. Some devices will THEN push to that folder for 
some time after it is loaded. Other clients have settings that allow 
you to pick and choose the folders you want pushed vs just available.
It's my observation too that only messages in the INBOX are actually 
pushed to the remote device. But with about 1083 folders (number 
increasing), already processing the list of available folders causes a 
significant slow-down on my remote...
On the other hand, my remote device won't let me access
folders not under my INBOX at all, and I definitely don't need all
the sub-folders as available through the web interface.
Curious as to what client this is. I've never seen a client limit 
folders to those only under INBOX.
It's the default client on WebOS 2.x (HP Pre3).

[Show Quoted Text - 10 lines]
Yes, I had anticipated that situation - being able to select which 
folders to sync, similar to selecting which calendars and address 
books to sync, would cover that.

[Show Quoted Text - 9 lines]
I fully agree to that position, and as I mentioned above, even I would 
have preferred to go a different route... a sufficient solution would 
have been if I could have used IMAP subscriptions and a second account 
- that would have been a favorable solution. But unfortunately I 
couldn't get that to work, probably to my insufficient knowledge of 
the details of the Cyrus IMAP server.

That's when I thought that a client-side subscription model, 
implemented on the ActiveSync part (rather than the final mobile 
client) ought to help... and the long time it seems to take to process 
the results of a folder sync, I decided to code a filter option for 
that call. But of course, there may be (and your reply says there 
*are*) side effects I couldn't foresee.

Do you see a different, better place to implement such a filter? I'd 
be willing to give this another try.

BTW: After running this filter on my server/account, processing time 
and data transfer volume have already decreased. But I see many 
instances of supposedly "filtered" folders in the ActiveSync debug 
log, there still is much work left to actually reduce the handling to 
only those folders I actually want to see on the remote :/
2013-01-20 19:35:16 Michael Rubinsky Comment #2
State ⇒ Feedback
Reply to this comment
I have a fairly large mailbox structure, plus access to various 
shared folders. When I synchronize with a remote device, I therefore 
generate tons of traffic and load, especially on the "small" remote 
device.
Currently, it sends all "subscribed" folders in the FOLDERSYNC 
request. The rationale is that if you have subscribed to the folder, 
you are interested enough in it to want to see it on the device.

Note that sending the folder via FOLDERSYNC does not necessarily mean 
it pushes that folder, just that it is available on the device for 
viewing. On most of the clients I test, the email is lazy loaded into 
folders other than the INBOX - it only polls the server when you view 
the folder. Some devices will THEN push to that folder for some time 
after it is loaded. Other clients have settings that allow you to pick 
and choose the folders you want pushed vs just available.
On the other hand, my remote device won't let me access folders not 
under my INBOX at all, and I definitely don't need all the 
sub-folders as available through the web interface.
Curious as to what client this is. I've never seen a client limit 
folders to those only under INBOX.
As I have been unsuccessful to create a second IMAP user with access 
to the same INBOX, but only few of the folders below, I have created 
a patch to ActiveSync that allows a user to (currently) select
- only to sync folders below and including the INBOX
- limit the nesting level of the folders synced
This of course will only work for IMAP servers that are 
structured/configured in this way. My server does not place folders 
under INBOX at all. I have a few hundred mailboxes (of course not all 
subscribed) and most are at the same level as the INBOX.
The according preferences are settable via the standard preferences 
dialog, per user.

A future enhancement may be to explicitly select folders to 
include/exclude during email sync.
I'd rather not go that route. I feel that this is a client 
responsibility. Also, filtering the folders that are returned via 
FOLDERSYNC responses could screw up search results

2013-01-20 16:09:02 jmozdzen (at) nde (dot) ag Comment #1
Type ⇒ Enhancement
State ⇒ New
Priority ⇒ 1. Low
Summary ⇒ limit synced mailboxes by user preferences
Queue ⇒ Synchronization
Milestone ⇒
Patch ⇒ Yes
New Attachment: limitsync.tar Download
Reply to this comment
I have a fairly large mailbox structure, plus access to various shared 
folders. When I synchronize with a remote device, I therefore generate 
tons of traffic and load, especially on the "small" remote device. On 
the other hand, my remote device won't let me access folders not under 
my INBOX at all, and I definitely don't need all the sub-folders as 
available through the web interface.

As I have been unsuccessful to create a second IMAP user with access 
to the same INBOX, but only few of the folders below, I have created a 
patch to ActiveSync that allows a user to (currently) select
- only to sync folders below and including the INBOX
- limit the nesting level of the folders synced

The according preferences are settable via the standard preferences 
dialog, per user.

A future enhancement may be to explicitly select folders to 
include/exclude during email sync.

I've attached two files:
- the definitions for the user preferences - currently added as 
$webdir/config/prefs.d/30-activesync.php
- the new method Horde_ActiveSync_Imap_Adapter::getMailboxes() from 
/usr/share/php5/PEAR/Horde/ActiveSync/Imap/Adapter.php

I hope this change finds your approval.

Regards,
Jens

Saved Queries