Kolab backend: IMAP cache does not get updated

Hi, I'm currently crosstesting the horde 5 RC with anothe

Mon, 29 Oct 2012 15:47:00 +0000

Hi, I'm currently crosstesting the horde 5 RC with another Kolab client. It looks like H5 does not see updates from the other client. I created a new note in the other client and checked on the server that it's synced out. mnemo of H5 will not see this new note. mnemo of horde -4- instantly sees this note. When I delete all horde cashes, the note gets synced in. Same behavior with nag. I suspect (read: untested) that another horde installation accessing the same IMAP server will behave in the same way. Cheers, Thomas

I tried to bring back Jan's old hotfix for #11498, (commit

Wed, 31 Oct 2012 10:11:49 +0000

I tried to bring back Jan's old hotfix for #11498, (commit 5bc02b6acd8acc2024626d9d21ec2834609d7721, trigger sync if UID next changes) though it didn't make a difference.

This is currently the expected behavior as the implemented s

Wed, 31 Oct 2012 18:34:35 +0000

This is currently the expected behavior as the implemented synchronization strategy is pretty dumb. The old code had no synchronization strategy and so it always synchronized the data on access. You should be able to go back to the old behavior by modifying Horde/Kolab/Storage/Synchronization.php. Add a "if (false && ..." to both conditionals present in that file. Does that revert back to the expected behavior? I lack a final plan concerning the synchronization strategy. The problem with the synchronization is that each list synchronization costs two IMAP calls (LIST and METADATA) and each data synchronization as well (STATUS and SEARCH - the latter identifies deleted messages). Doing this type of synchronization with each and every data access keeps the data in sync always. But of course it is a performance drag. Especially if you build the calendar and have access to ten or twenty different folders. And most of the times this type of synchronization is not really necessary. What describe - concurrent access with two clients to the same folder - is not the default case after all. At least not for personal folders. At some point I wanted to create the "kolab" Horde application to offer at least a "Sync now" button to provide the user with control over the synchronization strategy. This would put the user into control of the synchronization strategy, would avoid performance issues but would not be an improvement of usability. Probably there are better solutions. Any suggestions?

> This is currently the expected behavior as the implemented

Wed, 31 Oct 2012 19:03:52 +0000

> This is currently the expected behavior as the implemented > synchronization strategy is pretty dumb. The old code had no > synchronization strategy and so it always synchronized the data on > access. You should be able to go back to the old behavior by > modifying Horde/Kolab/Storage/Synchronization.php. Add a "if (false > && ..." to both conditionals present in that file. Does that revert > back to the expected behavior? > > I lack a final plan concerning the synchronization strategy. The > problem with the synchronization is that each list synchronization > costs two IMAP calls (LIST and METADATA) and each data > synchronization as well (STATUS and SEARCH - the latter identifies > deleted messages). > > Doing this type of synchronization with each and every data access > keeps the data in sync always. But of course it is a performance > drag. Especially if you build the calendar and have access to ten or > twenty different folders. And most of the times this type of > synchronization is not really necessary. What describe - concurrent > access with two clients to the same folder - is not the default case > after all. At least not for personal folders. > > At some point I wanted to create the "kolab" Horde application to > offer at least a "Sync now" button to provide the user with control > over the synchronization strategy. This would put the user into > control of the synchronization strategy, would avoid performance > issues but would not be an improvement of usability. Probably there > are better solutions. > > Any suggestions? At least when using Horde_Imap_Client we can leverage from the IMAP server's feedback whether the cache needs updating.

> I lack a final plan concerning the synchronization strateg

Wed, 31 Oct 2012 19:05:20 +0000

> I lack a final plan concerning the synchronization strategy. The > problem with the synchronization is that each list synchronization > costs two IMAP calls (LIST and METADATA) and each data > synchronization as well (STATUS and SEARCH - the latter identifies > deleted messages). Any reason you are not using CONDSTORE/QRESYNC if it is available? QRESYNC solves the last issue entirely (and CONDSTORE solves the last issue transparently within the Imap driver). However, a cache backend is needed to use these. Given the activesync may use MODSEQs also without using a cache driver, it may be useful to allow a MODSEQ to be explicitly given to the openMailbox command. Although this has major drawbacks, since openMailbox() is called extensively within the IMAP driver itself so a mailbox may already be open by the time a MODSEQ can be provided. A better option might be to create a custom cache driver to pass to the IMAP driver.

>> I lack a final plan concerning the synchronization strate

Wed, 31 Oct 2012 19:34:24 +0000

>> I lack a final plan concerning the synchronization strategy. The >> problem with the synchronization is that each list synchronization >> costs two IMAP calls (LIST and METADATA) and each data >> synchronization as well (STATUS and SEARCH - the latter identifies >> deleted messages). > > Any reason you are not using CONDSTORE/QRESYNC if it is available? No. I planned to use these but never found the time to actually get into this and write the necessary code. What you write sounds however as if this would avoid even more IMAP calls than I expected. My assumption was that CONDSTORE would allow me to reduce the two IMAP calls in each situation to a single one. If there is even more to be gained here then this makes implementing this even more attractive. > QRESYNC solves the last issue entirely (and CONDSTORE solves the last > issue transparently within the Imap driver). However, a cache > backend is needed to use these. > > Given the activesync may use MODSEQs also without using a cache > driver, it may be useful to allow a MODSEQ to be explicitly given to > the openMailbox command. Although this has major drawbacks, since > openMailbox() is called extensively within the IMAP driver itself so > a mailbox may already be open by the time a MODSEQ can be provided. > > A better option might be to create a custom cache driver to pass to > the IMAP driver.

>> Any reason you are not using CONDSTORE/QRESYNC if it is a

Wed, 31 Oct 2012 21:27:38 +0000

>> Any reason you are not using CONDSTORE/QRESYNC if it is available? > > No. I planned to use these but never found the time to actually get > into this and write the necessary code. What you write sounds however > as if this would avoid even more IMAP calls than I expected. My > assumption was that CONDSTORE would allow me to reduce the two IMAP > calls in each situation to a single one. This is not entirely correct. A bit of explanation is probably necessary. In short, without additional IMAP calls CONDSTORE can only tell you whether a mailbox has changed from a previous state. If caching on the local host, it still requires two separate IMAP calls (one to check for flag changes, one to check for deleted messages) to determine what has changed. Further, things may have changed in the mailbox that you don't care about (i.e. UIDs not visible), but CONDSTORE won't provide you this information. CONDSTORE will, however, allow you to forgo additional mailbox sync calls if the mod sequence value hasn't changed since the last access - thus, if the mailbox hasn't changed, you won't need to issue those 2 additional IMAP calls. If a client does its homework before opening a mailbox, QRESYNC instead allows the local cache to be updated without *any* further IMAP call (at the expense of a more involved SELECT/EXAMINE command) - all sync information is passed as untagged responses in the SELECT/EXAMINE return.

> This is currently the expected behavior as the implemented

Mon, 05 Nov 2012 15:57:31 +0000

> This is currently the expected behavior as the implemented > synchronization strategy is pretty dumb. The old code had no > synchronization strategy and so it always synchronized the data on > access. You should be able to go back to the old behavior by > modifying Horde/Kolab/Storage/Synchronization.php. Add a "if (false > && ..." to both conditionals present in that file. Does that revert > back to the expected behavior? Ok, I understand why the old behavior is inappropriate for many folders "subscribed" at the same time. The new strategy is quite a change to H4, we probably should have mentioned it in the changelog. > I lack a final plan concerning the synchronization strategy For plain dumb IMAP, what about syncing the folders every XX minutes (+ random threshold to distribute the server load a bit better)? Also on login and folder "subscribe". The sync time should be user configurable, may be 5 minutes as default. > What describe - concurrent access with two clients to the same folder - is not the default case after all. At least not for personal folders. For the "groupware" case it's the default. For example a secretary managing appointments for another person.

> You should be able to go back to the old behavior by > m

Mon, 05 Nov 2012 16:31:13 +0000

> You should be able to go back to the old behavior by > modifying Horde/Kolab/Storage/Synchronization.php. Add a "if (false > && ..." to both conditionals present in that file. Does that revert > back to the expected behavior? That -almost- worked out: It slows down to a crawl :o) It looks like all Kolab shares are synced on every click. That's slow for sure. It would be nice if it just syncs the shares of the currently active application. Though I consider this a workaround compared with my previously mentioned strategy.

I'll point you to this commit: http://git.horde.org/horde

Wed, 07 Nov 2012 03:36:08 +0000

I'll point you to this commit: http://git.horde.org/horde-git/-/commit/303efdcb9b58a4c65520ce8939e5c8f3772c75fc This (or something similar) will be a part of Horde_Imap_Client 2.2.0. The combo of getSyncToken()/sync() should provide the necessary information on whether the mailbox has changed. Not sure exactly what information you are caching, but most likely this function pair will help out at least a bit.

I've implemented a basic strategy to sync the folder data at

Wed, 07 Nov 2012 17:16:38 +0000

I've implemented a basic strategy to sync the folder data at least every XX minutes. Please see the attached patch. Right now the config values are constants in the class. Jan agreed to make these configurable if needed. The folder -list- is synced on login only. I need further testing if a similar strategy can be implemented here without much performance penalty.

Here's a patch to update the unit test, too.

Fri, 09 Nov 2012 09:55:16 +0000

Here's a patch to update the unit test, too.

Changes have been made in Git (master): commit 66955b963bd3

Thu, 22 Nov 2012 19:24:17 +0000

Changes have been made in Git (master): commit 66955b963bd3f85ddf712ca49a06784bc8811e93 Author: Gunnar Wrobel Date: Wed Nov 21 09:09:15 2012 +0100 Add a time based synchronization strategy. Bug #11590. This is only part of the job. Horde/Kolab/Storage/Factory.php now still needs to bridge the configuration and the selection of the matching strategy. .../lib/Horde/Kolab/Storage/Synchronization.php | 37 +++-- .../Storage/Synchronization/OncePerSession.php | 67 +++++++++ .../Kolab/Storage/Synchronization/TimeBased.php | 115 ++++++++++++++++ .../Unit/Synchronization/OncePerSessionTest.php | 131 ++++++++++++++++++ .../Storage/Unit/Synchronization/TimeBasedTest.php | 145 ++++++++++++++++++++ 5 files changed, 481 insertions(+), 14 deletions(-) http://git.horde.org/horde-git/-/commit/66955b963bd3f85ddf712ca49a06784bc8811e93

Thanks for the patch. I don't think the "Sync once per sessi

Thu, 22 Nov 2012 20:25:19 +0000

Thanks for the patch. I don't think the "Sync once per session" strategy is completely wrong though and makes sense in situations where Horde is the primary Kolab client and there is no synchronization with other clients necessary. So I did not want to simply overwrite it with the new time based strategy. Instead I added both. As discussed here there should probably additional strategies in the future that would rely on advanced IMAP features. The time-based strategy holds the core of your patch now. I only removed the "protection against clock jumps". If you are logged in to a server where the clock suddenly jumps backward the missing synchronization with other Kolab clients is the least of my worries ;) -> simply log out and you are fine again. What is missing now is the selection (and potential configuration) of the strategy within Horde/Kolab/Storage/Factory.php. In the long run I would probably prefer to have a Kolab specific user preference. For now I guess it should be server-wide based on a configuration value though. Can you prepare such a patch?

> I don't think the "Sync once per session" > strategy is

Fri, 23 Nov 2012 13:39:07 +0000

> I don't think the "Sync once per session" > strategy is completely wrong though and makes sense in situations > where Horde is the primary Kolab client and there is no > synchronization with other clients necessary. Synchronization with other clients doesn't imply synchronization with other users :) Even if Horde is the only client, I still want to see updates to shared calendars, f.e. a company wide calendar. -> I still opt for the time based sync by default.

ping Gunnar :)

Thu, 25 Jul 2013 15:24:58 +0000

ping Gunnar :)

This should really be implemented using synchronization feat

Tue, 27 Aug 2013 10:50:15 +0000

This should really be implemented using synchronization features of Horde_Imap_Client.