This should really be implemented using synchronization features of 
Horde_Imap_Client.

ping Gunnar :)

Synchronization with other clients doesn't imply synchronization with 
other users :)
Even if Horde is the only client, I still want to see updates to 
shared calendars, f.e. a company wide calendar.

-> I still opt for the time based sync by default.

Thanks for the patch. I don't think the "Sync once per session" 
strategy is completely wrong though and makes sense in situations 
where Horde is the primary Kolab client and there is no 
synchronization with other clients necessary. So I did not want to 
simply overwrite it with the new time based strategy. Instead I added 
both. As discussed here there should probably additional strategies in 
the future that would rely on advanced IMAP features.

The time-based strategy holds the core of your patch now. I only 
removed the "protection against clock jumps". If you are logged in to 
a server where the clock suddenly jumps backward the missing 
synchronization with other Kolab clients is the least of my worries ;) 
-> simply log out and you are fine again.

What is missing now is the selection (and potential configuration) of 
the strategy within Horde/Kolab/Storage/Factory.php. In the long run I 
would probably prefer to have a Kolab specific user preference.   For 
now I guess it should be server-wide based on a configuration value 
though. Can you prepare such a patch?

Changes have been made in Git (master):

commit 66955b963bd3f85ddf712ca49a06784bc8811e93
Author: Gunnar Wrobel <p@rdus.de>
Date:   Wed Nov 21 09:09:15 2012 +0100

     Add a time based synchronization strategy. Bug #11590.

     This is only part of the job. Horde/Kolab/Storage/Factory.php now 
still needs to bridge the configuration and the selection of the 
matching strategy.

  .../lib/Horde/Kolab/Storage/Synchronization.php    |   37 +++--
  .../Storage/Synchronization/OncePerSession.php     |   67 +++++++++
  .../Kolab/Storage/Synchronization/TimeBased.php    |  115 ++++++++++++++++
  .../Unit/Synchronization/OncePerSessionTest.php    |  131 ++++++++++++++++++
  .../Storage/Unit/Synchronization/TimeBasedTest.php |  145 
++++++++++++++++++++
  5 files changed, 481 insertions(+), 14 deletions(-)

http://git.horde.org/horde-git/-/commit/66955b963bd3f85ddf712ca49a06784bc8811e93

Here's a patch to update the unit test, too.

I've implemented a basic strategy to sync the folder data at least 
every XX minutes.
Please see the attached patch. Right now the config values are 
constants in the class.
Jan agreed to make these configurable if needed.

The folder -list- is synced on login only. I need further testing if a 
similar strategy can be implemented here without much performance 
penalty.

I'll point you to this commit:

http://git.horde.org/horde-git/-/commit/303efdcb9b58a4c65520ce8939e5c8f3772c75fc

This (or something similar) will be a part of Horde_Imap_Client 2.2.0. 
  The combo of getSyncToken()/sync() should provide the necessary 
information on whether the mailbox has changed.  Not sure exactly what 
information you are caching, but most likely this function pair will 
help out at least a bit.

That -almost- worked out: It slows down to a crawl :o) It looks like 
all Kolab shares are synced on every click. That's slow for sure. It 
would be nice if it just syncs the shares of the currently active 
application. Though I consider this a workaround compared with my 
previously mentioned strategy.

Ok, I understand why the old behavior is inappropriate for many 
folders "subscribed" at the same time. The new strategy is quite a 
change to H4, we probably should have mentioned it in the changelog.
For plain dumb IMAP, what about syncing the folders every XX minutes 
(+ random threshold to distribute the server load a bit better)? Also 
on login and folder "subscribe".
The sync time should be user configurable, may be 5 minutes as default.
For the "groupware" case it's the default. For example a secretary 
managing appointments for another person.

This is not entirely correct.  A bit of explanation is probably necessary.

In short, without additional IMAP calls CONDSTORE can only tell you 
whether a mailbox has changed from a previous state.  If caching on 
the local host, it still requires two separate IMAP calls (one to 
check for flag changes, one to check for deleted messages) to 
determine what has changed.  Further, things may have changed in the 
mailbox that you don't care about (i.e. UIDs not visible), but 
CONDSTORE won't provide you this information.  CONDSTORE will, 
however, allow you to forgo additional mailbox sync calls if the mod 
sequence value hasn't changed since the last access - thus, if the 
mailbox hasn't changed, you won't need to issue those 2 additional 
IMAP calls.

If a client does its homework before opening a mailbox, QRESYNC 
instead allows the local cache to be updated without *any* further 
IMAP call (at the expense of a more involved SELECT/EXAMINE command) - 
all sync information is passed as untagged responses in the 
SELECT/EXAMINE return.

No. I planned to use these but never found the time to actually get 
into this and write the necessary code. What you write sounds however 
as if this would avoid even more IMAP calls than I expected. My 
assumption was that CONDSTORE would allow me to reduce the two IMAP 
calls in each situation to a single one. If there is even more to be 
gained here then this makes implementing this even more attractive.

Any reason you are not using CONDSTORE/QRESYNC if it is available?   
QRESYNC solves the last issue entirely (and CONDSTORE solves the last 
issue transparently within the Imap driver).  However, a cache backend 
is needed to use these.

Given the activesync may use MODSEQs also without using a cache 
driver, it may be useful to allow a MODSEQ to be explicitly given to 
the openMailbox command.  Although this has major drawbacks, since 
openMailbox() is called extensively within the IMAP driver itself so a 
mailbox may already be open by the time a MODSEQ can be provided.

A better option might be to create a custom cache driver to pass to 
the IMAP driver.

At least when using Horde_Imap_Client we can leverage from the IMAP 
server's feedback whether the cache needs updating.

This is currently the expected behavior as the implemented 
synchronization strategy is pretty dumb. The old code had no 
synchronization strategy and so it always synchronized the data on 
access. You should be able to go back to the old behavior by modifying 
Horde/Kolab/Storage/Synchronization.php. Add a "if (false && ..." to 
both conditionals present in that file. Does that revert back to the 
expected behavior?

I lack a final plan concerning the synchronization strategy. The 
problem with the synchronization is that each list synchronization 
costs two IMAP calls (LIST and METADATA) and each data synchronization 
as well (STATUS and SEARCH - the latter identifies deleted messages).

Doing this type of synchronization with each and every data access 
keeps the data in sync always. But of course it is a performance drag. 
Especially if you build the calendar and have access to ten or twenty 
different folders. And most of the times this type of synchronization 
is not really necessary. What describe - concurrent access with two 
clients to the same folder - is not the default case after all. At 
least not for personal folders.

At some point I wanted to create the "kolab" Horde application to 
offer at least a "Sync now" button to provide the user with control 
over the synchronization strategy. This would put the user into 
control of the synchronization strategy, would avoid performance 
issues but would not be an improvement of usability. Probably there 
are better solutions.

Any suggestions?

I tried to bring back Jan's old hotfix for #11498,
(commit 5bc02b6acd8acc2024626d9d21ec2834609d7721, trigger sync if UID 
next changes)
though it didn't make a difference.

Hi,

I'm currently crosstesting the horde 5 RC with another Kolab client.
It looks like H5 does not see updates from the other client.

I created a new note in the other client and checked on the server 
that it's synced out.
mnemo of H5 will not see this new note. mnemo of horde -4- instantly 
sees this note.

When I delete all horde cashes, the note gets synced in.

Same behavior with nag. I suspect (read: untested) that another horde 
installation
accessing the same IMAP server will behave in the same way.


Cheers,
Thomas