Alright, you convinced me. Consider the matter solved.

Fixed.
It all depends on your code.
No, not necessarily.  At worst if would issue a NOOP.  Otherwise, the 
mailbox needs to be opened anyway for the SEARCH command.  Either way, 
the downside is the potential for sending a NOOP as a way to possibly 
prevent opening the mailbox.  The former is MUCH preferred over the 
latter (opening a mailbox is expensive).
But as you pointed out with vanished(), there's places internally 
where we call status().  A library user would otherwise have no 
control over these calls.  There has to be a way to disable caching so 
that this internal calls don't potentially return different data than 
a status() call with an explicit STATUS_FORCE_REFRESH.

Changes have been made in Git (master):

commit dd1b0fc6edc26f27ec3386b789edb5de2087e2c7
Author: Michael M Slusarz <slusarz@horde.org>
Date:   Fri Aug 23 12:36:45 2013 -0600

     Fix status flag (Bug #12589)

  .../Imap_Client/lib/Horde/Imap/Client/Base.php     |    2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)

http://git.horde.org/horde-git/-/commit/dd1b0fc6edc26f27ec3386b789edb5de2087e2c7

Thanks! Please take a look at the change again, I think you 
accidentally added the wrong flag.
(STATUS_UIDNEXT_FORCE instead of STATUS_FORCE_REFRESH).

Hmm, I'm still not sure if this is the ideal solution. As you pointed 
out earlier,
this might cause more round trips to the imap server than necessary.

If we look at the search() function for example, in the "long running 
script mode"
it would issue a STATUS command to the server and based on that
result send a SEARCH command or not. In that case it would be better
to just send the SEARCH directly.

I think for most use-cases of long running scripts, the new
STATUS_FORCE_REFRESH flag will do the job just fine as the
library users can decide when to use caching and when not.

This has been added.  To disable status() caching, 
Horde_Imap_Client_Base:$statuscache should be set to false.

Changes have been made in Git (master):

commit 384ceb2be3ac85dd68cca101718654e41d536030
Author: Michael M Slusarz <slusarz@horde.org>
Date:   Fri Aug 23 00:24:39 2013 -0600

     [mms] Added the 'statuscache' property to Horde_Imap_Client_Base 
(Request #12589).

  .../Imap_Client/doc/Horde/Imap/Client/UPGRADING    |    4 ++++
  .../Imap_Client/lib/Horde/Imap/Client/Base.php     |   17 +++++++++++++++++
  framework/Imap_Client/package.xml                  |    2 ++
  3 files changed, 23 insertions(+), 0 deletions(-)

http://git.horde.org/horde-git/-/commit/384ceb2be3ac85dd68cca101718654e41d536030

No.  This is the whole reason we are caching in the first place.  We 
might access the status of a single mailbox 5 times in 0.01 seconds in 
a script.  We should absolutely not be issuing 5 NOOP requests for 
something like this.

I could see adding something like a global "Long-running script" flag 
to the base IMAP object that would indicate that the object is being 
used in a long-running script and STATUS_FORCE_REFRESH should be used 
on every status() call.  But most scripts (under a second) don't need 
this.  Issuing NOOPs is expensive and unnecessary.

while debugging an ActiveSync problem, I've noticed that 
Imap_Client::vanished()
internally calls Imap_Client::search(). The search() function internally uses
status() to optimize for empty mailboxes.

Is that a safe thing to do for a long running script with just one 
selected folder?
I think we need add the STATUS_FORCE_REFRESH flag in case
"current_selected_folder == $mailbox_to_check".

There are also other places we call status().

Kolab pretty early made use of the ANNOTATEMORE extension draft. Back 
in the days you had to patch the c-client library and php to add 
support for it. For distributions that didn't support or want to patch 
their packages, the Net_IMAP pear backend was an alternative (though 
way slower!).

Gunnar kept this concept when he restructured Kolab_Storage though it 
was more of an academic exercise.

Yes, I already wanted to do this years ago, but Gunnar liked to keep 
them because they worked at that point. I doubt this is still true, so 
we should get rid of them.

FWIW, I have no problems with giving Thomas commit access.

Committed.

I would agree.  Although I will admit that I don't understand the 
original use-cases for these other drivers in the first place.

Changes have been made in Git (master):

commit c7bd7f84ee3d788aabb87da3ac8906b41cbd063f
Author: Thomas Jarosch <thomas.jarosch@intra2net.com>
Date:   Tue Aug 20 23:04:11 2013 +0200

     Use new Horde_Imap_Client::STATUS_FORCE_REFRESH flag

     It's more efficient than closing and re-opening the mailbox.

     Bump required Horde_Imap_Client version to 2.14.0.

     Bug #12589

     Signed-off-by: Michael M Slusarz <slusarz@horde.org>

  .../lib/Horde/Kolab/Storage/Driver/Imap.php        |    6 ++----
  framework/Kolab_Storage/package.xml                |    2 +-
  2 files changed, 3 insertions(+), 5 deletions(-)

http://git.horde.org/horde-git/-/commit/c7bd7f84ee3d788aabb87da3ac8906b41cbd063f

Ok, I've adapted the code to use the new flag. Works fine.
Thanks Michael for implementing it!


@Jan: We should make Horde_Imap_Client mandatory for Kolab_Storage.

The other IMAP backend drivers don't get any testing and I highly doubt
they work properly at all.

In fact I'd like to see them vanish for H5.x / H6: I don't see any 
specific use case in supporting the broken c-client, a Roundcube IMAP 
driver or the unmaintained Net_IMAP pear package.

What do you think?

thanks, that's exactly what I was looking for!
(I browsed the Imap_Client source for 30 minutes or so looking for a 
mechanism like this ;))

@mjr: I've only seen this problem with Kolab and only with non-mail 
data. Probably it was triggered because I was only syncing one 
collection and one folder at that time. Otherwise the imap connection 
gets re-used internally and issues SELECT on different folders.

I can look into the proper fix using the new flag probably at the end 
of September, may be earlier.
In fact I would assign the bug to myself but I didn't get my developer 
status back since horde transitioned to git (which was ages ago) ;)

The correct way of fixing this (at least as of Horde_Imap_Client 
2.14.0) will be to add Horde_Imap_Client::STATUS_FORCE_REFRESH to the 
mask passed in the $flags parameter to status().

Changes have been made in Git (master):

commit e8c8d319eec4d57e057e285996fddac4a08c1e7c
Author: Michael M Slusarz <slusarz@horde.org>
Date:   Tue Aug 20 11:31:38 2013 -0600

     [mms] Added the Horde_Imap_Client::STATUS_FORCE_REFRESH flag.

     See, e.g., Ticket #12589

  .../Imap_Client/doc/Horde/Imap/Client/UPGRADING    |   10 +++++
  framework/Imap_Client/lib/Horde/Imap/Client.php    |    2 +
  .../Imap_Client/lib/Horde/Imap/Client/Base.php     |   42 ++++++++++++++-----
  framework/Imap_Client/package.xml                  |   12 +++---
  4 files changed, 49 insertions(+), 17 deletions(-)

http://git.horde.org/horde-git/-/commit/e8c8d319eec4d57e057e285996fddac4a08c1e7c

While we are talking about this: Thomas, was there something specific 
about Kolab that was causing the status to be stale? I have not seen 
problems with status data being stale during normal syncing 
operations...just wondering if there is something that needs to be 
done in the ActiveSync code to avoid this (fringe?) issue.

I would rather see this fixed by introducing a separate status flag 
that forces status() to ping the server.  Closing the mailbox is NOT 
the best way to do this - reopening a mailbox is a fairly expensive 
activity.  The correct method would instead to do a NOOP if the 
mailbox is currently open - this will cause the internal cache to be 
updated without having to reopen the mailbox.

Changes have been made in Git (master):

commit 5df17adcb08d49d05e11897b126dcbfcb1be414d
Author: Jan Schneider <jan@horde.org>
Date:   Tue Aug 20 15:27:19 2013 +0200

     [jan] Retrieve status changes during running requests (Thomas 
Jarosch <thomas.jarosch@intra2net.com>, Bug #12589).

  framework/Kolab_Storage/package.xml |    4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)

http://git.horde.org/horde-git/-/commit/5df17adcb08d49d05e11897b126dcbfcb1be414d

Changes have been made in Git (master):

commit cb27b7cc468247889f8a594c8090c1206b585de7
Author: Thomas Jarosch <thomas.jarosch@intra2net.com>
Date:   Tue Aug 20 14:29:37 2013 +0200

     Ensure the result of status($folder) is not cached by the 
Horde_Imap_Client.

     Otherwise long running scripts (like ActiveSync polling for changes)
     will see outdated data.

     The close() call does not close the IMAP server socket connection.
     It returns to the authenticated, unselected state.

     Bug 12589

     Signed-off-by: Jan Schneider <jan@horde.org>

  .../lib/Horde/Kolab/Storage/Driver/Imap.php        |    3 +++
  1 files changed, 3 insertions(+), 0 deletions(-)

http://git.horde.org/horde-git/-/commit/cb27b7cc468247889f8a594c8090c1206b585de7

From the patch:

Ensure the result of status($folder) is not cached by the Horde_Imap_Client.

Otherwise long running scripts (like ActiveSync polling for changes)
will see outdated data.

The close() call does not close the IMAP server socket connection.
It returns to the authenticated, unselected state.