I guess a positive that came out of this is that I caught an errata in 
the CONDSTORE RFC.  So my initial misreading caused the internet to 
get better in general :)

If the changes are a handful of changes, I usually apply them 
manually. Otherwise, there is an option to download just one file from 
the webinterface of Git, which is more convenient if there are lots of 
changes. I just downloaded the latest version of 
framework/Imap_Client/lib/Horde/Imap/Client/Socket.php, which indeed 
solves the problem of not updating flags. Thanks for the quick 
solution to this problem.

Thanks for that.

I've just tested the fix and flags are updated properly, with the 
cache turned on.
Thank you.

That's a good question since you can't do it (at a commit level) from 
Chora (or even github for that matter).

And the file level Chora unified diff generation appears to be broken.

I will attach the patch.

What's the simplest way to generate a patch from that commit?

Can you try this fix and see if it fixes?

I was reading the RFC wrong.  A FETCH was required to be sent a 
FLAGS.SILENT for an UNCHANGEDSINCE STORE, but it was only required to 
return the MODSEQ data - there was no requirement to return the FLAG 
changes themselves.  So remove .SILENT for all STORE requests if we 
are implementing flag caching.

(You only need to apply the first patch; the second is just an optimization).

Changes have been made in Git (master):

commit 399f7d8a0c2ba476296ce87f0fe6c65978eac2a0
Author: Michael M Slusarz <slusarz@horde.org>
Date:   Thu Nov 8 15:19:48 2012 -0700

     Bug #11665: Optimize silent check

  .../Imap_Client/lib/Horde/Imap/Client/Socket.php   |    3 ++-
  1 files changed, 2 insertions(+), 1 deletions(-)

http://git.horde.org/horde-git/-/commit/399f7d8a0c2ba476296ce87f0fe6c65978eac2a0

Changes have been made in Git (master):

commit 2f9d565c1c04406f2510984fa4b80c623f691922
Author: Michael M Slusarz <slusarz@horde.org>
Date:   Thu Nov 8 15:14:52 2012 -0700

     [mms] Fix regression that may have caused message flags to have 
been improperly cached if debugging was disabled (Bug #11665).

  .../Imap_Client/lib/Horde/Imap/Client/Socket.php   |   13 +++++++------
  framework/Imap_Client/package.xml                  |    4 ++--
  2 files changed, 9 insertions(+), 8 deletions(-)

http://git.horde.org/horde-git/-/commit/2f9d565c1c04406f2510984fa4b80c623f691922

Oops, forgot the attachment. Indeed it shows up as the latter. I 
cleared the cache by running

horde-clear-cache -f
restarted memcached
restarted apache

I don't know what else to do to clear caches.

Somewhere in these logs you should see that I marked a previously 
'seen' message as 'unseen'. After logging out and logging in again, 
the message showed up in the Virtual Inbox (as before), but in the 
messagelist it was not marked 'unseen'.

The message you sent is empty.

Instead of:

SELECT "Software.Horde"

it might instead be

EXAMINE "Software.Horde"

If you don't see either of these, you are not correctly clearing your cache.

That doesn't work. I did exactly that and there are no lines with just

SELECT "Software.Horde"

without the QRESYNC on them. See private mail (problem persists, flag 
updates are not saved).

Unfortunately, these logs don't help me out very much since by the 
time you open Software.Horde in this log snippet, you have already 
cached the mailbox:

C: 3 SELECT "Software.Horde" (QRESYNC (1220878316 5347 
2687:2757,3554:3561,3563:3600,3602:3626,3506:3529,3531:3553,3562,3457:3491,3493:3505,3391:3420,3422:3456,3492,3627:3630))

I need to see the log when this call looks like:

C: # SELECT "Software.Horde"

because that is when the mailbox is initially sync'd.  Then I need to 
see the log when you are changing a flag in IMP.

I would assume this is what you would need to do:

Log out of IMP
Clear cache
Login to IMP
Go to Software.Horde
Toggle some flags (e.g. seen)

And almost immediately the error pops up again. I've sent the logs via 
private mail.

OK, done. I have cleared the cache again and we'll see what happens now.
Sent via private mail.

When debugging is inactive, FLAGS.SILENT is sent to update the flags 
instead of FLAGS.  This means that flag changes (FETCH responses) 
won't be returned by the server.  Which means we normally wouldn't 
update the cache...

...except this is all known behavior.  And irrelevant.  We only cache 
flags if CONDSTORE/QRESYNC is available.  If these are available and 
active, then flag changes MUST always be sent regardless of whether 
.SILENT is given (RFC 4551 [3.2]).  So my guess is that the server is 
broken because it is not sending back these FETCH responses.

The only way to be able to log this will be to alter the 
Horde_Imap_Client_Socket source (around line 3012):

         if (!empty($options['replace'])) {
             $cmd->add(array(
-               'FLAGS' . ($this->_debug->debug ? '' : '.SILENT'),
+               'FLAGS.SILENT',
                $options['replace']
             ));
             $cmds[] = $cmd;
         } else {
             foreach (array('add' => '+', 'remove' => '-') as $k => $v) {
                 if (!empty($options[$k])) {
                     $cmdtmp = clone $cmd;
                     $cmdtmp->add(array(
-                       $v . 'FLAGS' . ($this->_debug->debug ? '' : 
'.SILENT'),
+                      $v . 'FLAGS.SILENT',
                         $options[$k]
                     ));
                     $cmds[] = $cmdtmp;
                 }
             }
         }

Additionally, you can send me the IMAP log you have that was 
successful so I can check on things like CAPABILITY on your server.

There is something really weird. I have enabled the logging in 
imp/config/backends.local.php

$servers['advanced']['debug'] = '/tmp/imaplog';

Since doing that yesterday, the problem has not occured again. It 
almost looks like enabling the debug fixes the problem. If it doesn't 
occur again, I'll disable debugging to see if it resurfaces.
Same here.

I'm having the same problem.

Steps to reproduce
- Log in
- Update some flags within Horde
- Log out
- Log in a again

The changes performed within Horde have disappeared. It seems to only 
affect the list view. The message count next to the folder is correct.

Disabling the IMAP cache fixes the problem.

I just did, we'll see what happens.
Yes.

You will need to provide an IMAP log.  Preferably after you clear the 
cache up until the cache becomes broken.

Are you using the latest version of Horde_Imap_Client also (2.1.5)?

And it's broken again... :-(

I'm quite sure this will be resolved by clearing the cache, but that 
pretty much defeats the whole purpose of caching. I'm using 
Dovecot-2.0.16 by the way, but my feeling is that this has nothing to 
do with the IMAP server, since the Virtual Inbox picks up the changes 
in the seen/unseen status without a hitch.

Is there anything that can I can do to debug where the problem is?

I'm going to mark as resolved for now.  There's been so many moving 
parts recently with Imap_Client fixes and upgrade issues that unless 
there is a place that can be pinpointed, this ticket doesn't provide 
much input.  Clearing the IMAP cache in 6.0.2 should resolve any of 
these issues anyway.

I did clear the cache, but that was before I enabled the IMAP cache (I 
forgot to copy the 'imp/config/backend.local.php' from the H4 
configurations initially). I assumed this was enough, but in any case 
I just ran 'horde-clear-cache -f' again (after upgrading to the latest 
released versions of the H5 applications). No surprise that right 
after that, all is well, but I'll keep an eye on how this develops 
over the next few days. I'll let you know.

Have you cleared all your cache files?  If you upgraded from H4 there 
may be stale data, and it looks like I never added the purge cache 
code to the IMP 6.0 upgrade task.

It looks like this only occurs for folders with large numbers of 
messages in them (more than about 1000). Currently, the only folder 
where this occurs is the Horde folder... :-)

Coudl this be an action to update the flags on the IMAP server is 
timing out, which makes Horde cache the previous value while the 
actual flag has changed?

If I enable the cache in imp/config/backends.local.php (I'm using the 
advanced IMAP server)

$servers['advanced']['cache'] = true;

the seen/unseen status of messages in the folder summary doesn't 
always reflect the actual IMAP flag status (see attached picture, in 
the stable version of IMP 6 all messages in the Virtual Inbox should 
be marked as unseen). For most messages, it is correct, but sometimes 
messages are incorrectly marked as unseen.

The attached screenshot was made by going to the folder where I store 
all Horde related mail, selecting all messages and then marking them 
as unseen. Immediately afterwards, the messagelist is displayed with 
all messages as unseen (as expected). If I now go to another folder 
and back to the Horde folder, it looks as if I didn't change the flags 
as it shows exactly as before.

After logging out and logging in again, all messages from this folder 
are now listed in the Virtual Inbox (as expected). But the seen/unseen 
status is as before, some are marked as seen and some as unseen. In 
the stable version of IMP this should not be possible, as all messages 
in the Virtual Inbox have the unseen flag set (otherwise they wouldn't 
be in the Virtual Inbox), so the status must have been cached somewhere.

If I then go back to the Horde folder and mark all messages are seen, 
the messagelist shows all messages as seen (correct). But if I go to 
another folder and then back to the Horde folder, the same messages as 
before all this is marked as unseen (as if the flags were not changed).

After logging out and logging in again, there are no more messages in 
the Virtual Inbox (as expected), but the seen/unseen status in the 
messagelist still shows that some messages are unseen and some seen. 
By the way, this is not random, each time the same messages are marked 
as seen.

If I disable the IMAP cache by

$servers['advanced']['cache'] = false;

this doesn't happen and the seen/unseen status as shown in the 
messagelist always follows the actual IMAP status.

I use the following cache settings:

$conf['cache']['default_lifetime'] = 86400;
$conf['cache']['params']['driverconfig'] = 'horde';
$conf['cache']['driver'] = 'Sql';
$conf['cache']['compress'] = false;
$conf['cache']['use_memorycache'] = 'Memcache';
$conf['cachecssparams']['driver'] = 'filesystem';
$conf['cachecssparams']['lifetime'] = 86400;
$conf['cachecssparams']['compress'] = 'php';
$conf['cachecss'] = true;
$conf['cachejsparams']['driver'] = 'filesystem';
$conf['cachejsparams']['compress'] = 'php';
$conf['cachejsparams']['lifetime'] = 86400;
$conf['cachejs'] = true;
$conf['cachethemesparams']['check'] = 'appversion';
$conf['cachethemesparams']['lifetime'] = 604800;
$conf['cachethemes'] = true;
$conf['sessionhandler']['type'] = 'Memcache';
$conf['sessionhandler']['memcache'] = false;
$conf['memcache']['hostspec'] = array('localhost');
$conf['memcache']['port'] = array('11211');
$conf['memcache']['weight'] = array();
$conf['memcache']['persistent'] = true;
$conf['memcache']['compression'] = false;
$conf['memcache']['large_items'] = true;
$conf['memcache']['enabled'] = true;