Seen/unseen status doesn't match IMAP flags status

If I enable the cache in imp/config/backends.local.php (I'm

Wed, 07 Nov 2012 10:22:29 +0000

If I enable the cache in imp/config/backends.local.php (I'm using the advanced IMAP server) $servers['advanced']['cache'] = true; the seen/unseen status of messages in the folder summary doesn't always reflect the actual IMAP flag status (see attached picture, in the stable version of IMP 6 all messages in the Virtual Inbox should be marked as unseen). For most messages, it is correct, but sometimes messages are incorrectly marked as unseen. The attached screenshot was made by going to the folder where I store all Horde related mail, selecting all messages and then marking them as unseen. Immediately afterwards, the messagelist is displayed with all messages as unseen (as expected). If I now go to another folder and back to the Horde folder, it looks as if I didn't change the flags as it shows exactly as before. After logging out and logging in again, all messages from this folder are now listed in the Virtual Inbox (as expected). But the seen/unseen status is as before, some are marked as seen and some as unseen. In the stable version of IMP this should not be possible, as all messages in the Virtual Inbox have the unseen flag set (otherwise they wouldn't be in the Virtual Inbox), so the status must have been cached somewhere. If I then go back to the Horde folder and mark all messages are seen, the messagelist shows all messages as seen (correct). But if I go to another folder and then back to the Horde folder, the same messages as before all this is marked as unseen (as if the flags were not changed). After logging out and logging in again, there are no more messages in the Virtual Inbox (as expected), but the seen/unseen status in the messagelist still shows that some messages are unseen and some seen. By the way, this is not random, each time the same messages are marked as seen. If I disable the IMAP cache by $servers['advanced']['cache'] = false; this doesn't happen and the seen/unseen status as shown in the messagelist always follows the actual IMAP status. I use the following cache settings: $conf['cache']['default_lifetime'] = 86400; $conf['cache']['params']['driverconfig'] = 'horde'; $conf['cache']['driver'] = 'Sql'; $conf['cache']['compress'] = false; $conf['cache']['use_memorycache'] = 'Memcache'; $conf['cachecssparams']['driver'] = 'filesystem'; $conf['cachecssparams']['lifetime'] = 86400; $conf['cachecssparams']['compress'] = 'php'; $conf['cachecss'] = true; $conf['cachejsparams']['driver'] = 'filesystem'; $conf['cachejsparams']['compress'] = 'php'; $conf['cachejsparams']['lifetime'] = 86400; $conf['cachejs'] = true; $conf['cachethemesparams']['check'] = 'appversion'; $conf['cachethemesparams']['lifetime'] = 604800; $conf['cachethemes'] = true; $conf['sessionhandler']['type'] = 'Memcache'; $conf['sessionhandler']['memcache'] = false; $conf['memcache']['hostspec'] = array('localhost'); $conf['memcache']['port'] = array('11211'); $conf['memcache']['weight'] = array(); $conf['memcache']['persistent'] = true; $conf['memcache']['compression'] = false; $conf['memcache']['large_items'] = true; $conf['memcache']['enabled'] = true;

It looks like this only occurs for folders with large number

Wed, 07 Nov 2012 10:57:05 +0000

It looks like this only occurs for folders with large numbers of messages in them (more than about 1000). Currently, the only folder where this occurs is the Horde folder... :-) Coudl this be an action to update the flags on the IMAP server is timing out, which makes Horde cache the previous value while the actual flag has changed?

Have you cleared all your cache files? If you upgraded from

Wed, 07 Nov 2012 19:52:29 +0000

Have you cleared all your cache files? If you upgraded from H4 there may be stale data, and it looks like I never added the purge cache code to the IMP 6.0 upgrade task.

> Have you cleared all your cache files? If you upgraded fr

Wed, 07 Nov 2012 20:01:36 +0000

> Have you cleared all your cache files? If you upgraded from H4 there > may be stale data, and it looks like I never added the purge cache > code to the IMP 6.0 upgrade task. I did clear the cache, but that was before I enabled the IMAP cache (I forgot to copy the 'imp/config/backend.local.php' from the H4 configurations initially). I assumed this was enough, but in any case I just ran 'horde-clear-cache -f' again (after upgrading to the latest released versions of the H5 applications). No surprise that right after that, all is well, but I'll keep an eye on how this develops over the next few days. I'll let you know.

I'm going to mark as resolved for now. There's been so many

Wed, 07 Nov 2012 20:09:10 +0000

I'm going to mark as resolved for now. There's been so many moving parts recently with Imap_Client fixes and upgrade issues that unless there is a place that can be pinpointed, this ticket doesn't provide much input. Clearing the IMAP cache in 6.0.2 should resolve any of these issues anyway.

> I'm going to mark as resolved for now. There's been so ma

Wed, 07 Nov 2012 20:41:03 +0000

> I'm going to mark as resolved for now. There's been so many moving > parts recently with Imap_Client fixes and upgrade issues that unless > there is a place that can be pinpointed, this ticket doesn't provide > much input. Clearing the IMAP cache in 6.0.2 should resolve any of > these issues anyway. And it's broken again... :-( I'm quite sure this will be resolved by clearing the cache, but that pretty much defeats the whole purpose of caching. I'm using Dovecot-2.0.16 by the way, but my feeling is that this has nothing to do with the IMAP server, since the Virtual Inbox picks up the changes in the seen/unseen status without a hitch. Is there anything that can I can do to debug where the problem is?

You will need to provide an IMAP log. Preferably after you

Wed, 07 Nov 2012 20:57:31 +0000

You will need to provide an IMAP log. Preferably after you clear the cache up until the cache becomes broken. Are you using the latest version of Horde_Imap_Client also (2.1.5)?

> You will need to provide an IMAP log. Preferably after yo

Wed, 07 Nov 2012 21:30:49 +0000

> You will need to provide an IMAP log. Preferably after you clear the > cache up until the cache becomes broken. I just did, we'll see what happens. > Are you using the latest version of Horde_Imap_Client also (2.1.5)? Yes.

I'm having the same problem. Steps to reproduce - Log in

Thu, 08 Nov 2012 12:32:31 +0000

I'm having the same problem. Steps to reproduce - Log in - Update some flags within Horde - Log out - Log in a again The changes performed within Horde have disappeared. It seems to only affect the list view. The message count next to the folder is correct. Disabling the IMAP cache fixes the problem.

> I'm having the same problem. > > Steps to reproduce > -

Thu, 08 Nov 2012 12:45:01 +0000

> I'm having the same problem. > > Steps to reproduce > - Log in > - Update some flags within Horde > - Log out > - Log in a again > > The changes performed within Horde have disappeared. It seems to only > affect the list view. The message count next to the folder is correct. There is something really weird. I have enabled the logging in imp/config/backends.local.php $servers['advanced']['debug'] = '/tmp/imaplog'; Since doing that yesterday, the problem has not occured again. It almost looks like enabling the debug fixes the problem. If it doesn't occur again, I'll disable debugging to see if it resurfaces. > Disabling the IMAP cache fixes the problem. Same here.

>> I'm having the same problem. >> >> Steps to reproduce

Thu, 08 Nov 2012 19:21:20 +0000

>> I'm having the same problem. >> >> Steps to reproduce >> - Log in >> - Update some flags within Horde > There is something really weird. I have enabled the logging in > imp/config/backends.local.php > > $servers['advanced']['debug'] = '/tmp/imaplog'; > > Since doing that yesterday, the problem has not occured again. It > almost looks like enabling the debug fixes the problem. If it doesn't > occur again, I'll disable debugging to see if it resurfaces. When debugging is inactive, FLAGS.SILENT is sent to update the flags instead of FLAGS. This means that flag changes (FETCH responses) won't be returned by the server. Which means we normally wouldn't update the cache... ...except this is all known behavior. And irrelevant. We only cache flags if CONDSTORE/QRESYNC is available. If these are available and active, then flag changes MUST always be sent regardless of whether .SILENT is given (RFC 4551 [3.2]). So my guess is that the server is broken because it is not sending back these FETCH responses. The only way to be able to log this will be to alter the Horde_Imap_Client_Socket source (around line 3012): if (!empty($options['replace'])) { $cmd->add(array( - 'FLAGS' . ($this->_debug->debug ? '' : '.SILENT'), + 'FLAGS.SILENT', $options['replace'] )); $cmds[] = $cmd; } else { foreach (array('add' => '+', 'remove' => '-') as $k => $v) { if (!empty($options[$k])) { $cmdtmp = clone $cmd; $cmdtmp->add(array( - $v . 'FLAGS' . ($this->_debug->debug ? '' : '.SILENT'), + $v . 'FLAGS.SILENT', $options[$k] )); $cmds[] = $cmdtmp; } } } Additionally, you can send me the IMAP log you have that was successful so I can check on things like CAPABILITY on your server.

>>> I'm having the same problem. >>> >>> Steps to reproduc

Thu, 08 Nov 2012 19:43:48 +0000

>>> I'm having the same problem. >>> >>> Steps to reproduce >>> - Log in >>> - Update some flags within Horde >> There is something really weird. I have enabled the logging in >> imp/config/backends.local.php >> >> $servers['advanced']['debug'] = '/tmp/imaplog'; >> >> Since doing that yesterday, the problem has not occured again. It >> almost looks like enabling the debug fixes the problem. If it doesn't >> occur again, I'll disable debugging to see if it resurfaces. > > When debugging is inactive, FLAGS.SILENT is sent to update the flags > instead of FLAGS. This means that flag changes (FETCH responses) > won't be returned by the server. Which means we normally wouldn't > update the cache... > > ...except this is all known behavior. And irrelevant. We only cache > flags if CONDSTORE/QRESYNC is available. If these are available and > active, then flag changes MUST always be sent regardless of whether > .SILENT is given (RFC 4551 [3.2]). So my guess is that the server is > broken because it is not sending back these FETCH responses. > > The only way to be able to log this will be to alter the > Horde_Imap_Client_Socket source (around line 3012): > > if (!empty($options['replace'])) { > $cmd->add(array( > - 'FLAGS' . ($this->_debug->debug ? '' : '.SILENT'), > + 'FLAGS.SILENT', > $options['replace'] > )); > $cmds[] = $cmd; > } else { > foreach (array('add' => '+', 'remove' => '-') as $k => $v) { > if (!empty($options[$k])) { > $cmdtmp = clone $cmd; > $cmdtmp->add(array( > - $v . 'FLAGS' . ($this->_debug->debug ? '' : > '.SILENT'), > + $v . 'FLAGS.SILENT', > $options[$k] > )); > $cmds[] = $cmdtmp; > } > } > } OK, done. I have cleared the cache again and we'll see what happens now. > Additionally, you can send me the IMAP log you have that was > successful so I can check on things like CAPABILITY on your server. Sent via private mail.

> OK, done. I have cleared the cache again and we'll see wha

Thu, 08 Nov 2012 19:58:12 +0000

> OK, done. I have cleared the cache again and we'll see what happens now. And almost immediately the error pops up again. I've sent the logs via private mail.

Unfortunately, these logs don't help me out very much since

Thu, 08 Nov 2012 20:54:12 +0000

Unfortunately, these logs don't help me out very much since by the time you open Software.Horde in this log snippet, you have already cached the mailbox: C: 3 SELECT "Software.Horde" (QRESYNC (1220878316 5347 2687:2757,3554:3561,3563:3600,3602:3626,3506:3529,3531:3553,3562,3457:3491,3493:3505,3391:3420,3422:3456,3492,3627:3630)) I need to see the log when this call looks like: C: # SELECT "Software.Horde" because that is when the mailbox is initially sync'd. Then I need to see the log when you are changing a flag in IMP. I would assume this is what you would need to do: Log out of IMP Clear cache Login to IMP Go to Software.Horde Toggle some flags (e.g. seen)

> Unfortunately, these logs don't help me out very much sinc

Thu, 08 Nov 2012 21:17:48 +0000

> Unfortunately, these logs don't help me out very much since by the > time you open Software.Horde in this log snippet, you have already > cached the mailbox: > > C: 3 SELECT "Software.Horde" (QRESYNC (1220878316 5347 > 2687:2757,3554:3561,3563:3600,3602:3626,3506:3529,3531:3553,3562,3457:3491,3493:3505,3391:3420,3422:3456,3492,3627:3630)) > > I need to see the log when this call looks like: > > C: # SELECT "Software.Horde" > > because that is when the mailbox is initially sync'd. Then I need to > see the log when you are changing a flag in IMP. > > I would assume this is what you would need to do: > > Log out of IMP > Clear cache > Login to IMP > Go to Software.Horde > Toggle some flags (e.g. seen) That doesn't work. I did exactly that and there are no lines with just SELECT "Software.Horde" without the QRESYNC on them. See private mail (problem persists, flag updates are not saved).

> That doesn't work. I did exactly that and there are no lin

Thu, 08 Nov 2012 21:30:49 +0000

> That doesn't work. I did exactly that and there are no lines with just > > SELECT "Software.Horde" > > without the QRESYNC on them. See private mail (problem persists, flag > updates are not saved). The message you sent is empty. Instead of: SELECT "Software.Horde" it might instead be EXAMINE "Software.Horde" If you don't see either of these, you are not correctly clearing your cache.

>> That doesn't work. I did exactly that and there are no li

Thu, 08 Nov 2012 21:39:56 +0000

>> That doesn't work. I did exactly that and there are no lines with just >> >> SELECT "Software.Horde" >> >> without the QRESYNC on them. See private mail (problem persists, flag >> updates are not saved). > > The message you sent is empty. > > Instead of: > > SELECT "Software.Horde" > > it might instead be > > EXAMINE "Software.Horde" > > If you don't see either of these, you are not correctly clearing your cache. Oops, forgot the attachment. Indeed it shows up as the latter. I cleared the cache by running horde-clear-cache -f restarted memcached restarted apache I don't know what else to do to clear caches. Somewhere in these logs you should see that I marked a previously 'seen' message as 'unseen'. After logging out and logging in again, the message showed up in the Virtual Inbox (as before), but in the messagelist it was not marked 'unseen'.

Changes have been made in Git (master): commit 2f9d565c1c04

Thu, 08 Nov 2012 22:15:40 +0000

Changes have been made in Git (master): commit 2f9d565c1c04406f2510984fa4b80c623f691922 Author: Michael M Slusarz Date: Thu Nov 8 15:14:52 2012 -0700 [mms] Fix regression that may have caused message flags to have been improperly cached if debugging was disabled (Bug #11665). .../Imap_Client/lib/Horde/Imap/Client/Socket.php | 13 +++++++------ framework/Imap_Client/package.xml | 4 ++-- 2 files changed, 9 insertions(+), 8 deletions(-) http://git.horde.org/horde-git/-/commit/2f9d565c1c04406f2510984fa4b80c623f691922

Changes have been made in Git (master): commit 399f7d8a0c2b

Thu, 08 Nov 2012 22:20:19 +0000

Changes have been made in Git (master): commit 399f7d8a0c2ba476296ce87f0fe6c65978eac2a0 Author: Michael M Slusarz Date: Thu Nov 8 15:19:48 2012 -0700 Bug #11665: Optimize silent check .../Imap_Client/lib/Horde/Imap/Client/Socket.php | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) http://git.horde.org/horde-git/-/commit/399f7d8a0c2ba476296ce87f0fe6c65978eac2a0

Can you try this fix and see if it fixes? I was reading t

Thu, 08 Nov 2012 22:20:28 +0000

Can you try this fix and see if it fixes? I was reading the RFC wrong. A FETCH was required to be sent a FLAGS.SILENT for an UNCHANGEDSINCE STORE, but it was only required to return the MODSEQ data - there was no requirement to return the FLAG changes themselves. So remove .SILENT for all STORE requests if we are implementing flag caching. (You only need to apply the first patch; the second is just an optimization).

> Can you try this fix and see if it fixes? > What's the s

Thu, 08 Nov 2012 22:25:28 +0000

> Can you try this fix and see if it fixes? > What's the simplest way to generate a patch from that commit?

>> Can you try this fix and see if it fixes? >> > What's t

Thu, 08 Nov 2012 22:57:08 +0000

>> Can you try this fix and see if it fixes? >> > What's the simplest way to generate a patch from that commit? That's a good question since you can't do it (at a commit level) from Chora (or even github for that matter). And the file level Chora unified diff generation appears to be broken. I will attach the patch.

> I will attach the patch. Thanks for that. I've just te

Thu, 08 Nov 2012 23:23:07 +0000

> I will attach the patch. Thanks for that. I've just tested the fix and flags are updated properly, with the cache turned on. Thank you.

>>> Can you try this fix and see if it fixes? >>> >> What'

Fri, 09 Nov 2012 07:56:38 +0000

>>> Can you try this fix and see if it fixes? >>> >> What's the simplest way to generate a patch from that commit? > > That's a good question since you can't do it (at a commit level) from > Chora (or even github for that matter). > > And the file level Chora unified diff generation appears to be broken. > > I will attach the patch. If the changes are a handful of changes, I usually apply them manually. Otherwise, there is an option to download just one file from the webinterface of Git, which is more convenient if there are lots of changes. I just downloaded the latest version of framework/Imap_Client/lib/Horde/Imap/Client/Socket.php, which indeed solves the problem of not updating flags. Thanks for the quick solution to this problem.

I guess a positive that came out of this is that I caught an

Fri, 09 Nov 2012 14:28:53 +0000

I guess a positive that came out of this is that I caught an errata in the CONDSTORE RFC. So my initial misreading caused the internet to get better in general :)