Horde_Imap_Client 2.11.7.

Changes have been made in Git (master):

commit 043148a327cebb6ab817f758bb44155efe5686ab
Author: Michael M Slusarz <slusarz@horde.org>
Date:   Tue Jul 16 16:01:43 2013 -0600

     [mms] More graceful handling of servers that return broken FETCH 
information (Request #12441).

  .../Imap_Client/lib/Horde/Imap/Client/Socket.php   |    4 ++++
  framework/Imap_Client/package.xml                  |    4 ++--
  2 files changed, 6 insertions(+), 2 deletions(-)

http://git.horde.org/horde-git/-/commit/043148a327cebb6ab817f758bb44155efe5686ab

Yeah, I already replaced the courier server with a dovecot 
installation yesterday.  :-)  Didn't want to deal with the hassle of 
having to compile my own .deb file.

...except there is no guarantee that these methods will work either.

  I don't want to be writing workarounds for a single broken IMAP 
server that admittedly completely breaks the protocol.  It *is* a goal 
to make sure that a broken IMAP server doesn't break anything.  But 
there is no reason to perform complex workarounds to try to be able to 
display these messages.  Fix the server - that is the much easier 
solution.

Here's the IMAP log of opening that mailbox and reading that message 
using mutt.

[2013-07-10 06:37:29] 4> a0001 CAPABILITY
a0002 LIST "" ""
[2013-07-10 06:37:29] 4< * CAPABILITY IMAP4rev1 UIDPLUS CHILDREN 
NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE 
AUTH=PLAIN ACL ACL2=UNION
[2013-07-10 06:37:29] 4< a0001 OK CAPABILITY completed
[2013-07-10 06:37:29] 4< * LIST (\Noselect) "." ""
[2013-07-10 06:37:29] 4< a0002 OK LIST completed

[2013-07-10 06:37:29] Selecting INBOX...
[2013-07-10 06:37:29] 4> a0003 MYRIGHTS "INBOX"
a0004 STATUS "INBOX.Drafts" (MESSAGES)
a0005 SELECT "INBOX"
[2013-07-10 06:37:29] 4< * MYRIGHTS "INBOX" "acdilrsw"
[2013-07-10 06:37:29] Handling MYRIGHTS
[2013-07-10 06:37:29] Unknown right: "
[2013-07-10 06:37:29] Unknown right: "
[2013-07-10 06:37:29] 4< a0003 OK MYRIGHTS completed.

[2013-07-10 06:37:29] 4< a0004 NO Mailbox does not exist, or must be 
subscribed to.
[2013-07-10 06:37:29] 4< * FLAGS (\Draft \Answered \Flagged \Deleted 
\Seen \Recent)
[2013-07-10 06:37:29] 4< * OK [PERMANENTFLAGS (\* \Draft \Answered 
\Flagged \Deleted \Seen)] Limited
[2013-07-10 06:37:29] 4< * 1 EXISTS
[2013-07-10 06:37:29] Handling EXISTS
[2013-07-10 06:37:29] cmd_handle_untagged: New mail in INBOX - 1 
messages total.
[2013-07-10 06:37:29] 4< * 0 RECENT
[2013-07-10 06:37:29] 4< * OK [UIDVALIDITY 1370196747] Ok
[2013-07-10 06:37:29] 4< * OK [MYRIGHTS "acdilrsw"] ACL
[2013-07-10 06:37:29] 4< a0005 OK [READ-WRITE] Ok

[2013-07-10 06:37:29] Fetching message headers... 0/1 (0%)
[2013-07-10 06:37:29] 4> a0006 FETCH 1:1 (UID FLAGS INTERNALDATE 
RFC822.SIZE BODY.PEEK[HEADER.FIELDS (DATE FROM SUBJECT TO CC 
MESSAGE-ID REFERENCES CONTENT-TYPE CONTENT-DESCRIPTION IN-REPLY-TO 
REPLY-TO LINES LIST-POST X-LABEL)])
[2013-07-10 06:37:29] 4< * 1 FETCH (UID 4649 FLAGS (\Seen) 
INTERNALDATE "08-Jan-2012 19:31:49 +0000" RFC822.SIZE 737 
BODY[HEADER.FIELDS ("DATE" "FROM" "SUBJECT" "TO" "CC" "MESSAGE-ID" 
"REFERENCES" "CONTENT-TYPE" "CONTENT-DESCRIPTION" "IN-REPLY-TO" 
"REPLY-TO" "LINES" "LIST-POST" "X-LABEL")] {423}
[2013-07-10 06:37:29] imap_read_literal: reading 423 bytes
[2013-07-10 06:37:29] 4< )
[2013-07-10 06:37:29] 4< a0006 OK FETCH completed.
[2013-07-10 06:37:29] Overriding UIDNEXT: 0 -> 4650
[2013-07-10 06:37:29] imap_open_mailbox: msgcount is 1
[2013-07-10 06:37:29] Sorting mailbox...

[2013-07-10 06:37:42] Fetching message...
[2013-07-10 06:37:42] 4> a0007 UID FETCH 4649 BODY.PEEK[]
[2013-07-10 06:37:42] 4< * 1 FETCH (UID 4649 BODY[] {737}
[2013-07-10 06:37:42] Fetching message... 0K/0.7K (0%)
[2013-07-10 06:37:42] imap_read_literal: reading 737 bytes
[2013-07-10 06:37:42] Fetching message... 0K/0.7K (0%)
[2013-07-10 06:37:42] 4< )
[2013-07-10 06:37:42] 4< a0007 OK FETCH completed.

[2013-07-10 06:37:42] WEED is Set
[2013-07-10 06:37:42] Reorder: from matches From: 
=?US-ASCII?Q?Keith_Moore?= <moore@cs.utk.edu>
[2013-07-10 06:37:42] Reorder: x = 1; hdr_count = 6
[2013-07-10 06:37:42] Reorder: to matches To: 
=?ISO-8859-1?Q?Keld_J=F8rn_Simonsen?= <keld@dkuug.dk>, "Blah" 
<blah@example.com>
[2013-07-10 06:37:42] Reorder: x = 2; hdr_count = 6
[2013-07-10 06:37:42] Reorder: cc matches CC: =?ISO-8859-1?Q?Andr=E9?= 
Pirard <PIRARD@vm1.ulg.ac.be>
[2013-07-10 06:37:42] Reorder: x = 3; hdr_count = 6
[2013-07-10 06:37:42] Reorder: subject matches Subject: 
=?ISO-8859-1?B?SWYgeW91IGNhbiByZWFkIHRoaXMgeW8=?=
[2013-07-10 06:37:42] Reorder: x = 4; hdr_count = 6
[2013-07-10 06:37:42] Reorder: x = 4; hdr_count = 6
[2013-07-10 06:37:42] Reorder: date matches Date: Sun, 8 Jan 2012 
14:31:46 -0500
[2013-07-10 06:37:42] Reorder: x = 0; hdr_count = 6
[2013-07-10 06:38:33] Mailbox is unchanged.

I see what you mean about it being less-efficient, but perhaps it 
could be used a fall-back mode by IMP.  It would be less-efficient, 
slower, but it would increase compatibility.

My wild guess is that mutt is probably directly parsing the header 
text.  Which is less efficient, especially with new caching IMAP 
servers.

What's interesting is that mutt running against the same IMAP server 
doesn't have any problem reading this mailbox and its messages.  In 
fact, I was using mutt to edit the message and to toggle the unicode 
text within the header whilst experimenting with IMP.  I'll see how I 
can capture a log of the interaction that mutt has with the IMAP 
server and shall attach that too this bug in case that's useful.

C: 8 UID FETCH 4646 (ENVELOPE FLAGS RFC822.SIZE 
BODY.PEEK[HEADER.FIELDS (IMPORTANCE LIST-POST X-PRIORITY CONTENT-TYPE)])
S: * BYE [ALERT] Fatal error: Invalid argument

In this case, there is nothing we can do to get the information from 
that message.

Although maybe there is a way to handle this more gracefully in terms 
of returning data.  Like returning an entirely empty Fetch object 
instead of erroring out.  I'm going to leave this ticket open for a 
bit to think about this more.

For the record, here's the IMAP debug-log:

S: * OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE 
THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE ACL ACL2=UNION 
STARTTLS] Courier-IMAP ready. Copyright 1998-2011 Double Precision, 
Inc.  See COPYING for distribution information.
C: 1 STARTTLS
S: 1 OK Begin SSL/TLS negotiation now.
2 [LOGIN Command - username: account@domain.com]
S: 2 OK LOGIN Ok.
C: 3 SELECT INBOX
S: * FLAGS (\Draft \Answered \Flagged \Deleted \Seen \Recent)
S: * OK [PERMANENTFLAGS (\* \Draft \Answered \Flagged \Deleted \Seen)] Limited
S: * 1 EXISTS
S: * 0 RECENT
S: * OK [UIDVALIDITY 1370196747] Ok
S: * OK [MYRIGHTS "acdilrsw"] ACL
S: 3 OK [READ-WRITE] Ok
C: 4 UID FETCH * (UID)
S: * 1 FETCH (UID 4646)
S: 4 OK FETCH completed.
C: 5 UID FETCH * (UID)
S: * 1 FETCH (UID 4646)
S: 5 OK FETCH completed.
C: 6 UID SEARCH ALL
S: * SEARCH 4646
S: 6 OK SEARCH done.
C: 7 UID FETCH * (UID)
S: * 1 FETCH (UID 4646)
S: 7 OK FETCH completed.
C: 8 UID FETCH 4646 (ENVELOPE FLAGS RFC822.SIZE 
BODY.PEEK[HEADER.FIELDS (IMPORTANCE LIST-POST X-PRIORITY CONTENT-TYPE)])
S: * BYE [ALERT] Fatal error: Invalid argument
S: * OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE 
THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE ACL ACL2=UNION 
STARTTLS] Courier-IMAP ready. Copyright 1998-2011 Double Precision, 
Inc.  See COPYING for distribution information.
C: 9 STARTTLS
S: 9 OK Begin SSL/TLS negotiation now.
10 [LOGIN Command - username: account@domain.com]
S: 10 OK LOGIN Ok.
C: 11 STATUS INBOX (UNSEEN)
S: * STATUS "INBOX" (UNSEEN 1)
S: 11 OK STATUS Completed.

And here's the same sequence when there was no umlaut in the From field:

S: * OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE 
THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE ACL ACL2=UNION 
STARTTLS] Courier-IMAP ready. Copyright 1998-2011 Double Precision, 
Inc.  See COPYING for distribution information.
C: 1 STARTTLS
S: 1 OK Begin SSL/TLS negotiation now.
2 [LOGIN Command - username: account@domain.com]
S: 2 OK LOGIN Ok.
C: 3 SELECT INBOX
S: * FLAGS (\Draft \Answered \Flagged \Deleted \Seen \Recent)
S: * OK [PERMANENTFLAGS (\* \Draft \Answered \Flagged \Deleted \Seen)] Limited
S: * 1 EXISTS
S: * 0 RECENT
S: * OK [UIDVALIDITY 1370196747] Ok
S: * OK [MYRIGHTS "acdilrsw"] ACL
S: 3 OK [READ-WRITE] Ok
C: 4 UID FETCH * (UID)
S: * 1 FETCH (UID 4647)
S: 4 OK FETCH completed.
C: 5 UID FETCH * (UID)
S: * 1 FETCH (UID 4647)
S: 5 OK FETCH completed.
C: 6 UID SEARCH ALL
S: * SEARCH 4647
S: 6 OK SEARCH done.
C: 7 UID FETCH * (UID)
S: * 1 FETCH (UID 4647)
S: 7 OK FETCH completed.
C: 8 UID FETCH 4647 (ENVELOPE FLAGS RFC822.SIZE 
BODY.PEEK[HEADER.FIELDS (IMPORTANCE LIST-POST X-PRIORITY CONTENT-TYPE)])
S: * 1 FETCH (UID 4647 ENVELOPE ("Sun, 8 Jan 2012 14:31:46 -0500" 
"=?ISO-8859-1?B?SWYgeW91IGNhbiByZWFkIHRoaXMgeW8=?=     
=?ISO-8859-2?B?dSB1bmRlcnN0YW5kIHRoZSBleGFtcGxlLg==?=" 
(("=?US-ASCII?Q?Keith_Moore?=" NIL "moore" "cs.utk.edu")) 
(("=?US-ASCII?Q?Keith_Moore?=" NIL "moore" "cs.utk.edu")) 
(("=?US-ASCII?Q?Keith_Moore?=" NIL "moore" "cs.utk.edu")) 
(("=?ISO-8859-1?Q?Keld_J=F8rn_Simonsen?=" NIL "keld" 
"dkuug.dk")("Blah" NIL "blah" "example.com")) 
(("=?ISO-8859-1?Q?Andr=E9?= Pirard" NIL "PIRARD" "vm1.ulg.ac.be")) NIL 
NIL "<803734518.1326051106099.JavaMail.root@example.com>") FLAGS () 
RFC822.SIZE 737 BODY[HEADER.FIELDS ("IMPORTANCE" "LIST-POST" 
"X-PRIORITY" "CONTENT-TYPE")] {2}
S: [LITERAL DATA: 2 bytes]
S: )
S: 8 OK FETCH completed.
C: 9 SEARCH UNSEEN
S: * SEARCH 1
S: 9 OK SEARCH done.
C: 10 LOGOUT
S: * BYE Courier-IMAP server shutting down
S: 10 OK LOGOUT completed

You're right - I did some more digging, and it turns out it's a 
problem with my version of Courier IMAP.

http://sourceforge.net/mailarchive/forum.php?thread_name=loom.20121212T085156-912%40post.gmane.org&forum_name=courier-users

I use v4.9.1-1ubuntu4 of courier-imap currently, and v4.9.3 should 
have the fix for it.

You were 8 seconds too late Jan :)

Please upload the source of such a broken message. For the record: 
it's well possible that this only happens with your IMAP server, so 
please upload an IMAP log too.

To further debug this issue, we need details of the IMP -> IMAP/POP 
communication.

To enable debugging, see instructions contained in 
imp/config/backends.php (the 'debug' config parameter).

Debugging should not be enabled on a production server,   Attach/post 
only the portion of the log that directly deals with the problem 
reported (it may be simplest to clear the log file and then perform 
the event that causes the error).

No, that shouldn't matter.  *Our* code should handle it.  The issue is 
if the IMAP server sends something strange.

I can verify I can see that message fine when using Dovecot 2.2.4.

The pertinent part of the IMAP debug log:

C: 7 UID FETCH 1 (ENVELOPE FLAGS RFC822.SIZE BODY.PEEK[HEADER.FIELDS 
(IMPORTANCE LIST-POST X-PRIORITY CONTENT-TYPE)])
S: * 1 FETCH (UID 1 FLAGS (\Recent) RFC822.SIZE 534 ENVELOPE ("Sun, 8 
Jan 2012 14:31:46 -0500" {103}
S: [LITERAL DATA: 103 bytes]
S:  (("=?US-ASCII?Q?Keith_Moore?=" NIL "moore" "cs.utk.edu")) 
(("=?US-ASCII?Q?Keith_Moore?=" NIL "moore" "cs.utk.edu")) 
(("=?US-ASCII?Q?Keith_Moore?=" NIL "moore" "cs.utk.edu")) 
(("=?ISO-8859-1?Q?Keld_J=F8rn_Simonsen?=" NIL "keld" "dkuug.dk")({5}
S: [LITERAL DATA: 5 bytes]
S:  NIL "blah" "example.com")) (("=?ISO-8859-1?Q?Andr=E9?= Pirard" NIL 
"PIRARD" "vm1.ulg.ac.be")) NIL NIL 
"<803734518.1326051106099.JavaMail.root@example.com>") 
BODY[HEADER.FIELDS (IMPORTANCE LIST-POST X-PRIORITY CONTENT-TYPE)] {2}
S: [LITERAL DATA: 2 bytes]
S: )
S: 7 OK Fetch completed.

The "LITERAL DATA 5 bytes" is the "Bläh" text (3 single-byte chars + 1 
multibyte char = 5 octets"

I'm wondering if your IMAP server is sending the Bläh in a string 
instead of a literal.  Which would mean your IMAP server is broken.   
Need an IMAP log to verify.

---

To further debug this issue, we need details of the IMP -> IMAP/POP 
communication.

To enable debugging, see instructions contained in 
imp/config/backends.php (the 'debug' config parameter).

Debugging should not be enabled on a production server,   Attach/post 
only the portion of the log that directly deals with the problem 
reported (it may be simplest to clear the log file and then perform 
the event that causes the error).

Alright, so I figured out that the problem is broken mailers sending 
e-mails that don't conform to RFC2047.  If you access an IMAP folder 
for the MBX file shown below, you'll see a "Fatal error: Invalid 
argument" tooltip shown in the IMP UI.

=== Beginning of MBX file ====
From moore@cs.utk.edu Sun Jan  8 19:31:49 2012
Message-ID: <803734518.1326051106099.JavaMail.root@example.com>
From: =?US-ASCII?Q?Keith_Moore?= <moore@cs.utk.edu>
To: =?ISO-8859-1?Q?Keld_J=F8rn_Simonsen?= <keld@dkuug.dk>, "Bläh" 
<blah@example.com>
CC: =?ISO-8859-1?Q?Andr=E9?= Pirard <PIRARD@vm1.ulg.ac.be>
Subject: =?ISO-8859-1?B?SWYgeW91IGNhbiByZWFkIHRoaXMgeW8=?=
     =?ISO-8859-2?B?dSB1bmRlcnN0YW5kIHRoZSBleGFtcGxlLg==?=
MIME-Version: 1.0
Date: Sun, 8 Jan 2012 14:31:46 -0500
Content-Length: 40
Lines: 2

Body

=== End of MBX file ====

The problem is the umlaut-a (ä) in the "To:" line.  If the 'ä' is 
replaced with an 'a' then the message-index loads fine.

If the mailbox passes some number of messages, then the "Invalid 
Argument" tooltip isn't shown anymore.  At that point, the only way to 
identify a failure is the "Uncaught TypeError: Cannot set property 
'VP_rownum' of undefined " visible in the JS console.

This seems to be a recurrence of http://bugs.horde.org/ticket/9766

With a just-updated version of IMP 6.1.2, loading a specific mailbox 
folder causes the error message "Uncaught TypeError: Cannot set 
property 'VP_rownum' of undefined" to appear on the JS console.  The 
contents of that mailbox folder isn't displayed.  The folder has over 
4500 messages in it.

The complete stack trace:

Uncaught TypeError: Cannot set property 'VP_rownum' of undefined 
viewport.js?v=5e98e6c9c4aa9cc91ffb4a1defbcb044:1492
(anonymous function) viewport.js?v=5e98e6c9c4aa9cc91ffb4a1defbcb044:1492
_each prototype.js?v=2c0fc24b3d3038317dc51c05339856d0:1415
each prototype.js?v=2c0fc24b3d3038317dc51c05339856d0:846
Class.create.update viewport.js?v=5e98e6c9c4aa9cc91ffb4a1defbcb044:1491
Class.create._ajaxResponseDo 
viewport.js?v=5e98e6c9c4aa9cc91ffb4a1defbcb044:843
Class.create._ajaxResponse viewport.js?v=5e98e6c9c4aa9cc91ffb4a1defbcb044:819
Class.create.parseJSONResponse 
viewport.js?v=5e98e6c9c4aa9cc91ffb4a1defbcb044:807
DimpBase.tasksHandler dimpbase.js?v=5e98e6c9c4aa9cc91ffb4a1defbcb044:3707
(anonymous function) dimpbase.js?v=5e98e6c9c4aa9cc91ffb4a1defbcb044:4010
(anonymous function) prototype.js?v=2c0fc24b3d3038317dc51c05339856d0:417
(anonymous function) prototype.js?v=2c0fc24b3d3038317dc51c05339856d0:6753
fireEvent_DOM prototype.js?v=2c0fc24b3d3038317dc51c05339856d0:6613
fire prototype.js?v=2c0fc24b3d3038317dc51c05339856d0:6602
_methodized prototype.js?v=2c0fc24b3d3038317dc51c05339856d0:456
HordeCore.doActionComplete hordecore.js?v=2c0fc24b3d3038317dc51c05339856d0:231
(anonymous function) hordecore.js?v=2c0fc24b3d3038317dc51c05339856d0:85
Ajax.Request.Class.create.respondToReadyState 
prototype.js?v=2c0fc24b3d3038317dc51c05339856d0:1806
Ajax.Request.Class.create.onStateChange 
prototype.js?v=2c0fc24b3d3038317dc51c05339856d0:1749

And here's the last AJAX request/response

Request URL:https://mailserver/webmail/services/ajax.php/imp/viewPort
Request Method:POST
Status Code:200 OK

Request Headers
Accept:text/javascript, text/html, application/xml, text/xml, */*
Accept-Encoding:gzip,deflate,sdch
Accept-Language:en-US,en;q=0.8,de;q=0.6,de-CH;q=0.4
Connection:keep-alive
Content-Length:204
Content-type:application/x-www-form-urlencoded; charset=UTF-8
Cookie:userhash=USERHASHREDACTED; PHPSESSID=REDACTED; 
horde_secret_key=REDACTED; default_horde_view=auto; Horde=REDACTED
DNT:1
Host:mailserver
Origin:https://mailserver
Referer:https://mailserver/webmail/imp/dynamic.php?page=mailbox
User-Agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, 
like Gecko) Chrome/29.0.1547.15 Safari/537.36
X-Prototype-Version:1.7.1
X-Requested-With:XMLHttpRequest

Form Data
viewport:{"view":"SU5CT1guT2xkQXJjaGl2ZQ","initial":1,"after":144,"before":96,"slice":"1:241"}
view:SU5CT1guT2xkQXJjaGl2ZQ
token:REDACTED
Response Headersview source
Cache-Control:no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Connection:Keep-Alive
Content-Encoding:gzip
Content-Length:1294
Content-Type:application/json
Date:Tue, 09 Jul 2013 19:09:44 GMT
Expires:Thu, 19 Nov 1981 08:52:00 GMT
Keep-Alive:timeout=5, max=100
Pragma:no-cache
Server:Apache/2.2.22 (Ubuntu)
Set-Cookie:Horde=kuuttiqsjjvv11u7ma8d0mfgc0; path=/; 
domain=mailserver; HttpOnly
Vary:Accept-Encoding
X-Powered-By:PHP/5.3.10-1ubuntu3