compose/reply malformed header

Using the attached message, traditional mode, If I reply to

Tue, 25 Jan 2011 09:17:36 +0000

Using the attached message, traditional mode, If I reply to this message and go back to Inbox, I can't open the received message nor the sent message, I get the following errors : 2011-01-25T10:06:17+01:00 DEBUG: HORDE [imp] PHP ERROR: Undefined index: size [pid 31329 on line 756 of "/var/www/html/hordetest/imp/mailbox.php"] 2011-01-25T10:06:17+01:00 DEBUG: HORDE [imp] PHP ERROR: Undefined index: flags [pid 31329 on line 770 of "/var/www/html/hordetest/imp/mailbox.php"] 2011-01-25T10:06:17+01:00 DEBUG: HORDE [imp] PHP ERROR: Undefined index: flags [pid 31329 on line 774 of "/var/www/html/hordetest/imp/mailbox.php"] 2011-01-25T10:06:17+01:00 DEBUG: HORDE [imp] PHP ERROR: Undefined index: headers [pid 31329 on line 775 of "/var/www/html/hordetest/imp/mailbox.php"] 2011-01-25T10:06:17+01:00 DEBUG: HORDE [imp] PHP ERROR: in_array() expects parameter 2 to be array, null given [pid 31329 on line 47 of "/var/www/html/hordetest/imp/lib/Flag/Imap.php"] ... 2011-01-25T10:06:17+01:00 DEBUG: HORDE [imp] PHP ERROR: in_array() expects parameter 2 to be array, null given [pid 31329 on line 60 of "/var/www/html/hordetest/imp/lib/Flag/System/Unseen.php"] 2011-01-25T10:06:33+01:00 DEBUG: HORDE [imp] PHP ERROR: Undefined offset: 8 [pid 31329 on line 2919 of "/var/www/html/hordetest/libs/Horde/Imap/Client/Socket.php"] ... If I log out and login again using dynamic view, I get the following error when trying to access INBOX : ==> /var/log/httpd/error_log <== [Tue Jan 25 10:11:03 2011] [error] [client 192.168.1.73] PHP Fatal error: Call to a member function getValue() on a non-object in /var/www/html/hordetest/imp/lib/Views/ListMessages.php on line 469, referer: http://192.168.1.22/hordetest/imp/ 2011-01-25T10:11:03+01:00 DEBUG: HORDE [imp] PHP ERROR: Undefined offset: 8 [pid 31329 on line 2919 of "/var/www/html/hordetest/libs/Horde/Imap/Client/Socket.php"] 2011-01-25T10:11:03+01:00 DEBUG: HORDE [imp] PHP ERROR: Undefined offset: 8 [pid 31329 on line 2919 of "/var/www/html/hordetest/libs/Horde/Imap/Client/Socket.php"] ...

Clear your cache - looks like it is corrupt for some reason.

Tue, 25 Jan 2011 09:57:00 +0000

Clear your cache - looks like it is corrupt for some reason.

> Clear your cache - looks like it is corrupt for some reaso

Tue, 25 Jan 2011 12:53:46 +0000

> Clear your cache - looks like it is corrupt for some reason. I don't have cache enabled. $servers['imap']['cache'] => false $conf['group']['cache'] = false; $conf['share']['cache'] = false; $conf['cache']['driver'] = 'Null'; $conf['cachecss'] = false; $conf['cachejs'] = false; $conf['cachethemes'] = false; $conf['sessionhandler']['memcache'] = false; $conf['memcache']['enabled'] = false; The header of the sent message doesn't look right : To: "=?utf-8?b?IkZyYW7Dp29pcw==?= Xavier. XXXXXX\""

> The header of the sent message doesn't look right : > To:

Tue, 25 Jan 2011 19:21:45 +0000

> The header of the sent message doesn't look right : > To: "=?utf-8?b?IkZyYW7Dp29pcw==?= Xavier. XXXXXX\"" There's nothing technically wrong with this header. There was a bug in the encoding algorithm (since fixed) that caused the name to be double wrapped in quotes, but that's a quality of display issue, not a fatal error.

>> The header of the sent message doesn't look right : >> T

Wed, 26 Jan 2011 10:44:10 +0000

>> The header of the sent message doesn't look right : >> To: "=?utf-8?b?IkZyYW7Dp29pcw==?= Xavier. XXXXXX\"" > > There's nothing technically wrong with this header. There was a bug > in the encoding algorithm (since fixed) that caused the name to be > double wrapped in quotes, but that's a quality of display issue, not > a fatal error. Ok, I updated from git this morning and indeed, the encoding part is fixed. But (there's always a but :-) ), If there is nothing technically wrong with this header, then there is a problem with DIMP. I've attached the received message (before the encoding algorithm fix). when I click on the folder containing this email, I get : - DIMP : The server was unable to generate the message list. - httpd/error_log [Wed Jan 26 11:35:55 2011] [error] [client 192.168.1.73] PHP Fatal error: Call to a member function getValue() on a non-object in /var/www/html/hordetest/imp/lib/Views/ListMessages.php on line 469, referer: http://192.168.1.22/hordetest/imp/ - horde.log see attached file

No problem here. My guess is that your IMAP server is choki

Wed, 26 Jan 2011 17:57:22 +0000

No problem here. My guess is that your IMAP server is choking on your message. You need to provide the IMAP log when viewing that mailbox. --- To further debug this issue, we need details of the IMP -> IMAP/POP communication. To enable debugging, see instructions contained in imp/config/servers.php (the 'debug' config parameter). Debugging should not be enabled on a production server, Attach/post only the portion of the log that directly deals with the problem reported (it may be simplest to clear the log file and then perform the event that causes the error).

> No problem here. My guess is that your IMAP server is cho

Thu, 27 Jan 2011 08:02:08 +0000

> No problem here. My guess is that your IMAP server is choking on > your message. You need to provide the IMAP log when viewing that > mailbox. attached is the IMAP log. There is only one email in the folder.

> attached is the IMAP log. > There is only one email in th

Thu, 27 Jan 2011 10:01:04 +0000

> attached is the IMAP log. > There is only one email in the folder. Your IMAP server is broken. This response is invalid: (1296114958,4055) S: * 1 FETCH (UID 51 [snip] (("=?utf-8?b?IkZyYW7Dp29pcw==?= Xavier. XXXXXX\\" NIL "rsalmon" "mbpgroup.com")) [snip]) The personal name part resolves to: "François Xavier. XXXXXX\ The IMAP server is chopping off the trailing quote. The FETCH response MUST be a literal string (instead of a quoted string) due to the presence of the double quote. e.g.: * 1 FETCH (UID 51 [snip] (({45} =?utf-8?b?IkZyYW7Dp29pcw==?= Xavier. XXXXXX\" NIL "rsalmon" "mbpgroup.com")) [snip])

> Your IMAP server is broken. In Courier version 4.7.0 (a

Thu, 27 Jan 2011 10:24:04 +0000

> Your IMAP server is broken. In Courier version 4.7.0 (according to changelog, looks like nothing substantial has changed to the present version), I get this: (("=?utf-8?b?IkZyYW7Dp29pcw==?= Xavier. XXXXXX" NIL "rsalmon" "mbpgroup.com")) So it is silently stripping the trailing double quote, which means it is still broken, although in a different way. However, IMP seems to handle this just fine. The proper way of quoting this string would be: (("=?utf-8?b?IkZyYW7Dp29pcw==?= Xavier. XXXXXX\\\"" NIL "rsalmon" "mbpgroup.com")) Or better yet, use a real server (like dovecot) that does literal quoting the second it sees either a backslash or double quote.

Not sure if we can catch broken IMAP servers like this witho

Thu, 27 Jan 2011 10:29:16 +0000

Not sure if we can catch broken IMAP servers like this without breaking expected behavior. Since there really is not an easy way for me to debug this, it would be great if you could track down where things are breaking. Might want to start in IMP_Mailbox_List#getMailboxArray() and work from there (namely, the results from the imp_imap->fetch() call).

> Or better yet, use a real server (like dovecot) that does

Thu, 27 Jan 2011 11:06:13 +0000

> Or better yet, use a real server (like dovecot) that does literal > quoting the second it sees either a backslash or double quote. I guess there are pros and cons for using courier-imap or dovecot :-)

> Not sure if we can catch broken IMAP servers like this wit

Thu, 27 Jan 2011 11:10:01 +0000

> Not sure if we can catch broken IMAP servers like this without > breaking expected behavior. Since there really is not an easy way > for me to debug this, it would be great if you could track down where > things are breaking. Might want to start in > IMP_Mailbox_List#getMailboxArray() and work from there (namely, the > results from the imp_imap->fetch() call). Since we are using an old version (4.3.0) I'll try first to update courier-imap, but not easy to do on a production server since updating courier-imap means updating other packages. And if this is an issue with courier-imap, I should try to get the courier-imap guys to fix this. Nevertheless, I'll try to debug as you described above.

>> Your IMAP server is broken. > > In Courier version 4.7.

Thu, 27 Jan 2011 21:25:12 +0000

>> Your IMAP server is broken. > > In Courier version 4.7.0 (according to changelog, looks like nothing > substantial has changed to the present version), I get this: I was using 4.3.0. I've justed updated to 4.8.1. Works like a charm :-) I'm hoping that everything will be Ok tomorrow morning... The following courier-imap fix probably did it : 2008-06-13 Mr. Sam * rfc822_getaddr.c: Backslashed special characters in address names weren't being dequoted correctly by rfc822_getname() and rfc822_getname_orlist(). We probably ran into this problem before but I'm guessing all end user software always handled this issue silently. This includes older IMP. Amazing! I won't be able to debug this anymore (Although I tried this afternoon, I got nowhere).

There still is an small issue with this message. using DI

Thu, 27 Jan 2011 21:27:52 +0000

There still is an *small* issue with this message. using DIMP, if I reply to all (or just sender), the "To" filed is never automatically completed.

> We probably ran into this problem before but I'm guessing

Fri, 28 Jan 2011 17:15:13 +0000

> We probably ran into this problem before but I'm guessing all end > user software always handled this issue silently. This includes older > IMP. Amazing! This probably means that we rebuild the message data differently than other software. I would still like to find where this happens, if possible, since Horde/IMP should work (as much as possible) with garbage input.

> There still is an small issue with this message. > usin

Fri, 28 Jan 2011 20:14:26 +0000

> There still is an *small* issue with this message. > using DIMP, if I reply to all (or just sender), the "To" filed is > never automatically completed. Works for me. Might not be working for you since you would be replying to yourself.

>> There still is an small issue with this message. >> us

Sat, 29 Jan 2011 15:43:55 +0000

>> There still is an *small* issue with this message. >> using DIMP, if I reply to all (or just sender), the "To" filed is >> never automatically completed. > > Works for me. Might not be working for you since you would be > replying to yourself. Yes, this is the case. I got misled since the user name is displayed and not the email address (which was mine).

> This probably means that we rebuild the message data diffe

Mon, 31 Jan 2011 09:13:04 +0000

> This probably means that we rebuild the message data differently than > other software. I would still like to find where this happens, if > possible, since Horde/IMP should work (as much as possible) with > garbage input. I've managed to install courier-imap-4.3.0 on an old server, and I can reproduce this issue with the messages attached to this ticket. Can you try with courier-imap-4.3.0 and see if you can reproduce the errors ? If not, then I'll try to debug further.

> Can you try with courier-imap-4.3.0 and see if you can rep

Tue, 01 Feb 2011 21:22:21 +0000

> Can you try with courier-imap-4.3.0 and see if you can reproduce the > errors ? If not, then I'll try to debug further. I can reproduce. The problem is that the parsing of the envelope entry is broken. Instead of finding 10 fields, it only parses 5 (because the quote is not closed). This results in various data entries not existing even though the API guarantees they should be available. There will never be a way to properly parse this entry - all fields after the bad field will be garbage. But the code should be able to handle this more gracefully - by defaulting to blank entries for example. This is going to take a bit of work; my current thinking is we should be returning objects from the IMAP library fetch() call and deal with the issue of bad data when accessing the object properties.

Changes have been made in Git for this ticket: Bug #9538: E

Fri, 18 Feb 2011 22:39:17 +0000

Changes have been made in Git for this ticket: Bug #9538: Envelope data now returned in an object. create mode 100644 framework/Imap_Client/lib/Horde/Imap/Client/Data/Envelope.php http://git.horde.org/horde-git/-/commit/5350fa93016e7698bfd09485ff649e2ef88785ac

Changes have been made in Git for this ticket: Bug #9538: A

Fri, 18 Feb 2011 22:39:21 +0000

Changes have been made in Git for this ticket: Bug #9538: Add Horde_Imap_Client_Fetch_Query Provides an OO-based way of creating a query, rather than relying on complex array configuration rules. create mode 100644 framework/Imap_Client/lib/Horde/Imap/Client/Fetch/Query.php http://git.horde.org/horde-git/-/commit/59d11b153a16de754edbcffb426c52324e4c17f3

Changes have been made in Git for this ticket: Bug #9538: A

Fri, 18 Feb 2011 22:39:24 +0000

Changes have been made in Git for this ticket: Bug #9538: Add Horde_Imap_Client_Ids and Horde_Imap_Client_Data_Fetch Now, bad data returned from an IMAP server will never cause failure since every fetch return entry has automatic default values returned. Additionally, using objects/IDs like this vastly improves the API by removing complex array configurations and reducing the number of parameters. Side benefit: ALL literal data returned from the IMAP server is now stored internally as streams - it is auto-converted to a string as needed. create mode 100644 framework/Imap_Client/lib/Horde/Imap/Client/Data/Fetch.php create mode 100644 framework/Imap_Client/lib/Horde/Imap/Client/Ids.php http://git.horde.org/horde-git/-/commit/0224531ec7ab9062c8935f076bb82becb4d5996f

Fixed. Required a rewrite of the way we handled fetch data

Fri, 18 Feb 2011 22:43:42 +0000

Fixed. Required a rewrite of the way we handled fetch data in the IMAP client. Considering this is about 75% of the activity of an IMAP client, this was not a small undertaking. Not super excited about rewriting a good portion of the IMAP client a couple of months before we release H4, but this issue was a showstopper. Broken IMAP servers, which exist and over which we have no control, could render a IMP user completely unusable. That was unacceptable. Now, the user may get garbage for certain elements of the message if the IMAP server is broke, but the message is viewable/deletable and it doesn't break the underlying mailbox.

> Fixed. Required a rewrite of the way we handled fetch dat

Mon, 21 Feb 2011 08:48:36 +0000

> Fixed. Required a rewrite of the way we handled fetch data in the > IMAP client. Considering this is about 75% of the activity of an > IMAP client, this was not a small undertaking. I confirmed that IMP is working with this buggy message using courier-imap 4.3.0 and 4.8.1.