I confirmed that IMP is working with this buggy message using 
courier-imap 4.3.0 and 4.8.1.

Fixed.  Required a rewrite of the way we handled fetch data in the 
IMAP client.  Considering this is about 75% of the activity of an IMAP 
client, this was not a small undertaking.

Not super excited about rewriting a good portion of the IMAP client a 
couple of months before we release H4, but this issue was a 
showstopper.  Broken IMAP servers, which exist and over which we have 
no control, could render a IMP user completely unusable.  That was 
unacceptable.  Now, the user may get garbage for certain elements of 
the message if the IMAP server is broke, but the message is 
viewable/deletable and it doesn't break the underlying mailbox.

Changes have been made in Git for this ticket:

Bug #9538: Add Horde_Imap_Client_Ids and Horde_Imap_Client_Data_Fetch
Now, bad data returned from an IMAP server will never cause failure
since every fetch return entry has automatic default values returned.

Additionally, using objects/IDs like this vastly improves the API by
removing complex array configurations and reducing the number of
parameters.

Side benefit: ALL literal data returned from the IMAP server is now
stored internally as streams - it is auto-converted to a string as
needed.

  create mode 100644 framework/Imap_Client/lib/Horde/Imap/Client/Data/Fetch.php
  create mode 100644 framework/Imap_Client/lib/Horde/Imap/Client/Ids.php
http://git.horde.org/horde-git/-/commit/0224531ec7ab9062c8935f076bb82becb4d5996f

Changes have been made in Git for this ticket:

Bug #9538: Add Horde_Imap_Client_Fetch_Query
Provides an OO-based way of creating a query, rather than relying on
complex array configuration rules.

  create mode 100644 
framework/Imap_Client/lib/Horde/Imap/Client/Fetch/Query.php
http://git.horde.org/horde-git/-/commit/59d11b153a16de754edbcffb426c52324e4c17f3

Changes have been made in Git for this ticket:

Bug #9538: Envelope data now returned in an object.

  create mode 100644 
framework/Imap_Client/lib/Horde/Imap/Client/Data/Envelope.php
http://git.horde.org/horde-git/-/commit/5350fa93016e7698bfd09485ff649e2ef88785ac

I can reproduce.  The problem is that the parsing of the envelope 
entry is broken.  Instead of finding 10 fields, it only parses 5 
(because the quote is not closed).  This results in various data 
entries not existing even though the API guarantees they should be 
available.

There will never be a way to properly parse this entry - all fields 
after the bad field will be garbage.  But the code should be able to 
handle this more gracefully - by defaulting to blank entries for 
example.  This is going to take a bit of work; my current thinking is 
we should be returning objects from the IMAP library fetch() call and 
deal with the issue of bad data when accessing the object properties.

I've managed to install courier-imap-4.3.0 on an old server, and I can 
reproduce this issue with the messages attached to this ticket.

Can you try with courier-imap-4.3.0 and see if you can reproduce the 
errors ? If not, then I'll try to debug further.

Yes, this is the case. I got misled since the user name is displayed 
and not the email address (which was mine).

Works for me.  Might not be working for you since you would be 
replying to yourself.

This probably means that we rebuild the message data differently than 
other software.  I would still like to find where this happens, if 
possible, since Horde/IMP should work (as much as possible) with 
garbage input.

There still is an *small* issue with this message.
using DIMP, if I reply to all (or just sender), the "To" filed is 
never automatically completed.

I was using 4.3.0. I've justed updated to  4.8.1. Works like a charm 
:-) I'm hoping that everything will be Ok tomorrow morning...

The following courier-imap fix probably did it :

  2008-06-13  Mr. Sam  <mrsam@courier-mta.com>
         * rfc822_getaddr.c: Backslashed special characters in address names
         weren't being dequoted correctly by rfc822_getname() and
         rfc822_getname_orlist().

We probably ran into this problem before but I'm guessing all end user 
software always handled this issue silently. This includes older IMP. 
Amazing!

I won't be able to debug this anymore (Although I tried this 
afternoon, I got nowhere).

Since we are using an old version (4.3.0) I'll try first to update 
courier-imap, but not easy to do on a production server since updating 
courier-imap means updating other packages.
And if this is an issue with courier-imap, I should try to get the 
courier-imap guys to fix this.

Nevertheless, I'll try to debug as you described above.

I guess there are pros and cons for using courier-imap or dovecot :-)

Not sure if we can catch broken IMAP servers like this without 
breaking expected behavior.  Since there really is not an easy way for 
me to debug this, it would be great if you could track down where 
things are breaking.  Might want to start in 
IMP_Mailbox_List#getMailboxArray() and work from there (namely, the 
results from the imp_imap->fetch() call).

In Courier version 4.7.0 (according to changelog, looks like nothing 
substantial has changed to the present version), I get this:

(("=?utf-8?b?IkZyYW7Dp29pcw==?= Xavier. XXXXXX" NIL "rsalmon" "mbpgroup.com"))

So it is silently stripping the trailing double quote, which means it 
is still broken, although in a different way.  However, IMP seems to 
handle this just fine.

The proper way of quoting this string would be:

(("=?utf-8?b?IkZyYW7Dp29pcw==?= Xavier. XXXXXX\\\"" NIL "rsalmon" 
"mbpgroup.com"))

Or better yet, use a real server (like dovecot) that does literal 
quoting the second it sees either a backslash or double quote.

Your IMAP server is broken.

This response is invalid:

(1296114958,4055) S: * 1 FETCH (UID 51 [snip] 
(("=?utf-8?b?IkZyYW7Dp29pcw==?= Xavier. XXXXXX\\" NIL "rsalmon" 
"mbpgroup.com")) [snip])

The personal name part resolves to:
"François Xavier. XXXXXX\

The IMAP server is chopping off the trailing quote.  The FETCH 
response MUST be a literal string (instead of a quoted string) due to 
the presence of the double quote.  e.g.:

* 1 FETCH (UID 51 [snip] (({45}
=?utf-8?b?IkZyYW7Dp29pcw==?= Xavier. XXXXXX\"
  NIL "rsalmon" "mbpgroup.com")) [snip])

attached is the IMAP log.
There is only one email in the folder.

No problem here.  My guess is that your IMAP server is choking on your 
message.  You need to provide the IMAP log when viewing that mailbox.

---

To further debug this issue, we need details of the IMP -> IMAP/POP 
communication.

To enable debugging, see instructions contained in 
imp/config/servers.php (the 'debug' config parameter).

Debugging should not be enabled on a production server,   Attach/post 
only the portion of the log that directly deals with the problem 
reported (it may be simplest to clear the log file and then perform 
the event that causes the error).

Ok, I updated from git this morning and indeed, the encoding part is 
fixed. But (there's always a but :-) ), If there is nothing 
technically wrong with this header, then there is a problem with DIMP.

I've attached the received message (before the encoding algorithm fix).

when I click on the folder containing this email, I get :
- DIMP : The server was unable to generate the message list.

- httpd/error_log
[Wed Jan 26 11:35:55 2011] [error] [client 192.168.1.73] PHP Fatal 
error:  Call to a member function getValue() on a non-object in 
/var/www/html/hordetest/imp/lib/Views/ListMessages.php on line 469, 
referer: http://192.168.1.22/hordetest/imp/

- horde.log
see attached file

There's nothing technically wrong with this header.  There was a bug 
in the encoding algorithm (since fixed) that caused the name to be 
double wrapped in quotes, but that's a quality of display issue, not a 
fatal error.

I don't have cache enabled.
$servers['imap']['cache'] => false

$conf['group']['cache'] = false;
$conf['share']['cache'] = false;
$conf['cache']['driver'] = 'Null';
$conf['cachecss'] = false;
$conf['cachejs'] = false;
$conf['cachethemes'] = false;
$conf['sessionhandler']['memcache'] = false;
$conf['memcache']['enabled'] = false;

The header of the sent message doesn't look right :
To: "=?utf-8?b?IkZyYW7Dp29pcw==?= Xavier. XXXXXX\"" <rsalmon@mbpgroup.com>

Clear your cache - looks like it is corrupt for some reason.

Using the attached message, traditional mode,
If I reply to this message and go back to Inbox, I can't open the 
received message nor the sent message, I get the following errors :

2011-01-25T10:06:17+01:00 DEBUG: HORDE [imp] PHP ERROR: Undefined 
index: size [pid 31329 on line 756 of 
"/var/www/html/hordetest/imp/mailbox.php"]
2011-01-25T10:06:17+01:00 DEBUG: HORDE [imp] PHP ERROR: Undefined 
index: flags [pid 31329 on line 770 of 
"/var/www/html/hordetest/imp/mailbox.php"]
2011-01-25T10:06:17+01:00 DEBUG: HORDE [imp] PHP ERROR: Undefined 
index: flags [pid 31329 on line 774 of 
"/var/www/html/hordetest/imp/mailbox.php"]
2011-01-25T10:06:17+01:00 DEBUG: HORDE [imp] PHP ERROR: Undefined 
index: headers [pid 31329 on line 775 of 
"/var/www/html/hordetest/imp/mailbox.php"]
2011-01-25T10:06:17+01:00 DEBUG: HORDE [imp] PHP ERROR: in_array() 
expects parameter 2 to be array, null given [pid 31329 on line 47 of 
"/var/www/html/hordetest/imp/lib/Flag/Imap.php"]
...
2011-01-25T10:06:17+01:00 DEBUG: HORDE [imp] PHP ERROR: in_array() 
expects parameter 2 to be array, null given [pid 31329 on line 60 of 
"/var/www/html/hordetest/imp/lib/Flag/System/Unseen.php"]
2011-01-25T10:06:33+01:00 DEBUG: HORDE [imp] PHP ERROR: Undefined 
offset: 8 [pid 31329 on line 2919 of 
"/var/www/html/hordetest/libs/Horde/Imap/Client/Socket.php"]
...



If I log out and login again using dynamic view, I get the following 
error when trying to access INBOX :

==> /var/log/httpd/error_log <==
[Tue Jan 25 10:11:03 2011] [error] [client 192.168.1.73] PHP Fatal 
error:  Call to a member function getValue() on a non-object in 
/var/www/html/hordetest/imp/lib/Views/ListMessages.php on line 469, 
referer: http://192.168.1.22/hordetest/imp/
2011-01-25T10:11:03+01:00 DEBUG: HORDE [imp] PHP ERROR: Undefined 
offset: 8 [pid 31329 on line 2919 of 
"/var/www/html/hordetest/libs/Horde/Imap/Client/Socket.php"]
2011-01-25T10:11:03+01:00 DEBUG: HORDE [imp] PHP ERROR: Undefined 
offset: 8 [pid 31329 on line 2919 of 
"/var/www/html/hordetest/libs/Horde/Imap/Client/Socket.php"]
...