Changes have been made in Git (master):

commit a210ca8bb1cb4689a8752063cf0a7bdcc3d4a3ee
Author: Michael M Slusarz <slusarz@horde.org>
Date:   Thu Nov 17 14:07:12 2011 -0700

     Bug #10093: Updates to reflect that this will be fixed in 5.1

  imp/docs/UPGRADING                        |   38 +++++++++++++---------------
  imp/lib/LoginTasks/SystemTask/Upgrade.php |    6 ++--
  2 files changed, 21 insertions(+), 23 deletions(-)

http://git.horde.org/horde-git/-/commit/a210ca8bb1cb4689a8752063cf0a7bdcc3d4a3ee

Changes have been made in Git (master):

commit 28b604652b636aaa464896e7ac5ab63b9716dac2
Author: Michael M Slusarz <slusarz@horde.org>
Date:   Fri Oct 28 18:27:45 2011 -0600

     Bug #10093: Convert IMP to use new Horde_Imap_Client 1.2 features

     IMP now entirely handles IMAP mailbox names internally in UTF-8.

     This does, however, mean several preferences and configuration
     parameters may need to be changed.  Current preference values will be
     automatically updated.  This is a necessary evil, or else there will be
     NO way to accurately determine the charset in future versions.  (Most
     users should be using default values, so this will not affect them.
     Addtionally, from feeback, it seems likely that very few users are
     currently using the fixed_folders configuration option).

     Conflicts:

             imp/docs/CHANGES
             imp/package.xml

  imp/config/backends.php                           |    6 +-
  imp/config/conf.xml                               |    5 +-
  imp/config/hooks.php.dist                         |    8 +-
  imp/config/prefs.php                              |   20 +----
  imp/docs/CHANGES                                  |    1 +
  imp/docs/UPGRADING                                |   15 +++
  imp/folders.php                                   |   12 +--
  imp/lib/Ajax/Application.php                      |    7 +-
  imp/lib/Api.php                                   |   20 ++--
  imp/lib/Compose.php                               |    2 +-
  imp/lib/Flag/User.php                             |    2 +-
  imp/lib/Flags.php                                 |    2 +-
  imp/lib/Imap.php                                  |   98 +++++++++++++++++++-
  imp/lib/Imap/Tree.php                             |    4 +-
  imp/lib/LoginTasks/SystemTask/Upgrade.php         |   38 ++++++++-
  imp/lib/LoginTasks/Task/DeleteSentmailMonthly.php |    4 +-
  imp/lib/LoginTasks/Task/RenameSentmailMonthly.php |    2 +-
  imp/lib/Mailbox.php                               |   95 
++++++++++++---------
  imp/lib/Message.php                               |    2 +-
  imp/lib/Prefs/Ui.php                              |    3 +-
  imp/lib/Ui/Folder.php                             |   21 ++---
  imp/package.xml                                   |    1 +
  imp/templates/imp/folders/import.html             |    2 +-
  23 files changed, 253 insertions(+), 117 deletions(-)

http://git.horde.org/horde-git/-/commit/28b604652b636aaa464896e7ac5ab63b9716dac2

Changing target to 5.1.  I am more than comfortable with the stability 
of the release.  However, it makes several (6-7) changes in 
configuration files, and this is a significant enough change to 
justify releasing in a minor, rather than point, release.  if just to 
make the UPGRADING file easier to parse.

Changes have been made in Git for this ticket:

Revert "Bug #10093: Convert IMP to use new Horde_Imap_Client 1.2 features"
This reverts commit fd89d722f80b5b9715033d21e5ecbb06605b7ab0.

Conflicts:

        imp/docs/CHANGES
        imp/lib/LoginTasks/SystemTask/Upgrade.php
        imp/package.xml

  23 files changed, 117 insertions(+), 254 deletions(-)
http://git.horde.org/horde-git/-/commit/2efc8b4f267c211bfb30c324ee0c82459304d9a1

Fixed for 5.0.15.

Changes have been made in Git for this ticket:

Bug #10093: Convert IMP to use new Horde_Imap_Client 1.2 features
IMP now entirely handles IMAP mailbox names internally in UTF-8.

This does, however, mean several preferences and configuration
parameters may need to be changed.  Current preference values will be
automatically updated.  This is a necessary evil, or else there will be
NO way to accurately determine the charset in future versions.  (Most
users should be using default values, so this will not affect them.
Addtionally, from feeback, it seems likely that very few users are
currently using the fixed_folders configuration option).

  23 files changed, 253 insertions(+), 117 deletions(-)
http://git.horde.org/horde-git/-/commit/fd89d722f80b5b9715033d21e5ecbb06605b7ab0

Changes have been made in Git for this ticket:

Bug #10093: Use Horde_Imap_Client_Mailbox for mailbox parameters/return values

  10 files changed, 663 insertions(+), 424 deletions(-)
http://git.horde.org/horde-git/-/commit/20c50a0d1bb0cb15d9fd185f240f02d1b0fd4b7a

Changes have been made in Git for this ticket:

Bug #10093: Added Horde_Imap_Client_Mailbox
Provides way to accurately switch between UTF7-IMAP and UTF-8 mailbox
representations.

  5 files changed, 270 insertions(+), 6 deletions(-)
http://git.horde.org/horde-git/-/commit/69d52da8792bd68a536a3fb969201e8a7dc921d0

Thanks for this hint. It was kind of hard for me to figure out, since 
the problem started appearing immediately after upgrading to Horde 
4.0, completely unrelated to any Courier IMAP upgrades.

The good news is that upgrading from Courier IMAP 4.9.1 to 4.9.3 
solved the problem:

4.9.3

2011-05-22  Sam Varshavchik  <mrsam@courier-mta.com>

        * msgenvelope.c (msgappends): Fix a fatal error upon encountering
        8-bit header content. Heuristically try to interpret it as UTF-8, and
        just ignore invalid UTF-8 sequences.

4.9.2

2011-05-17  Sam Varshavchik  <mrsam@courier-mta.com>

        * rfc2045/rfc2045cdecode.c: Tolerate lowercase hexadecimal characters
        in quoted-printable-encoded content.

2011-05-06  Thomas Jacob <jacob@internet24.de>

        * unicode/unicode.c: Compilation fixes.

[...]
As expected, IMP can access this mailbox just fine.
Your IMAP server is broken.  See (RFC 3501 [7.1.5]) (Untagged BYE 
response can occur in 1 of 4 instances: the only instance that fits 
here is "2) as a panic shutdown announcement.")  That is a perfectly 
valid FETCH statement, so the problem is not with Horde/IMP.

I finally have some IMAP debug information (for folder 
Maildir/.Vigtigt.Ordrebekr\&AOY-ftelser/ (Vigtigt/Ordrebekræftelser), 
which has the same problem):
(1312367349,9868) S: * OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN 
NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE ACL 
ACL2=UNION STARTTLS XMAGICTRASH] Courier-IMAP ready. Copyright 
1998-2011 Double Precision, Inc.  See COPYING for distribution 
information.
(1312367349,9874) C: 1 STARTTLS
(1312367349,9878) S: 1 OK Begin SSL/TLS negotiation now.
(1312367350,0878) C: [LOGIN Command - username: xxx]
(1312367350,3669) S: 2 OK LOGIN Ok.
(1312367350,3677) C: 3 SELECT INBOX.Vigtigt.Ordrebekr&AOY-ftelser
(1312367350,3882) S: * FLAGS (NonJunk $Forwarded \Draft \Answered 
\Flagged \Deleted \Seen \Recent)
(1312367350,3886) S: * OK [PERMANENTFLAGS (NonJunk $Forwarded \* 
\Draft \Answered \Flagged \Deleted \Seen)] Limited
(1312367350,3889) S: * 428 EXISTS
(1312367350,3891) S: * 0 RECENT
(1312367350,3893) S: * OK [UIDVALIDITY 1068675967] Ok
(1312367350,3894) S: * OK [MYRIGHTS "acdilrsw"] ACL
(1312367350,3895) S: 3 OK [READ-WRITE] Ok
(1312367350,5602) C: 4 UID SEARCH 428
(1312367350,6022) S: * SEARCH 437
(1312367350,6028) S: 4 OK SEARCH done.
(1312367350,6061) C: 5 UID SORT (ARRIVAL) US-ASCII ALL
(1312367350,6875) S: * SORT 3 43 42 41 40 39 38 37 36 35 34 4 2 64 33 
32 63 62 61 60 59 58 31 30 29 28 145 57 56 55 54 53 52 51 27 26 25 24 
23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 50  [etc., red.]
(1312367350,6933) S: 5 OK SORT done.
(1312367350,6957) C: 6 UID SEARCH 428
(1312367350,6967) S: * SEARCH 437
(1312367350,6970) S: 6 OK SEARCH done.
(1312367350,6981) C: 7 UID SEARCH CHARSET UTF-8 UNSEEN
(1312367350,7396) S: * SEARCH
(1312367350,7400) S: 7 OK SEARCH done.
(1312367350,7435) C: 8 UID SEARCH 428
(1312367350,7846) S: * SEARCH 437
(1312367350,7851) S: 8 OK SEARCH done.
(1312367350,7984) C: 9 UID FETCH 
3,43,42,41,40,39,38,37,36,35,34,4,2,64,33,32,63,62,61,60,59,58,31,30,29 
(ENVELOPE FLAGS RFC822.SIZE BODY.PEEK[HEADER.FIELDS (IMPORTANCE 
LIST-POST X-PRIORITY)])
(1312367350,8455) S: * BYE [ALERT] Fatal error: Invalid argument

From the Horde log:
2011-08-03T11:15:24+02:00 DEBUG: HORDE [imp] Unexpectedly disconnected 
from the mail server. [pid 29118 on line 27 of 
"/usr/local/lib/php/Horde/Core/Notification/Handler/Decorator/Hordelog.php"]
2011-08-03T11:15:24+02:00 ERR: HORDE [imp] IMAP Server closed the 
connection: BYE [ALERT] Fatal error: Invalid argument [pid 29118 on 
line 340 of "/var/www/secure/horde4/imp/lib/Imap.php"]
2011-08-03T11:15:24+02:00 DEBUG: 1. 
IMP_Mailbox_List->getMailboxArray() 
/var/www/secure/horde4/imp/mailbox.php:259
2. IMP_Imap->fetch() /var/www/secure/horde4/imp/lib/Mailbox/List.php:170
3. IMP_Imap->__call() /var/www/secure/horde4/imp/lib/Mailbox/List.php:170

Works fine here.  From the IMAP debug log:

(1310495155.4528) C: 4 SELECT Testing.Nyt&AOU-r
(1310495155.4914) S: * OK [CLOSED] Previous mailbox closed.
(1310495155.4915) S: * FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
(1310495155.4916) S: * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted 
\Seen \Draft \*)] Flags permitted.
(1310495155.4917) S: * 1 EXISTS
(1310495155.4918) S: * 0 RECENT
(1310495155.4919) S: * OK [UIDVALIDITY 1255685480] UIDs valid
(1310495155.4919) S: * OK [UIDNEXT 2] Predicted next UID
(1310495155.4920) S: * OK [HIGHESTMODSEQ 2] Highest
(1310495155.4920) S: 4 OK [READ-WRITE] Select completed.

You will need to provide the IMAP debug log (see below for 
instructions) for when you try to access that mailbox.
This is a separate issue.  Open another ticket and provide the IMAP debug log.

----

To further debug this issue, we need details of the IMP -> IMAP/POP 
communication.

To enable debugging, see instructions contained in 
imp/config/backends.php (the 'debug' config parameter).

Debugging should not be enabled on a production server,   Attach/post 
only the portion of the log that directly deals with the problem 
reported (it may be simplest to clear the log file and then perform 
the event that causes the error).

I'm running pear.horde.org/Horde_Imap_Client-1.0.9 - installed with 
Horde 4.0.7 and IMP 5.0.8 two days ago.
For example, I'm having problems with folder name "Nytår" 
("Maildir/.Nyt\&AOU-r/").

However, another strange problem has appeared lately. When I have a 
mail that IMP doesn't like, it will not only refuse to show the mail. 
It will also give the "Unexpectedly disconnected from the mail 
server." error when trying to access the page on which the e-mail is 
located. I can go to previous page and next page, but the page itself 
can't be viewed. This might be related to ticket 10169, which was 
rejected (and unresolved).

I use UTF-8 everywhere on my system, which is a pretty clean CentOS 
5.5 installation. PHP (5.3.6) is compiled with:

--with-iconv
--enable-mbstring=all
--with-gettext

Other than that I use a custom built libxml2 (version 2.7.8) without 
any configure options set, i.e. quite straight-forward.

So I'm a bit puzzled, but don't really know where to look.

No - this doesn't sound right at all.  There should be extremely 
limited cases where this won't work - 99.9% of non ASCII characters 
should display correctly (I have a bunch of these mailboxes that 
display just fine).

Are you sure you have updated Horde_Imap_Client?  Alternatively, what 
is an example of a mailbox name (the actual name on the IMAP server, 
not the translated name) that you can not view?

I'm sorry, but ticket 10102 was closed as a doublet to this ticket. 
This (still) means that I can't access many of my folders at all (if 
they contain a Danish character like æ, ø or å) - I simply get: 
"Unexpectedly disconnected from the mail server." So I don't agree 
that it's only "broken for very fringe use cases" - at least if it's 
the same problem as in ticket 10102. It's actually quite broken. IMP 
5.0.7, Horde 4.0.6.

No, it's not that vague ("major" or "substantial" changes). The 
definition is clear. A minor version bump means that functionality has 
been added or an API has been *extended*. *Changing* the API requires 
a major version bump.

I could see us allowing to require a certain version of a dependency 
though. We already allow this for packages that haven't been released 
yet. But the important question is:

So this would clearly be a BC break. And it won't only affect IMP, but 
also any 3rd party developers that use this library and depend on 
auto-conversion.

We don't support them for potentially years, unless there isn't 
anything in the APIs that require BC breaking changes. If a BC break 
is necessary, we have to decide whether we want to have this into the 
next release (which would be a new major version then). Releases are 
done every half a year.

There is no easy solution for a buggy API design. Like Chuck said, 
this can always, and it's something you have to live with if you want 
to provide a stable and reliable release policy. And in this specific 
case, it can really only be fixed by changing the API. But then again, 
it's only still broken for very fringe use cases, and I doubt we will 
see a single bug report until the next major release with the improved 
auto-detection in place.

This was my understanding/belief also.  A minor version bump is 
intended to indicate that the basic framework is the same (i.e. no 
major data/configuration changes), but that there may be some 
substantial changes to the underlying system to fix limitations in the 
previous version.
Right now, Horde_Imap_Client accepts either UTF-8 or UTF7-IMAP strings 
for all mailbox parameters.  This needs to be changed to 
Horde_Imap_Client accepting UTF-8 strings only (since it can not be 
100% reliably auto-detected).

I am positive that IMP passes both types of strings to H_I_C methods.   
So this needs to be changed to UTF-8 only.  But this change in IMP is 
worthless if there is not a corresponding change in H_I_C - since we 
need to be calling a version of H_I_C that does not do any 
auto-conversion.

There are other things that need to be fixed in H_I_C (allowing 
multiple mailboxes to specified to search(); returning the list of 
UIDs that were altered in store() instead of a boolean) that should be 
done sooner rather than later.
That is probably true - this is a bit broad of a statement on my part. 
  But locking ourselves into an API, without the benefit of 
broad-based, thorough testing, is a terrible thing.  This was the #1 
failure of Horde 3 - we should not be supporting broken APIs for 
(potentially) years simply because that happen to be the state of 
things the second version x.0 came out.

1. Why can't IMP 5.1 require a newer version of a specific package? 
Isn't that part of the intent of the separate framework packages?

2. Is the change to Horde_Imap_Client one that will break IMP 5.0, or 
is it just that this bug won't be fixed without it?

3. No matter how long we took on the release, we would have found 
something like this.

H> Not possible, which means it has to wait for IMP 6.

No. This is **completely** unacceptable.  This is EXACTLY the concerns 
I raised when I said our release process was moving forward too 
quickly. And I was told that backward compatibility would not be an 
issue for minor releases that fix major bugs.

This cannot happen.  This makes the entire packaging system worthless. 
  I will not go through another round of releases where we can't move 
forward because of BC.  That is ridiculous.

If that is not the case, then there will be no IMP 5.1.  I will begin 
work on IMP 6 immediately with the expectation that it will be 
released next month.

Not possible, which means it has to wait for IMP 6.

Reboosting priority, since the workaround doesn't work properly on all 
servers.

This issue can only be resolved by fixing in BOTH imp and 
Horde_Imap_Client.  So IMP 5.1 MUST require Horde_Imap_Client v1.1.0 
or higher.

Changes have been made in Git for this ticket:

Bug #10093: Revert previous attempt to catch UTF7-IMAP mailbox names

  2 files changed, 17 insertions(+), 21 deletions(-)
http://git.horde.org/horde-git/-/commit/843142515711988f4424bdf40baf24209b27d3ae

As a theoretical matter, there still remains some mailbox names that 
won't be handled correctly.  But as a practical matter, these 
mailboxes would require a string of nonsensical letters, bounded by 
'&' and '-', so the odds of running into this are exceedingly slim.   
Still, IMP should be fixed to handle everything in UTF-8 and then have 
the Imap Client do all the necessary UTF7-IMAP abstraction.

Changes have been made in Git for this ticket:

Bug #10093: More thorough checking for UTF7-IMAP encoding
At a minimum, it catches the previous failing case

  2 files changed, 35 insertions(+), 21 deletions(-)
http://git.horde.org/horde-git/-/commit/1f21ab7b25138d1327dfc14a27f2c2bc688f9722

True. :( Well, as matter of fact, this combination will happen less 
often than an ampersand without a dash in the same mailbox name. So I 
guess that's as best as we can get for now.

Changes have been made in Git for this ticket:

Add failing test case for bug #10093.

  1 files changed, 19 insertions(+), 0 deletions(-)
http://git.horde.org/horde-git/-/commit/802c396ddd58c9a3677f6f7b5e42faac4d259738

This isn't a valid fix.  It breaks with this mailbox: "Foo&Bar-2011".

No need to go that far, though I agree that we should consistently use 
either of the encodings throughout the internal APIs.

Changes have been made in Git for this ticket:

Fix detecting of already encoded UTF7-IMAP strings (Bug #10093).

  2 files changed, 7 insertions(+), 2 deletions(-)
http://git.horde.org/horde-git/-/commit/2f85ba2a089919c6253fa8f4cfe362b1e5844531

Then all of Horde_Imap_Client is broken.  Because we allow UTF-8 input 
for all mailbox parameters, which are all internally converted to 
UTF7-IMAP, so all of Horde_Imap_Client is unusable with mailboxes with 
ampersands according to its API.

The best solution is to ALWAYS require UTF-8 mailbox names.  But this 
is going to require a complete rewrite of much of IMP, since we mostly 
use UTF7-IMAP to internally represent mailbox names (although we 
interchangably use the two in several locations).

Which means that IMP 5.1 needs to be released (and soon).  This is a 
major issue to me since it completely prevents entire mailboxes from 
being accessed.

This is EXACTLY why RFC 3501 tells you not to use ampersands in 
mailbox names.  Just because it is technically feasible does not mean 
that we should allow it.

This test whether a string is already encoded doesn't work. Ampersands 
are used for other escapings than escaping the ampersand itself too. I 
can't access the folder "Quarantäne" anymore, for example, because it 
translates to "Quarant&AOQ-ne".

Turns out that this was an issue with the UTF-8 -> UTF7-IMAP 
conversion code in Horde_Imap_Client.

But we still do need to prevent creation of mailboxes with '*' and '%'.

Changes have been made in Git for this ticket:

Bug #10093: Fix UTF-8 -> UTF7-IMAP encoding of ampersands

  3 files changed, 32 insertions(+), 4 deletions(-)
http://git.horde.org/horde-git/-/commit/47607d4a9d59d63c4c4037fdd6f5d0e328dd7370

Also, mailboxes should not have '*' or '%'.  Additionally, it is 
recommended that the delimiter character is not allowed, but since we 
allow power users to create a container + sub-mailbox in one command, 
we don't need to strip delimiters out (or, instead, make this an 
optional check).

I was almost positive that we had a check for this at some point (I 
believe it was a SAPO request), but for some reason we took it out.   
IIRC, it would have been in IMP_Imap_Tree::createMailboxName().  It 
most likely could have been removed simply because we did not have the 
infrastructure in place to handle an error in mailbox name creation.   
Or because we couldn't come up with an adequate way to tell the users 
what mailbox names are acceptable.

Makes sense.
Why? They are perfectly valid. We only have to encode ampersands 
properly to "&-", which should already happen by converting UTF-8 -> 
UTF7-IMAP.

We shouldn't allow creation of mailboxes with ampersands at all.  From 
RFC 3501:

    5)    Two characters, "#" and "&", have meanings by convention, and
          should be avoided except when used in that convention.

The problem is that ampersands are used to encode UTF7-IMAP.  So you 
wouldn't see them in an error message because we are trying to convert 
them from UTF7-IMAP -> UTF-8.

But we shouldn't even get to this point - we should simply reject 
these mailbox names.

The same happens when renaming a folder.

Trying to create a folder through DIMP like "foo & bar" gives me the 
error message "the folder "foo  bar" cannot be created". So these are 
actually two bugs. At one place the folder name isn't correctly 
encoded to be a valid IMAP mailbox name, and the ampersand gets lost 
when reporting back the error.