I'm closing since I can verify that all sorting works fine using 
either IMP 6.0.5 or IMP 6.1 and the latest version of 
Horde_Imap_Client (tested via manually disabling server side sorting 
via backends.php).

I have verified both work fine.

You need to be aware that "From" sorting is by **e-mail address**, NOT 
Name.  So it may look things aren't sorted when, in fact, they are.

Same with subject - if you are sorting by subject and thread sorting 
is active, the subject text won't look very sorted if there are no 
threads in the mailbox.

here, sorting is broken for  "From" or "Subject" ., sorting "Date" and 
"Size" is working fine.  but we will wait to 6.0.5 too :)

Will revisit after 6.0.5 is released.

Thank you for the clarification.  I will wait until the new version of 
IMP is out with your changes and report back.  I assume I have to wait 
for this to be released before I can test it as I was unable to issue 
any type of pear command to download the new version you made the 
changes in.

Except it's not.  That's what I fixed.  Arrival time != sequence 
numbers.  Sequence numbers is no overhead.  Arrival time MAY be less 
expensive than message date, but that depends on the IMAP 
implementation.  Sequence sort will almost certainly look like things 
are not sorted by date, since it has nothing to do with arrival time 
or display dates.
Nope (see below).
This is most likely "arrival time" sort.  And arrival time sort is not 
going to have any bearing to the display dates.
Nope.  This is sequence sort.  And sequence sort is dependent on the 
order in which an IMAP server processes the messages in a mailbox.
You can't switch between arrival and message date in basic view.   
Clicking on the Date header is locked to whatever value the 'datesort' 
parameter is.

Well then I am not sure what to say.  I clearly see the sorting 
problem in Basic mode as well as other modes.  In Basic mode, if I 
login using horde_imap_client version after 2.6.0, the default sort 
order is 'arrival time' which is what I want and is low overhead.   
However, if I click the 'Date' column header, the sorting is changed 
to 'sort by message date' to my understanding.  Once this happens, the 
sorting is completely messed up.  If I click the 'Clear Sort' button, 
then from my understanding, the sort is returned to 'arrival time' 
order and everything is OK again.  So, not sure why you said that date 
sort is not available in Basic mode.  On my production IMP 4.x system, 
this is how it operates as well, however, there there is not clear 
sort button, you have to click the column header with a # sign in it.

I still suspect something is broken between 2.6 and 2.7 in 
horde_imap_client.  Perhaps something that actually processes the mail 
headers and dates within them.

Changes have been made in Git (master):

commit f93b4be4374ec426f8dcdec41273384d4c9c7e42
Author: Michael M Slusarz <slusarz@horde.org>
Date:   Tue Mar 26 16:54:17 2013 -0600

     Bug #12139: Correct usage of arrival vs. sequence for sorting

  imp/js/dimpbase.js          |    2 +-
  imp/lib/Dynamic/Mailbox.php |    9 ++++++---
  2 files changed, 7 insertions(+), 4 deletions(-)

http://git.horde.org/horde-git/-/commit/f93b4be4374ec426f8dcdec41273384d4c9c7e42

It doesn't happen in basic mode since there is no mechanism to 
explicitly change the Date sort - it ALWAYS uses the 'datesort' 
preference.

Since the datesort preference defaults to sequence (pre-6.0.4: 
arrival, which is slightly different), basic view out-of-the-box will 
NEVER display messages in date *display* order (i.e. the dates that 
are actually displayed on the screen).  Sequence sort can appear to 
not be sorted because it is not using those display dates - it is 
using the order the messages were added to the mailbox.

Message date sorting is a totally different sort type - and is 
disfavored because it can be expensive to determine since all message 
headers must be parsed.

One more thing.  I see this behavior in both Traditional (basic) and 
Dynamic modes.  You said that the problem in IMP was limited to 
dynamic I think.

Thanks for the update.  I can get you a trace using IMP but I will 
wait for IMP 6.0.5 to become available and test then.  If the problem 
still persists, I will then get you a trace in the right format for 
two identical mailbox states.

I can clearly see this problem appear after 2.6.0.  I do have caching 
turned on but I purge all the horde cache files, restart the server 
and purge browser cache before testing.

Thanks, and I will report back once 6.0.5 is available.

Aria

Unfortunately, those IMAP logs are useless because:
1. They weren't output by IMP.
2. You are sorting two different mailbox states (one has 782 messages 
in it, one has 785 messages).

I still think what you are seeing is the result of a problem in IMP - 
namely that it was impossible to switch to the non-default datesort in 
dynamic view (it would just display the same as the default datesort). 
  This has been fixed for IMP 6.0.5.

If IMP was truly sending the correct sort type for message date 
searches, you should see something like:

C: 5 UID FETCH [...] (INTERNALDATE ENVELOPE)

However, your IMAP logs both show identical sort FETCHs:

UID FETCH [...] (INTERNALDATE)

which to me tells me that the calling application (IMP) is at fault.   
Although I am not sure why going back to 2.6.0 would fix since no sort 
changes were made between 2.6.0 -> 2.7.0 (unless you are using 
caching, which would explain the behavior).

Changes have been made in Git (master):

commit 116e002681a7d298f1774a4495deb52efafca2f6
Author: Michael M Slusarz <slusarz@horde.org>
Date:   Mon Mar 25 17:31:47 2013 -0600

     [mms] Fix ability to switch to non-default date sorting algorithm 
in dynamic view (Bug #12139).

  imp/docs/CHANGES                          |    2 ++
  imp/js/dimpbase.js                        |    9 ++++-----
  imp/lib/Ajax/Application/ListMessages.php |    2 +-
  imp/lib/Dynamic/Mailbox.php               |    6 +++++-
  imp/package.xml                           |    2 ++
  5 files changed, 14 insertions(+), 7 deletions(-)

http://git.horde.org/horde-git/-/commit/116e002681a7d298f1774a4495deb52efafca2f6

Here is the second attachment.

I am attaching two files.  One is the sort_without_problem which is 
sort by message date using horde_imap_client version 2.4.2 and the 
second sort_with_problem which uses the latest horde_imap_client.

In both, I have removed the trace section where the fetch message 
envelope begins, leaving the first one.

Please let me know if this is not useful or you would like a different 
or more complete trace.

Thanks,
Aria

Can't reproduce.

Now there is something funny going on with **IMP** - in that it seems 
to be sending arrival sorts when date sorting is explicitly selected:

C: 3 UID SORT RETURN (ALL COUNT) (ARRIVAL) US-ASCII ALL
S: * ESEARCH (TAG "3") UID ALL 
37309,44536,46010:46011,46472,46720,46800,46816 COUNT 8
S: 3 OK Sort completed (0.000 secs).

It looks like an issue where the explicit sort is not overriding the 
default date sort set in prefs.

But unless someone can prove otherwise, this doesn't seem to be an 
issue within Horde_Imap_Client.  At a minimum I will need an IMAP log 
showing a problem.



To further debug this issue, we need details of the IMP -> IMAP/POP 
communication.

To enable debugging, see instructions contained in 
imp/config/backends.php (the 'debug' config parameter).

Debugging should not be enabled on a production server,   Attach/post 
only the portion of the log that directly deals with the problem 
reported (it may be simplest to clear the log file and then perform 
the event that causes the error).

Hi,
I just upgraded from horde/webmail 5.0.3 to 5.0.4 which is horde/imp 6.0.3
to 6.0.4.  After the upgrade, when you sort a mailbox by using the 'Message
Date', the sort order is incorrect.  That is you can see groups of message
that are correctly sorted but are not in the right sort order overall.
Sorting by 'Arrival time' appears correct.  I see this in Dynamic/Basic and
Mobile mode.  Unfortunately, in Mobile mode, there is no way to set the sort
order to 'Arrival time' so the Mobile mode is unusable in this manner.

I reported this on the IMP list but got no responses.  I have since 
then installed older versions of horde_imap_client, moving back from 
2.7.2 and have found that the last version that the sorting is correct 
is 2.6.0.  Starting from 2.7.0, the sort by message date is incorrect.

I can provide a IMAP trace for a session with a working 
horde_imap_client and one for a broken one if you provide a secure 
location where I can upload.

Thanks,
Aria