6.0.0-git
2019-03-24

[#9842] Cyrus IMAP server does not support extended search with charset/suhosin broken
Summary Cyrus IMAP server does not support extended search with charset/suhosin broken
Queue IMP
Queue Version 5.0
Type Bug
State Resolved
Priority 2. Medium
Owners slusarz (at) horde (dot) org
Requester info (at) standa-david (dot) com
Created 2011-04-08 (2907 days ago)
Due
Updated 2011-05-25 (2860 days ago)
Assigned 2011-05-18 (2867 days ago)
Resolved 2011-05-25 (2860 days ago)
Milestone 5.0.5
Patch No

History
2011-05-25 16:29:30 Michael Slusarz State ⇒ Resolved
 
2011-05-25 16:28:57 Git Commit Comment #43 Reply to this comment
Changes have been made in Git for this ticket:

Bug #9842: Base64url encode mailbox names in form data

  22 files changed, 152 insertions(+), 122 deletions(-)
http://git.horde.org/horde-git/-/commit/32c3fac7b3a089cd7914cfdd7578d5e8775f0d15
2011-05-23 20:18:11 Michael Slusarz Comment #42 Reply to this comment
No.  I committed a prior patch to the imp_5_1 branch, but then deleted 
the branch.  I don't really understand the concept of the develop 
branch at this time, so I did not repush there.

What you are probably seeing is base64url encoding of dimp's hash 
(breadcrumb) information.  But this has nothing to do with encoding 
mailbox names in form data/url links.
2011-05-23 20:07:16 Jan Schneider Comment #41 Reply to this comment
Unless I'm missing something, you *did* already commit this. At least 
my URL hashes in DIMP are all encoded since a few days now.
2011-05-23 19:08:27 Michael Slusarz Comment #40
Milestone ⇒ 5.0.5
Reply to this comment
Changing milestone.  Will commit to master once 5.0.4 is released.
2011-05-18 05:52:28 Michael Slusarz Comment #39
State ⇒ Assigned
Milestone ⇒ 5.1
Reply to this comment
Unfortunately, it appears that debian may ship PHP by default with 
suhosin.  Which is stupid but whatever.  (The benefits of suhosin (are 
there any?) are entirely debatable.  Completely breaking 
functionality, in the name of quote security unquote, is not a valid 
solution.  But I guess you throw the security term around and people 
will pick up on it as a buzz word so they will use your product, no 
matter what it actually does.  Off soapbox).

So to avoid IMP being broken on all debian servers by default, and to 
work around this issue, we will instead always pass around the mailbox 
in form data (both POST and GET) as base64url encoded.  I have locally 
fixed this and will commit to the develop branch (or a imp 5.1 topic 
branch) as soon as I am finished testing/squashing all the bugs.
2011-04-15 17:44:05 Michael Slusarz Summary ⇒ Cyrus IMAP server does not support extended search with charset/suhosin broken
State ⇒ Resolved
 
2011-04-15 17:43:22 Git Commit Comment #38 Reply to this comment
Changes have been made in Git for this ticket:

Bug #9842: IMP 5 will not work with broken suhosin extension

  1 files changed, 5 insertions(+), 0 deletions(-)
http://git.horde.org/horde-git/-/commit/63ba95c6b2027fc7995ec122ad200c33bbbeaa80
2011-04-15 09:04:06 xk3 (at) mompl (dot) org Comment #37 Reply to this comment
My PHP 5.3.3 (debian) is suhosin patched AND had the suhosin 
extension. Without the extension, IMAP search does work now.

Thanks for the hint!
2011-04-14 22:19:52 Jan Schneider Comment #36 Reply to this comment
Did you use the Suhosin extension or the patch? The extension already 
caused other problems in the past, so I don't have any problem with 
discouraging its usage altogether.
2011-04-14 19:51:29 Michael Slusarz Comment #35 Reply to this comment
That is true.. I disabled suhosin and everything is working:-)
Is there something to do to able working with suhosin?
Where is the problem?
Thank you for your time
Ugh.  Suhosin sucks then.  Apparently it feels it is a security threat 
to have a null character in URL parameters, even though it is 
perfectly permissible.  I'm not sure there is anything we can do about 
this other than to say IMP does not work with suhosin.
2011-04-14 19:40:27 info (at) standa-david (dot) com Comment #34 Reply to this comment
2011-04-14T00:41:21+02:00 DEBUG: Variable information:
And there's your problem.  The mailbox entry is nowhere to be found. 
  Either your PHP or web server is incorrectly munging the URL 
parameter data.

What version of PHP are you using?  You wouldn't happen to be using 
something like suhosin either?
That is true.. I disabled suhosin and everything is working:-)
Is there something to do to able working with suhosin?
Where is the problem?
Thank you for your time

2011-04-14 18:40:00 Michael Slusarz Comment #33 Reply to this comment
2011-04-14T00:41:21+02:00 DEBUG: Variable information:
And there's your problem.  The mailbox entry is nowhere to be found.   
Either your PHP or web server is incorrectly munging the URL parameter 
data.

What version of PHP are you using?  You wouldn't happen to be using 
something like suhosin either?
2011-04-13 22:46:56 xk3 (at) mompl (dot) org Comment #32 Reply to this comment
Now we need a Horde::debug($vars) on line 55 of mailbox.php.  And 
for good measure, put a Horde::debug($_REQUEST) on the line after 
that.
<516> root@duff/tmp/h4# less horde_debug.txt

2011-04-14T00:41:21+02:00 DEBUG: Variable information:
object(Horde_Variables)#231 (3) {
   ["_vars":protected]=>
   array(7) {
     ["horde_sidebar_expanded"]=>
     string(1) "0"
     ["horde_menu_expanded"]=>
     string(21) "expimp,administration"
     ["imp_key"]=>
     string(32) "c6ad08b2e81994328a077e2962a2741a"
     ["default_horde_view"]=>
     string(11) "traditional"
     ["Horde4"]=>
     string(32) "c6ad08b2e81994328a077e2962a2741a"
     ["auth_key"]=>
     string(23) "18847229984da5c8a601048"
     ["SESSc0d2b4ced4b5e9b5a4f261e22dcfe7ba"]=>
     string(32) "08db564168fd858ea6bcd95cd5bbb32a"
   }
   ["_expectedVariables":protected]=>
   array(0) {
   }
   ["_sanitized":protected]=>
   bool(false)
}

Backtrace:
1. Horde::debug() /var/www/h4/imp/mailbox.php:55

2011-04-14T00:41:21+02:00 DEBUG: Variable information:
array(7) {
   ["horde_sidebar_expanded"]=>
   string(1) "0"
   ["horde_menu_expanded"]=>
   string(21) "expimp,administration"
   ["imp_key"]=>
   string(32) "c6ad08b2e81994328a077e2962a2741a"
   ["default_horde_view"]=>
   string(11) "traditional"
   ["Horde4"]=>
   string(32) "c6ad08b2e81994328a077e2962a2741a"
   ["auth_key"]=>
   string(23) "18847229984da5c8a601048"
   ["SESSc0d2b4ced4b5e9b5a4f261e22dcfe7ba"]=>
   string(32) "08db564168fd858ea6bcd95cd5bbb32a"
}

Backtrace:
1. Horde::debug() /var/www/h4/imp/mailbox.php:56

2011-04-13 21:59:22 Michael Slusarz Comment #31 Reply to this comment
Now we're getting somewhere.  Looks like your server is munging the 
URL parameters somehow.

Now we need a Horde::debug($vars) on line 55 of mailbox.php.  And for 
good measure, put a Horde::debug($_REQUEST) on the line after that.
2011-04-13 21:20:08 xk3 (at) mompl (dot) org Comment #30 Reply to this comment
What does the mailbox look like after you perform the search.  Does 
it say Search Results at the top?  Or does it say INBOX (or some 
other mailbox)?
Mail :: Inbox, it's the normal INBOX without any sign of the search.
A Horde::debug(IMP::$mailbox) on line 55 of imp/mailbox.php would be useful.
2011-04-13T23:17:17+02:00 DEBUG: Variable information:
object(IMP_Mailbox)#112 (2) {
   ["_cache":protected]=>
   array(0) {
   }
   ["_mbox":protected]=>
   string(5) "INBOX"
}

Backtrace:
1. Horde::debug() /var/www/h4/imp/mailbox.php:55

2011-04-13 20:56:11 Michael Slusarz Comment #29 Reply to this comment
Unfortunately, this tells me nothing other than the search is 
successfully being stored in the session.  Which is what is expected.

What does the mailbox look like after you perform the search.  Does it 
say Search Results at the top?  Or does it say INBOX (or some other 
mailbox)?

A Horde::debug(IMP::$mailbox) on line 55 of imp/mailbox.php would be useful.
2011-04-13 18:05:41 xk3 (at) mompl (dot) org Comment #28
New Attachment: cyrus.2.2.13_search_INBOX_Subject_XXX.txt Download
Reply to this comment
The IMAP search is still not working for me. The result page is the 
same mailbox again, with search icon and all the mails.

Although I use a cyrus 2.2.13 without the ESEARCH, I patched Socket.php
(CAPABILITY IMAP4 IMAP4rev1 ACL QUOTA LITERAL+ NAMESPACE UIDPLUS ID 
NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY SORT 
THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE XIMAPPROXY)

I attached the log for $vars, $q_ob and $imp_search for a simple 
search for Subject 'XXX' on INBOX.
2011-04-13 07:33:36 info (at) standa-david (dot) com Comment #27
New Attachment: log.txt Download
Reply to this comment
I would like to see the debug results for $vars (put 
Horde::debug($vars) on line 77 of search-basic.php), $q_ob (put 
Horde::debug($q_ob) on line 84 of search-basic.php), and $imp_search 
(put Horde::debug($imp_search) on line 55 of mailbox.php).
I am attaching log
2011-04-13 06:39:13 Michael Slusarz Comment #26
Taken from Jan Schneider
Reply to this comment
It is not working for me. I am using Dovecot Imap server. May you 
please help me?
I have no idea.  You will need to debug yourself to determine what is 
wrong because no developers can reproduce.

I would focus on search-basic.php since it is much simpler than the 
advanced search.  Perform a simple search and track some variables 
using Horde::debug() (see instructions below).

I would like to see the debug results for $vars (put 
Horde::debug($vars) on line 77 of search-basic.php), $q_ob (put 
Horde::debug($q_ob) on line 84 of search-basic.php), and $imp_search 
(put Horde::debug($imp_search) on line 55 of mailbox.php).

Horde::debug() documentation: http://wiki.horde.org/Doc/Dev/DebugH4
2011-04-13 06:27:15 info (at) standa-david (dot) com Comment #25 Reply to this comment
It is not working for me. I am using Dovecot Imap server. May you 
please help me?
2011-04-13 06:27:02 Git Commit Comment #24 Reply to this comment
Changes have been made in Git for this ticket:

Bug #9842: Work around broken ESEARCH on Cyrus

  2 files changed, 5 insertions(+), 4 deletions(-)
http://git.horde.org/horde-git/-/commit/a8c5884b2ec30eed062f614c360901666a356345
2011-04-13 06:12:43 scott (at) spotman (dot) net Comment #23 Reply to this comment
See if this fixes things.
It sure does.  Thanks a million!  Will keep an eye out for cyrus 2.4.8 
as well :)

cheers!

2011-04-13 06:09:32 Michael Slusarz Comment #22
Priority ⇒ 2. Medium
Reply to this comment
See if this fixes things.
2011-04-13 06:08:37 Git Commit Comment #21 Reply to this comment
Changes have been made in Git for this ticket:

Bug #9842: Possible workaround for broken Cyrus IMAP behavior

  1 files changed, 14 insertions(+), 1 deletions(-)
http://git.horde.org/horde-git/-/commit/d2b5777a18dd8b740de9cb2a43b34f3e9ad2dbf2
2011-04-13 05:39:13 Michael Slusarz Summary ⇒ Cyrus IMAP server does not support extended search with charset
 
2011-04-13 05:35:55 Michael Slusarz Comment #20 Reply to this comment
Working w/ charset:
----------------------------------
. SEARCH CHARSET UTF-8 RETURN (ALL COUNT) FROM horde
* ESEARCH (TAG ".") ALL
This is flat-out wrong.  Pursuant to RFC 4466 [2.6.1], the order of 
the arguments goes as follows:

    Arguments:  OPTIONAL result specifier
                OPTIONAL [CHARSET] specification
                searching criteria (one or more)
So it appears the order of the CHARSET option.  Not sure if a cyrus 
specific workaround is needed or if this how its supposed to be.   
Every mention of charset in this paper shows it at the beginning: 
http://www.faqs.org/rfcs/rfc5182.html
That's because RFC 5182 is not using the optional result specifier argument.

Sure enough, this is a Cyrus bug:
http://bugzilla.cyrusimap.org/bugzilla3/show_bug.cgi?id=3435

2011-04-13 05:12:59 scott (at) spotman (dot) net Comment #19 Reply to this comment
Thanks for looking at the log.  Strangely, other clients, including
the previous version of horde (I think H3? over a year+ old)  appear
to work fine with imap search.  I am not sure if they are searching
in the same manner or not, but definitely performing successful imap
search against cyrus 2.4.7, but will see if I can get a log.
Older versions of IMP (and other clients) are probably not sending 
extended search commands.
Ok, making some progress by spying on another client that its working 
with, then duplicating results in telnet.  here is a paste of my 
telnet session:

Working: (no charset)
-------------------------------
. SEARCH RETURN (ALL COUNT) FROM horde
* ESEARCH (TAG ".") ALL 
380,766,1051,1300:1301,1303,1927:1928,3102:3103,4880,4900,4921,4923,5722,5749,5757,6051:6052,22753,22778,22782,22788,22796:22799 COUNT 
27
. OK Completed (27 msgs in 0.060 secs)


Working w/ charset:
----------------------------------
. SEARCH CHARSET UTF-8 RETURN (ALL COUNT) FROM horde
* ESEARCH (TAG ".") ALL 
380,766,1051,1300:1301,1303,1927:1928,3102:3103,4880,4900,4921,4923,5722,5749,5757,6051:6052,22753,22778,22782,22788,22796:22799 COUNT 
27
. OK Completed (27 msgs in 0.060 secs)

Broken in imp:
--------------------------------------------------
. SEARCH RETURN (ALL COUNT) CHARSET UTF-8 FROM horde
. BAD Invalid Search criteria

So it appears the order of the CHARSET option.  Not sure if a cyrus 
specific workaround is needed or if this how its supposed to be.   
Every mention of charset in this paper shows it at the beginning: 
http://www.faqs.org/rfcs/rfc5182.html

Thanks!
2011-04-13 04:54:11 Michael Slusarz Comment #18 Reply to this comment
Thanks for looking at the log.  Strangely, other clients, including 
the previous version of horde (I think H3? over a year+ old)  appear 
to work fine with imap search.  I am not sure if they are searching 
in the same manner or not, but definitely performing successful imap 
search against cyrus 2.4.7, but will see if I can get a log.
Older versions of IMP (and other clients) are probably not sending 
extended search commands.
2011-04-13 04:45:42 scott (at) spotman (dot) net Comment #17 Reply to this comment

[Show Quoted Text - 20 lines]
Thanks for looking at the log.  Strangely, other clients, including 
the previous version of horde (I think H3? over a year+ old)  appear 
to work fine with imap search.  I am not sure if they are searching in 
the same manner or not, but definitely performing successful imap 
search against cyrus 2.4.7, but will see if I can get a log.



2011-04-13 04:35:09 Michael Slusarz Comment #16
Assigned to Jan Schneider
Assigned to Michael Slusarz
State ⇒ Feedback
Reply to this comment
Your IMAP server is broken.  Your server (Cyrus 2.4.7) advertises that 
it supports ESEARCH (RFC 4731).  Thus we send an extended search 
command, but the IMAP server is telling us it is bad:

(1302667094.7783) C: 5 UID SEARCH RETURN (ALL COUNT) CHARSET UTF-8 FROM horde
(1302667094.7785) S: 5 BAD Invalid Search criteria

However, there's nothing wrong with that search command.  Here's what 
it looks like when I run it on a dovecot installation:

5 UID SEARCH RETURN (ALL COUNT) CHARSET UTF-8 FROM horde
* ESEARCH (TAG "5") UID COUNT 0
5 OK Search completed (0.031 secs).

It's not a charset issue since the server must return a NO response 
rather than a BAD response (and it should return a BADCHARSET response 
also).

Jan, I think you run Cyrus.  Are you running 2.4.7?  Can you verify 
that this command works (or is broken) on your installation?
2011-04-13 04:03:42 scott (at) spotman (dot) net Comment #15
New Attachment: imaplog.txt Download
Reply to this comment
and 3 times at the top I see errors:
Mailbox listing failed: Bad IMAP request: Invalid Search criteria
Can you provide imap logs? (see previous entries for instructions on 
how to do this)  We might be getting somewhere here ... 'Invalid 
Search criteria' appears nowhere in the Horde codebase so it MUST be 
something that is returned from the IMAP server.
The imap log is attached.  I have snipped (marked with <snip> ) a 
lengthly list of mailboxes out of the log, as well as changed user 
ids's but nothing else has been changed.  I do see the Invalid Search 
Criteria in here so hopefully it helps!

Thanks!

2011-04-12 08:19:01 Michael Slusarz Comment #14 Reply to this comment
and 3 times at the top I see errors:
Mailbox listing failed: Bad IMAP request: Invalid Search criteria
Can you provide imap logs? (see previous entries for instructions on 
how to do this)  We might be getting somewhere here ... 'Invalid 
Search criteria' appears nowhere in the Horde codebase so it MUST be 
something that is returned from the IMAP server.
2011-04-12 07:12:38 scott (at) spotman (dot) net Comment #13 Reply to this comment
Same here as well.  Cyrus 2.4.7 with firefox 3, 4, chrome, and safari 
on mac 10.6.7.

Upon searching in traditional interface, I land at url:

https://webmail.xyz.com/imp/mailbox.php?mailbox=impsearch%00OBeSBUt1iUdNo-sg5QESIHA

and 3 times at the top I see errors:
Mailbox listing failed: Bad IMAP request: Invalid Search criteria

cheers,
-scott
2011-04-12 05:21:06 info (at) standa-david (dot) com Comment #12 Reply to this comment
What is the URL of your search page?  It should look something like:

[...]/horde/imp/mailbox.php?mailbox=impsearch%00impbsearch
/beta/imp/mailbox.php?mailbox=impsearch%00lJNSYRiV5G1No_BwtrJCIcA

2011-04-11 20:07:12 Michael Slusarz Comment #11 Reply to this comment
What is the URL of your search page?  It should look something like:

[...]/horde/imp/mailbox.php?mailbox=impsearch%00impbsearch
2011-04-11 12:42:04 info (at) standa-david (dot) com Comment #10 Reply to this comment
What browser are you using?
Iam using chrome and firefox under ubuntu and fedora.

2011-04-11 07:17:33 xk3 (at) mompl (dot) org Comment #9 Reply to this comment
What browser are you using?
Safari 5 and FF4 on MacOS
2011-04-11 06:29:52 Michael Slusarz Comment #8 Reply to this comment
forgot to tell: that's true for all search forms I found as well as 
for dynview and traditional view (dont know about mobile ones)
What browser are you using?
2011-04-10 14:53:45 xk3 (at) mompl (dot) org Comment #7 Reply to this comment
forgot to tell: that's true for all search forms I found as well as 
for dynview and traditional view (dont know about mobile ones)
2011-04-10 14:51:06 xk3 (at) mompl (dot) org Comment #6 Reply to this comment
IMAP search is broken for me, too.
Using current Horde 4.0, IMP 5.0, cyrus imapd 2.2.13

Neither the imp debug log nor the protocol log of cyrus imapd show any 
sign of the search commands. Seems they are not issued at all.

my ./imp/config/backends.local.php
$servers['imap'] = array(
     'disabled' => false,
     'name' => 'duff IMAP',
     'hostspec' => 'localhost',
     'hordeauth' => true,
     'protocol' => 'imap',
     'port' => 143,
     'secure' => false,
     'maildomain' => '',
     'acl' => true,
     'cache' => true,
     //'debug' => '/tmp/mail4.log',
     //'debug_raw' => true,
);

2011-04-09 19:25:37 info (at) standa-david (dot) com Comment #5 Reply to this comment
I logged in Horde -> IMP -> then I went into advanced search (simple 
search not working too) and insert simple search in From field in 
Inbox folder. Result is all messages.:-(
2011-04-08 20:13:22 Michael Slusarz Comment #4 Reply to this comment
I am attaching imap log. I made mailbox with two messages. I made 
query which should return just one message. I returns both.
Thank you for your reply.
Nothing is being searched for there.

How exactly can you reproduce - e.g. what view 
(dynamic/traditional/mobile), what page, etc.

2011-04-08 19:57:43 info (at) standa-david (dot) com Comment #3
New Attachment: imaplog Download
Reply to this comment
I am attaching imap log. I made mailbox with two messages. I made 
query which should return just one message. I returns both.
Thank you for your reply.

2011-04-08 18:11:34 Michael Slusarz Comment #2
Priority ⇒ 1. Low
Reply to this comment
It works for everyone else, so at a minimum you will need to provide 
an IMAP log.

-----

To further debug this issue, we need details of the IMP -> IMAP/POP 
communication.

To enable debugging, see instructions contained in 
imp/config/backends.php (the 'debug' config parameter).

Debugging should not be enabled on a production server,   Attach/post 
only the portion of the log that directly deals with the problem 
reported (it may be simplest to clear the log file and then perform 
the event that causes the error).
2011-04-08 10:01:50 info (at) standa-david (dot) com Comment #1
Type ⇒ Bug
State ⇒ Unconfirmed
Priority ⇒ 3. High
Summary ⇒ Search does not work
Queue ⇒ IMP
Milestone ⇒
Patch ⇒ No
Reply to this comment
I am using latest Horde 4 with latest IMP 5. Search doesn't work at 
all no matter what
option I choose. I try it in IMP and in DIMP. Search returns all 
messages everytime.
latest Horde 3 with Imp 4 works great with the same settings.

Saved Queries