Cyrus IMAP server does not support extended search with charset/suhosin broken

I am using latest Horde 4 with latest IMP 5. Search doesn't

Fri, 08 Apr 2011 10:01:50 +0000

I am using latest Horde 4 with latest IMP 5. Search doesn't work at all no matter what option I choose. I try it in IMP and in DIMP. Search returns all messages everytime. latest Horde 3 with Imp 4 works great with the same settings.

It works for everyone else, so at a minimum you will need to

Fri, 08 Apr 2011 18:11:34 +0000

It works for everyone else, so at a minimum you will need to provide an IMAP log. ----- To further debug this issue, we need details of the IMP -> IMAP/POP communication. To enable debugging, see instructions contained in imp/config/backends.php (the 'debug' config parameter). Debugging should not be enabled on a production server, Attach/post only the portion of the log that directly deals with the problem reported (it may be simplest to clear the log file and then perform the event that causes the error).

I am attaching imap log. I made mailbox with two messages. I

Fri, 08 Apr 2011 19:57:43 +0000

I am attaching imap log. I made mailbox with two messages. I made query which should return just one message. I returns both. Thank you for your reply.

> I am attaching imap log. I made mailbox with two messages.

Fri, 08 Apr 2011 20:13:22 +0000

> I am attaching imap log. I made mailbox with two messages. I made > query which should return just one message. I returns both. > Thank you for your reply. Nothing is being searched for there. How exactly can you reproduce - e.g. what view (dynamic/traditional/mobile), what page, etc.

I logged in Horde -> IMP -> then I went into advanced search

Sat, 09 Apr 2011 19:25:37 +0000

I logged in Horde -> IMP -> then I went into advanced search (simple search not working too) and insert simple search in From field in Inbox folder. Result is all messages.:-(

IMAP search is broken for me, too. Using current Horde 4.0,

Sun, 10 Apr 2011 14:51:06 +0000

IMAP search is broken for me, too. Using current Horde 4.0, IMP 5.0, cyrus imapd 2.2.13 Neither the imp debug log nor the protocol log of cyrus imapd show any sign of the search commands. Seems they are not issued at all. my ./imp/config/backends.local.php $servers['imap'] = array( 'disabled' => false, 'name' => 'duff IMAP', 'hostspec' => 'localhost', 'hordeauth' => true, 'protocol' => 'imap', 'port' => 143, 'secure' => false, 'maildomain' => '', 'acl' => true, 'cache' => true, //'debug' => '/tmp/mail4.log', //'debug_raw' => true, );

forgot to tell: that's true for all search forms I found as

Sun, 10 Apr 2011 14:53:45 +0000

forgot to tell: that's true for all search forms I found as well as for dynview and traditional view (dont know about mobile ones)

> forgot to tell: that's true for all search forms I found a

Mon, 11 Apr 2011 06:29:52 +0000

> forgot to tell: that's true for all search forms I found as well as > for dynview and traditional view (dont know about mobile ones) What browser are you using?

> What browser are you using? Safari 5 and FF4 on MacOS

Mon, 11 Apr 2011 07:17:33 +0000

> What browser are you using? Safari 5 and FF4 on MacOS

> What browser are you using? Iam using chrome and firefo

Mon, 11 Apr 2011 12:42:04 +0000

> What browser are you using? Iam using chrome and firefox under ubuntu and fedora.

What is the URL of your search page? It should look somethi

Mon, 11 Apr 2011 20:07:12 +0000

What is the URL of your search page? It should look something like: [...]/horde/imp/mailbox.php?mailbox=impsearch%00impbsearch

> What is the URL of your search page? It should look somet

Tue, 12 Apr 2011 05:21:06 +0000

> What is the URL of your search page? It should look something like: > > [...]/horde/imp/mailbox.php?mailbox=impsearch%00impbsearch /beta/imp/mailbox.php?mailbox=impsearch%00lJNSYRiV5G1No_BwtrJCIcA

Same here as well. Cyrus 2.4.7 with firefox 3, 4, chrome, a

Tue, 12 Apr 2011 07:12:38 +0000

Same here as well. Cyrus 2.4.7 with firefox 3, 4, chrome, and safari on mac 10.6.7. Upon searching in traditional interface, I land at url: https://webmail.xyz.com/imp/mailbox.php?mailbox=impsearch%00OBeSBUt1iUdNo-sg5QESIHA and 3 times at the top I see errors: Mailbox listing failed: Bad IMAP request: Invalid Search criteria cheers, -scott

> and 3 times at the top I see errors: > Mailbox listing fa

Tue, 12 Apr 2011 08:19:01 +0000

> and 3 times at the top I see errors: > Mailbox listing failed: Bad IMAP request: Invalid Search criteria Can you provide imap logs? (see previous entries for instructions on how to do this) We might be getting somewhere here ... 'Invalid Search criteria' appears nowhere in the Horde codebase so it MUST be something that is returned from the IMAP server.

>> and 3 times at the top I see errors: >> Mailbox listing

Wed, 13 Apr 2011 04:03:42 +0000

>> and 3 times at the top I see errors: >> Mailbox listing failed: Bad IMAP request: Invalid Search criteria > > Can you provide imap logs? (see previous entries for instructions on > how to do this) We might be getting somewhere here ... 'Invalid > Search criteria' appears nowhere in the Horde codebase so it MUST be > something that is returned from the IMAP server. The imap log is attached. I have snipped (marked with ) a lengthly list of mailboxes out of the log, as well as changed user ids's but nothing else has been changed. I do see the Invalid Search Criteria in here so hopefully it helps! Thanks!

Your IMAP server is broken. Your server (Cyrus 2.4.7) adver

Wed, 13 Apr 2011 04:35:09 +0000

Your IMAP server is broken. Your server (Cyrus 2.4.7) advertises that it supports ESEARCH (RFC 4731). Thus we send an extended search command, but the IMAP server is telling us it is bad: (1302667094.7783) C: 5 UID SEARCH RETURN (ALL COUNT) CHARSET UTF-8 FROM horde (1302667094.7785) S: 5 BAD Invalid Search criteria However, there's nothing wrong with that search command. Here's what it looks like when I run it on a dovecot installation: 5 UID SEARCH RETURN (ALL COUNT) CHARSET UTF-8 FROM horde * ESEARCH (TAG "5") UID COUNT 0 5 OK Search completed (0.031 secs). It's not a charset issue since the server must return a NO response rather than a BAD response (and it should return a BADCHARSET response also). Jan, I think you run Cyrus. Are you running 2.4.7? Can you verify that this command works (or is broken) on your installation?

> Your IMAP server is broken. Your server (Cyrus 2.4.7) adv

Wed, 13 Apr 2011 04:45:42 +0000

> Your IMAP server is broken. Your server (Cyrus 2.4.7) advertises > that it supports ESEARCH (RFC 4731). Thus we send an extended search > command, but the IMAP server is telling us it is bad: > > (1302667094.7783) C: 5 UID SEARCH RETURN (ALL COUNT) CHARSET UTF-8 FROM horde > (1302667094.7785) S: 5 BAD Invalid Search criteria > > However, there's nothing wrong with that search command. Here's what > it looks like when I run it on a dovecot installation: > > 5 UID SEARCH RETURN (ALL COUNT) CHARSET UTF-8 FROM horde > * ESEARCH (TAG "5") UID COUNT 0 > 5 OK Search completed (0.031 secs). > > It's not a charset issue since the server must return a NO response > rather than a BAD response (and it should return a BADCHARSET > response also). > > Jan, I think you run Cyrus. Are you running 2.4.7? Can you verify > that this command works (or is broken) on your installation? Thanks for looking at the log. Strangely, other clients, including the previous version of horde (I think H3? over a year+ old) appear to work fine with imap search. I am not sure if they are searching in the same manner or not, but definitely performing successful imap search against cyrus 2.4.7, but will see if I can get a log.

> Thanks for looking at the log. Strangely, other clients,

Wed, 13 Apr 2011 04:54:11 +0000

> Thanks for looking at the log. Strangely, other clients, including > the previous version of horde (I think H3? over a year+ old) appear > to work fine with imap search. I am not sure if they are searching > in the same manner or not, but definitely performing successful imap > search against cyrus 2.4.7, but will see if I can get a log. Older versions of IMP (and other clients) are probably not sending extended search commands.

>> Thanks for looking at the log. Strangely, other clients,

Wed, 13 Apr 2011 05:12:59 +0000

>> Thanks for looking at the log. Strangely, other clients, including >> the previous version of horde (I think H3? over a year+ old) appear >> to work fine with imap search. I am not sure if they are searching >> in the same manner or not, but definitely performing successful imap >> search against cyrus 2.4.7, but will see if I can get a log. > > Older versions of IMP (and other clients) are probably not sending > extended search commands. Ok, making some progress by spying on another client that its working with, then duplicating results in telnet. here is a paste of my telnet session: Working: (no charset) ------------------------------- . SEARCH RETURN (ALL COUNT) FROM horde * ESEARCH (TAG ".") ALL 380,766,1051,1300:1301,1303,1927:1928,3102:3103,4880,4900,4921,4923,5722,5749,5757,6051:6052,22753,22778,22782,22788,22796:22799 COUNT 27 . OK Completed (27 msgs in 0.060 secs) Working w/ charset: ---------------------------------- . SEARCH CHARSET UTF-8 RETURN (ALL COUNT) FROM horde * ESEARCH (TAG ".") ALL 380,766,1051,1300:1301,1303,1927:1928,3102:3103,4880,4900,4921,4923,5722,5749,5757,6051:6052,22753,22778,22782,22788,22796:22799 COUNT 27 . OK Completed (27 msgs in 0.060 secs) Broken in imp: -------------------------------------------------- . SEARCH RETURN (ALL COUNT) CHARSET UTF-8 FROM horde . BAD Invalid Search criteria So it appears the order of the CHARSET option. Not sure if a cyrus specific workaround is needed or if this how its supposed to be. Every mention of charset in this paper shows it at the beginning: http://www.faqs.org/rfcs/rfc5182.html Thanks!

> Working w/ charset: > ----------------------------------

Wed, 13 Apr 2011 05:35:55 +0000

> Working w/ charset: > ---------------------------------- > . SEARCH CHARSET UTF-8 RETURN (ALL COUNT) FROM horde > * ESEARCH (TAG ".") ALL This is flat-out wrong. Pursuant to RFC 4466 [2.6.1], the order of the arguments goes as follows: Arguments: OPTIONAL result specifier OPTIONAL [CHARSET] specification searching criteria (one or more) > So it appears the order of the CHARSET option. Not sure if a cyrus > specific workaround is needed or if this how its supposed to be. > Every mention of charset in this paper shows it at the beginning: > http://www.faqs.org/rfcs/rfc5182.html That's because RFC 5182 is not using the optional result specifier argument. Sure enough, this is a Cyrus bug: http://bugzilla.cyrusimap.org/bugzilla3/show_bug.cgi?id=3435

Changes have been made in Git for this ticket: Bug #9842: P

Wed, 13 Apr 2011 06:08:37 +0000

Changes have been made in Git for this ticket: Bug #9842: Possible workaround for broken Cyrus IMAP behavior 1 files changed, 14 insertions(+), 1 deletions(-) http://git.horde.org/horde-git/-/commit/d2b5777a18dd8b740de9cb2a43b34f3e9ad2dbf2

See if this fixes things.

Wed, 13 Apr 2011 06:09:32 +0000

See if this fixes things.

> See if this fixes things. It sure does. Thanks a milli

Wed, 13 Apr 2011 06:12:43 +0000

> See if this fixes things. It sure does. Thanks a million! Will keep an eye out for cyrus 2.4.8 as well :) cheers!

Changes have been made in Git for this ticket: Bug #9842: W

Wed, 13 Apr 2011 06:27:02 +0000

Changes have been made in Git for this ticket: Bug #9842: Work around broken ESEARCH on Cyrus 2 files changed, 5 insertions(+), 4 deletions(-) http://git.horde.org/horde-git/-/commit/a8c5884b2ec30eed062f614c360901666a356345

It is not working for me. I am using Dovecot Imap server. Ma

Wed, 13 Apr 2011 06:27:15 +0000

It is not working for me. I am using Dovecot Imap server. May you please help me?

> It is not working for me. I am using Dovecot Imap server.

Wed, 13 Apr 2011 06:39:13 +0000

> It is not working for me. I am using Dovecot Imap server. May you > please help me? I have no idea. You will need to debug yourself to determine what is wrong because no developers can reproduce. I would focus on search-basic.php since it is much simpler than the advanced search. Perform a simple search and track some variables using Horde::debug() (see instructions below). I would like to see the debug results for $vars (put Horde::debug($vars) on line 77 of search-basic.php), $q_ob (put Horde::debug($q_ob) on line 84 of search-basic.php), and $imp_search (put Horde::debug($imp_search) on line 55 of mailbox.php). Horde::debug() documentation: http://wiki.horde.org/Doc/Dev/DebugH4

> I would like to see the debug results for $vars (put > H

Wed, 13 Apr 2011 07:33:36 +0000

> I would like to see the debug results for $vars (put > Horde::debug($vars) on line 77 of search-basic.php), $q_ob (put > Horde::debug($q_ob) on line 84 of search-basic.php), and $imp_search > (put Horde::debug($imp_search) on line 55 of mailbox.php). I am attaching log

The IMAP search is still not working for me. The result page

Wed, 13 Apr 2011 18:05:41 +0000

The IMAP search is still not working for me. The result page is the same mailbox again, with search icon and all the mails. Although I use a cyrus 2.2.13 without the ESEARCH, I patched Socket.php (CAPABILITY IMAP4 IMAP4rev1 ACL QUOTA LITERAL+ NAMESPACE UIDPLUS ID NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY SORT THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE XIMAPPROXY) I attached the log for $vars, $q_ob and $imp_search for a simple search for Subject 'XXX' on INBOX.

Unfortunately, this tells me nothing other than the search i

Wed, 13 Apr 2011 20:56:11 +0000

Unfortunately, this tells me nothing other than the search is successfully being stored in the session. Which is what is expected. What does the mailbox look like after you perform the search. Does it say Search Results at the top? Or does it say INBOX (or some other mailbox)? A Horde::debug(IMP::$mailbox) on line 55 of imp/mailbox.php would be useful.

> What does the mailbox look like after you perform the sear

Wed, 13 Apr 2011 21:20:08 +0000

> What does the mailbox look like after you perform the search. Does > it say Search Results at the top? Or does it say INBOX (or some > other mailbox)? Mail :: Inbox, it's the normal INBOX without any sign of the search. > A Horde::debug(IMP::$mailbox) on line 55 of imp/mailbox.php would be useful. 2011-04-13T23:17:17+02:00 DEBUG: Variable information: object(IMP_Mailbox)#112 (2) { ["_cache":protected]=> array(0) { } ["_mbox":protected]=> string(5) "INBOX" } Backtrace: 1. Horde::debug() /var/www/h4/imp/mailbox.php:55

Now we're getting somewhere. Looks like your server is mung

Wed, 13 Apr 2011 21:59:22 +0000

Now we're getting somewhere. Looks like your server is munging the URL parameters somehow. Now we need a Horde::debug($vars) on line 55 of mailbox.php. And for good measure, put a Horde::debug($_REQUEST) on the line after that.

> Now we need a Horde::debug($vars) on line 55 of mailbox.ph

Wed, 13 Apr 2011 22:46:56 +0000

> Now we need a Horde::debug($vars) on line 55 of mailbox.php. And for > good measure, put a Horde::debug($_REQUEST) on the line after that. <516> root@duff/tmp/h4# less horde_debug.txt 2011-04-14T00:41:21+02:00 DEBUG: Variable information: object(Horde_Variables)#231 (3) { ["_vars":protected]=> array(7) { ["horde_sidebar_expanded"]=> string(1) "0" ["horde_menu_expanded"]=> string(21) "expimp,administration" ["imp_key"]=> string(32) "c6ad08b2e81994328a077e2962a2741a" ["default_horde_view"]=> string(11) "traditional" ["Horde4"]=> string(32) "c6ad08b2e81994328a077e2962a2741a" ["auth_key"]=> string(23) "18847229984da5c8a601048" ["SESSc0d2b4ced4b5e9b5a4f261e22dcfe7ba"]=> string(32) "08db564168fd858ea6bcd95cd5bbb32a" } ["_expectedVariables":protected]=> array(0) { } ["_sanitized":protected]=> bool(false) } Backtrace: 1. Horde::debug() /var/www/h4/imp/mailbox.php:55 2011-04-14T00:41:21+02:00 DEBUG: Variable information: array(7) { ["horde_sidebar_expanded"]=> string(1) "0" ["horde_menu_expanded"]=> string(21) "expimp,administration" ["imp_key"]=> string(32) "c6ad08b2e81994328a077e2962a2741a" ["default_horde_view"]=> string(11) "traditional" ["Horde4"]=> string(32) "c6ad08b2e81994328a077e2962a2741a" ["auth_key"]=> string(23) "18847229984da5c8a601048" ["SESSc0d2b4ced4b5e9b5a4f261e22dcfe7ba"]=> string(32) "08db564168fd858ea6bcd95cd5bbb32a" } Backtrace: 1. Horde::debug() /var/www/h4/imp/mailbox.php:56

> 2011-04-14T00:41:21+02:00 DEBUG: Variable information:

Thu, 14 Apr 2011 18:40:00 +0000

> 2011-04-14T00:41:21+02:00 DEBUG: Variable information: And there's your problem. The mailbox entry is nowhere to be found. Either your PHP or web server is incorrectly munging the URL parameter data. What version of PHP are you using? You wouldn't happen to be using something like suhosin either?

>> 2011-04-14T00:41:21+02:00 DEBUG: Variable information: >

Thu, 14 Apr 2011 19:40:27 +0000

>> 2011-04-14T00:41:21+02:00 DEBUG: Variable information: > > And there's your problem. The mailbox entry is nowhere to be found. > Either your PHP or web server is incorrectly munging the URL > parameter data. > > What version of PHP are you using? You wouldn't happen to be using > something like suhosin either? That is true.. I disabled suhosin and everything is working:-) Is there something to do to able working with suhosin? Where is the problem? Thank you for your time

> That is true.. I disabled suhosin and everything is workin

Thu, 14 Apr 2011 19:51:29 +0000

> That is true.. I disabled suhosin and everything is working:-) > Is there something to do to able working with suhosin? > Where is the problem? > Thank you for your time Ugh. Suhosin sucks then. Apparently it feels it is a security threat to have a null character in URL parameters, even though it is perfectly permissible. I'm not sure there is anything we can do about this other than to say IMP does not work with suhosin.

Did you use the Suhosin extension or the patch? The extensio

Thu, 14 Apr 2011 22:19:52 +0000

Did you use the Suhosin extension or the patch? The extension already caused other problems in the past, so I don't have any problem with discouraging its usage altogether.

My PHP 5.3.3 (debian) is suhosin patched AND had the suhosin

Fri, 15 Apr 2011 09:04:06 +0000

My PHP 5.3.3 (debian) is suhosin patched AND had the suhosin extension. Without the extension, IMAP search does work now. Thanks for the hint!

Changes have been made in Git for this ticket: Bug #9842: I

Fri, 15 Apr 2011 17:43:22 +0000

Changes have been made in Git for this ticket: Bug #9842: IMP 5 will not work with broken suhosin extension 1 files changed, 5 insertions(+), 0 deletions(-) http://git.horde.org/horde-git/-/commit/63ba95c6b2027fc7995ec122ad200c33bbbeaa80

Unfortunately, it appears that debian may ship PHP by defaul

Wed, 18 May 2011 05:52:28 +0000

Unfortunately, it appears that debian may ship PHP by default with suhosin. Which is stupid but whatever. (The benefits of suhosin (are there any?) are entirely debatable. Completely breaking functionality, in the name of quote security unquote, is not a valid solution. But I guess you throw the security term around and people will pick up on it as a buzz word so they will use your product, no matter what it actually does. Off soapbox). So to avoid IMP being broken on all debian servers by default, and to work around this issue, we will instead always pass around the mailbox in form data (both POST and GET) as base64url encoded. I have locally fixed this and will commit to the develop branch (or a imp 5.1 topic branch) as soon as I am finished testing/squashing all the bugs.

Changing milestone. Will commit to master once 5.0.4 is rel

Mon, 23 May 2011 19:08:27 +0000

Changing milestone. Will commit to master once 5.0.4 is released.

Unless I'm missing something, you did already commit this.

Mon, 23 May 2011 20:07:16 +0000

Unless I'm missing something, you *did* already commit this. At least my URL hashes in DIMP are all encoded since a few days now.

No. I committed a prior patch to the imp_5_1 branch, but th

Mon, 23 May 2011 20:18:11 +0000

No. I committed a prior patch to the imp_5_1 branch, but then deleted the branch. I don't really understand the concept of the develop branch at this time, so I did not repush there. What you are probably seeing is base64url encoding of dimp's hash (breadcrumb) information. But this has nothing to do with encoding mailbox names in form data/url links.

Changes have been made in Git for this ticket: Bug #9842: B

Wed, 25 May 2011 16:28:57 +0000

Changes have been made in Git for this ticket: Bug #9842: Base64url encode mailbox names in form data 22 files changed, 152 insertions(+), 122 deletions(-) http://git.horde.org/horde-git/-/commit/32c3fac7b3a089cd7914cfdd7578d5e8775f0d15