6.0.0-git
2019-05-25

[#14810] Server side email searching through ActiveSync doesn't work
Summary Server side email searching through ActiveSync doesn't work
Queue IMP
Queue Version FRAMEWORK_5_2
Type Bug
State Not Reproducible
Priority 1. Low
Owners
Requester s.arcus (at) open-t (dot) co (dot) uk
Created 2018-04-17 (403 days ago)
Due
Updated 2018-04-26 (394 days ago)
Assigned
Resolved 2018-04-26 (394 days ago)
Milestone
Patch No

History
2018-04-26 01:07:22 Michael Rubinsky State ⇒ Not Reproducible
 
2018-04-25 15:09:48 s (dot) arcus (at) open-t (dot) co (dot) uk Comment #6 Reply to this comment
I'm sorry to keep on persisting with this bug, but the server side 
search through ActiveSync is still not working for me properly. It is 
completely unpredictable at best - maybe 1 in 20 searches actually 
produce results, if I force the search to be passed to the server.

I have checked that I don't have more than one IMAP server configured 
in backends.php/backends.local.php.

I have also turned on and up debugging in Dovecot - and when I run 
searches from the phone, all I see is a bunch of EXAMINE commands in 
the Dovecot log. I don't see any SEARCH imap command being passed on 
to Dovecot. Maybe Horde doesn't search using the SEARCH imap command? 
In any event, the server side searches are not working. I have tried 
it on two different servers, with two iPhones and one Jolla phone - 
and it is the same on all of them. I have enabled full text indexing 
on Dovecot - so any searches actually passed to Dovecot should return 
in a matter of seconds.

Any further suggestions would be much appreciated - unless someone can 
take a look at the Horde code and confirm what is supposed or not 
supposed to happen when doing ActiveSync server side searches.
2018-04-18 19:24:21 Michael Rubinsky Comment #5 Reply to this comment
servers['imap'] ( ... )

and

servers['advanced'] ( ... )

So, I can only assume that 'secure-imap' syntax has been deprecated 
at some point in time.
The array key is arbitrary, it's not part of the configuration syntax, 
you just need to be sure that you only have 1 "active" configuration 
(or enable multiple servers but that's another story).

2018-04-18 18:52:05 s (dot) arcus (at) open-t (dot) co (dot) uk Comment #4 Reply to this comment
"I'm not sure what you mean by an "old syntax" and you don't specify 
what your configuration actually was.... "

In my comment I mentioned that I was using the format:

servers['secure-imap'] ( ... )

The current horde/imp/config/backends.php sample config file only 
lists two types of syntax for IMAP backends:

servers['imap'] ( ... )

and

servers['advanced'] ( ... )

So, I can only assume that 'secure-imap' syntax has been deprecated at 
some point in time.

I have turned debugging all the way up in Dovecot, and when using 
'secure-imap', no SEARCH commands are sent to Dovecot. I have also 
enabled full text indexing in Dovecot, and searching initiated over a 
telnet session to Dovecot using the 'SEARCH BODY' imap command are now 
down from 5+ minutes on a large mailbox, to less than one second. In 
spite of all this, server side email searching through ActiveSync 
still didn't work and didn't send any SEARCH commands to the imap 
server.

"My guess is the search was timing out looking for the IMAP server"

If you mean that Horde couldn't connect to the IMAP server at all, I 
don't think that was the case, as emails were downloading fine on the 
phone and I could open them, send emails etc.

If you mean that the search was timing out waiting for a reply to the 
SEARCH command, again, I don't think that is the case, as no SEARCH 
command was showing up in Dovecot logs, and, as mentioned above, after 
enabling FTS indexing, the search results in Dovecot would have taken 
less than a second.

What I can only guess was happening (but this would need confirming by 
looking at the code) is that the 'secure-imap' syntax in 
backends.local.php was interpreted by Horde as a server of simple imap 
type (not advanced) - and somehow this doesn't support server side 
email searches. Or maybe something else was going on.

If the above is correct, what I think should happen is for Horde to 
detect that the selected IMAP back-end doesn't support server-side 
searching (because it is of simple type, not advanced) - and output 
some warnings or errors in the Horde log, when ActiveSync devices 
attempt to use search commands

Also, if it is correct that the 'secure-imap' syntax has been 
deprecated, again, maybe Horde should flag this in the logs - instead 
of silently accepting it (if that is the case).

Please find below the backend configuration which I was using BEFORE - 
which wasn't passing SEARCH commands to the imap server:

$servers['secure-imap'] = array(
     'disabled' => false,
     'name' => 'my_imap_config',
     'hostspec' => 'localhost',
     'hordeauth' => 'full',
     'protocol' => 'imap',
     'port' => 993,
     'secure' => 'ssl',
     'maildomain' => 'my_mail_domain.com',
     'smtphost' => 'localhost',
     'smtpport' => 587,
     'acl' => false,
     'cache' => false,
);
2018-04-18 16:48:11 Michael Rubinsky Priority ⇒ 1. Low
 
2018-04-18 16:47:53 Michael Rubinsky Comment #3 Reply to this comment
I'm not sure what you mean by an "old syntax" and you don't specify 
what your configuration actually was....

Either way, not sure what we can do about a misconfiguration on the 
server. EAS doesn't provide for any mechanism to display arbitrary 
errors like that. My guess is the search was timing out looking for 
the IMAP server.  So, I'll look to make sure we aren't missing some 
exception being caught in that code path.

2018-04-18 08:14:37 s (dot) arcus (at) open-t (dot) co (dot) uk Comment #2 Reply to this comment
OK - I think I finally figured out the mystery. My IMAP server in 
horde/imp/config/backends.local.php was using an old syntax:

$servers['secure-imap'] = array(

This syntax is not even present any more in the current backends.php - 
so I guess it must have been defaulting to the simple IMAP server - 
which, again, I am guessing that it doesn't support server side email 
searches (although I couldn't find anything int the Horde docs to 
confirm or disprove this)

I should have read the upgrade DOCS a lot more closely - but it would 
have been helpful if there would have been some relevant errors in the 
Horde log.

Could one of the admins change the title of this bug to "No 
error/warning in logs if ActiveSync device attempts server-side email 
searches on an unsupported back-end" please.

Maybe Horde could output some relevant errors to the logs when this 
happens, to make it easier to track the reason why server-side email 
searches don't work.
2018-04-17 07:21:03 s (dot) arcus (at) open-t (dot) co (dot) uk Comment #1
Type ⇒ Bug
State ⇒ Unconfirmed
Priority ⇒ 2. Medium
Summary ⇒ Server side email searching through ActiveSync doesn't work
Queue ⇒ IMP
Milestone ⇒
Patch ⇒ No
Reply to this comment
Server side searches through ActiveSync appear to not work at all. My setup:

Lighttpd 1.4.49
Dovecot 2.3.1
Exim 4.90
PHP 5.6.35
Postgresql 9.2.24
Horde 5.2.17

I have tested on an iPhone with iOS 11.2.6 and a Jolla phone with 
Sailfish 2.1. On both phones, if I choose the setting to perform the 
search on the server, they just sit there saying "searching", and 
after a few minutes return no results.

I have logged in through telnet on the IMAP serrver and ran a SEARCH 
BODY command, which returns results. I have enabled full text search 
(FTS) Lucene indexing on the Dovecot side, and the searches now are 
down from 5+ minutes to less than 1 second. Still no difference on the 
phones.

I can see in the Horde ActiveSync debug logs that the search command 
is being sent to Horde by the phones (see below), but even with 
debugging turned all the way up in Dovecot, I don't see any search 
command being passed by Horde to Dovecot.

Here is what the phone sends to Horde when starting the server-side search:

[12466]INFO: ----------SEARCH request received for user 
myusername@mydomain.com
[12466][2018-04-16T15:27:38+01:00] INFO: ----------Request being 
handled for device: JP359745050349650, Supporting protocol version: 
14.0, Using Horde_ActiveSync v2.39.2
[12466] I: <Search:Search>
[12466] I:  <Search:Store>
[12466] I:   <Search:Name>
[12466] I:     Mailbox
[12466] I:   </Search:Name>
[12466] I:   <Search:Query>
[12466] I:    <Search:And>
[12466] I:     <Search:FreeText>
[12466] I:       dea
[12466] I:     </Search:FreeText>
[12466] I:     <FolderType>
[12466] I:       Email
[12466] I:     </FolderType>
[12466] I:    </Search:And>
[12466] I:   </Search:Query>
[12466] I:   <Search:Options>
[12466] I:    <Search:Range>
[12466] I:      0-100
[12466] I:    </Search:Range>
[12466] I:    <Search:DeepTraversal />
[12466] I:    <Search:RebuildResults />
[12466] I:    <AirSyncBase:BodyPreference>
[12466] I:     <AirSyncBase:Type>
[12466] I:       2
[12466] I:     </AirSyncBase:Type>
[12466] I:     <AirSyncBase:AllOrNone>
[12466] I:       1
[12466] I:     </AirSyncBase:AllOrNone>
[12466] I:     <AirSyncBase:Preview>
[12466] I:       255
[12466] I:     </AirSyncBase:Preview>
[12466] I:    </AirSyncBase:BodyPreference>
[12466] I:    <MIMESupport>
[12466] I:      0
[12466] I:    </MIMESupport>
[12466] I:   </Search:Options>
[12466] I:  </Search:Store>
[12466] I: </Search:Search>

Saved Queries