[#13554] String contains non-ASCII characters.
Summary String contains non-ASCII characters.
Queue Horde Framework Packages
Queue Version Git master
Type Bug
State Resolved
Priority 2. Medium
Owners slusarz@horde.org
Requester azurit@pobox.sk
Created 2014-09-08 (1897 days ago)
Due
Updated 2014-09-17 (1888 days ago)
Assigned 2014-09-09 (1896 days ago)
Resolved 2014-09-16 (1889 days ago)
Milestone
Patch No

Comments
azurit@pobox.sk 2014-09-08 12:50:52
While logging into IMP, this error is displayed and login is not possible:
String contains non-ASCII characters.

How to reproduce:
1.) Login into account.
2.) Create folder 'Odoslaná po?ta' (without quotes)
3.) Logout and try to login again.


Jan Schneider <jan@horde.org> 2014-09-08 16:56:35
Backtrace?

azurit@pobox.sk 2014-09-08 17:06:44
i can see only this in logs:
2014-09-04T18:13:37+02:00 EMERG: HORDE [imp] String contains non-ASCII 
characters. [pid 25850 on line 73 of 
"/usr/share/php/Horde/Imap/Client/Data/Format/String.php"]

do you need something more?


Jan Schneider <jan@horde.org> 2014-09-08 18:51:21
Yes, the backtrace. Enable debug logging.

azurit@pobox.sk 2014-09-08 19:04:26
this is it?



2014-09-08T21:01:48+02:00 EMERG: HORDE [horde] String contains 
non-ASCII characters. [pid 25718 on line 73 of 
"/usr/share/php/Horde/Imap/Client/Data/Format/String.php"]
2014-09-08T21:01:48+02:00 DEBUG: HORDE  1. 
Horde_Core_Auth_Application->authenticate() 
/usr/share/horde/login.php:155
  2. Horde_Core_Auth_Application->authenticate() 
/usr/share/php/Horde/Core/Auth/Application.php:126
  3. Horde_Auth_Base->authenticate() 
/usr/share/php/Horde/Core/Auth/Application.php:129
  4. Horde_Core_Auth_Application->_authenticate() 
/usr/share/php/Horde/Auth/Base.php:160
  5. Horde_Registry->callAppMethod() 
/usr/share/php/Horde/Core/Auth/Application.php:158
  6. call_user_func_array() /usr/share/php/Horde/Registry.php:1201
  7. IMP_Application->authAuthenticate()
  8. IMP_Auth::authenticate() /usr/share/horde/imp/lib/Application.php:371
  9. IMP_Imap->login() /usr/share/horde/imp/lib/Auth.php:86
10. IMP_Imap->__call() /usr/share/horde/imp/lib/Auth.php:86
11. call_user_func_array() /usr/share/horde/imp/lib/Imap.php:718
12. Horde_Imap_Client_Base->login()
13. Horde_Imap_Client_Socket->_login() 
/usr/share/php/Horde/Imap/Client/Base.php:794
14. Horde_Imap_Client_Socket->_tryLogin() 
/usr/share/php/Horde/Imap/Client/Socket.php:497
15. Horde_Imap_Client_Data_Format_String->__construct() 
/usr/share/php/Horde/Imap/Client/Socket.php:730


marth@tsvschlieben.de 2014-09-09 07:14:19
It seems, this commit breaks several other things too:

https://github.com/horde/horde/commit/10a822329f9fcaa8026de4c1e56de637554cf261

For exapmle if the Blacklist of an user contain non-ascii chars he's 
not able to access webmail anymore.

Jan Schneider <jan@horde.org> 2014-09-09 11:20:03
This doesn't have anything to do with non-ascii folder names, but you 
are using a password that contains non-ascii characters which is not 
supported.

Jan Schneider <jan@horde.org> 2014-09-09 11:20:46
This is in Horde_Imap_Client.

azurit@pobox.sk 2014-09-09 11:23:54
my password contains only ascii letters and digits

azurit@pobox.sk 2014-09-09 11:25:17
and i'm able to reproduce the problem exactly how i describe it in bug 
report. after i delete that folder (via shell), login start to work 
again.

Jan Schneider <jan@horde.org> 2014-09-09 12:11:28
Then you posted an incorrect, unrelated backtrace, because it 
explicitly points to what I said. Plus, I can reproduce this as soon 
as I provide a non-ascii password.

marth@tsvschlieben.de 2014-09-09 12:24:13
Hi Jan,

can you reproduce this cases too?

- folder name contain special chars
- filter contain special chars

Both cases weren't problematic before this commit (At least here). Now 
they brake the whole application.

We're using client side filtering (no sieve or similar). I'm not sure 
if the new sanitizing process is necessary in that case. From my view 
it should degrade graceful instead of just throwing an exception and 
leave the application in an useless state.


> Then you posted an incorrect, unrelated backtrace, because it 
> explicitly points to what I said. Plus, I can reproduce this as soon 
> as I provide a non-ascii password.


azurit@pobox.sk 2014-09-09 12:28:40
the mailing list is starting to be full of reports about this error 
and you are just saying there's no problem, how typical to Horde 
project..

Jan Schneider <jan@horde.org> 2014-09-09 12:38:21
> the mailing list is starting to be full of reports about this error 
> and you are just saying there's no problem, how typical to Horde 
> project..
No, but we're trying to be precise. And your report doesn't match what 
you said, and is indeed not a problem with Horde.
That doesn't mean that others may not weigh in with further reports, 
like it just happened.

azurit@pobox.sk 2014-09-09 12:41:57
if i get back to the password you mentioned - just login with random 
login and random password which will contain unicode chars and the 
whole horde crashes. i don't think this is correct behavior of any 
application.

Jan Schneider <jan@horde.org> 2014-09-09 12:46:53
> Hi Jan,
>
> can you reproduce this cases too?
>
> - folder name contain special chars

No.

> - filter contain special chars

Yes.

> Both cases weren't problematic before this commit (At least here). 
> Now they brake the whole application.
>
> We're using client side filtering (no sieve or similar). I'm not 
> sure if the new sanitizing process is necessary in that case. From 
> my view it should degrade graceful instead of just throwing an 
> exception and leave the application in an useless state.

Happing when executing IMAP filter on login was the crucial 
information. This needs to be investigated.

I don't see how this "leaves the application in an useless state" though.

azurit@pobox.sk 2014-09-09 13:31:18
> I don't see how this "leaves the application in an useless state" though.

for a common user, when application shows big error message and all 
control elements disappears, this is 'useless state' which he/she 
doesn't know how to correct and is usually contacting technical support


Michael Slusarz <slusarz@horde.org> 2014-09-16 07:51:20
> Then you posted an incorrect, unrelated backtrace, because it 
> explicitly points to what I said. Plus, I can reproduce this as soon 
> as I provide a non-ascii password.

This should be fixed to simply report a generic authentication 
failure.  Which should be done internally in Horde_Imap_Client.

Michael Slusarz <slusarz@horde.org> 2014-09-16 07:54:01
> Happing when executing IMAP filter on login was the crucial 
> information. This needs to be investigated.

Between this report, and a report in #13555, can I safely assume that 
everybody that has this problem is both:
- using imap filtering in ingo
- has a non-ascii character in one of their filter strings?

azurit@pobox.sk 2014-09-16 07:58:02
>> Happing when executing IMAP filter on login was the crucial
>> information. This needs to be investigated.
>
> Between this report, and a report in #13555, can I safely assume 
> that everybody that has this problem is both:
> - using imap filtering in ingo
> - has a non-ascii character in one of their filter strings?


One of our users experienced this problem with apostrophe (´) in one 
of the filtering rules - don't know if this is ascii or non-ascii char.


Michael Slusarz <slusarz@horde.org> 2014-09-16 08:15:10
> One of our users experienced this problem with apostrophe (´) in one 
> of the filtering rules - don't know if this is ascii or non-ascii 
> char.

That's not an apostrophe.  That's an acute diacritic and it is non-ASCII.

steven@tkfast.com 2014-09-16 15:43:07
The user I have that has this problem has a blacklist filter that has 
the email
noreply@körung.com
in the blacklist list that causes the error.

Git Commit <commits@lists.horde.org> 2014-09-16 20:38:29
Changes have been made in Git (FRAMEWORK_5_2):

commit bbdbdc665cb631104eaab305212384880d58b829
Author: Michael M Slusarz <slusarz@horde.org>
Date:   Tue Sep 16 14:36:50 2014 -0600

     [mms] IMAP driver needs to explicitly search for UTF-8 text, 
since we don't restrict charset input at the UI level (Bug #13554).

  ingo/docs/CHANGES        |    2 ++
  ingo/lib/Script/Imap.php |    5 +++++
  ingo/package.xml         |    2 ++
  3 files changed, 9 insertions(+), 0 deletions(-)

http://github.com/horde/horde/commit/bbdbdc665cb631104eaab305212384880d58b829

Michael Slusarz <slusarz@horde.org> 2014-09-16 20:44:06
Ingo 3.2.2

This is the best thing that could have happened.  IMAP driver was 
completely broken for non-ASCII searches in ingo since forever, so the 
recent change in Horde_Imap_Client easily/quickly identified that issue.

This still won't work as expected though.  e.g. blacklisting IDN 
addresses probably won't work due to the way IDN messages are 
sent/stored.  There's a bunch of recent RFCs involved with trying to 
handle this better and, in fact, I'm working with at least one Gmail 
guy right now in terms of trying to get more widespread support for 
non-ASCII data at both the client and server level.

Git Commit <commits@lists.horde.org> 2014-09-17 07:38:17
Changes have been made in Git (master):

commit ecea895075319793c034a30a625be1bc5db60a6c
Author: Michael M Slusarz <slusarz@horde.org>
Date:   Tue Sep 16 13:59:50 2014 -0600

     [mms] Better error handling when attempting to authenticate via 
IMAP LOGIN command with non-ASCII credentials (Bug #13554).

  .../Imap_Client/lib/Horde/Imap/Client/Socket.php   |   16 ++++++++++++++--
  framework/Imap_Client/package.xml                  |    2 ++
  2 files changed, 16 insertions(+), 2 deletions(-)

http://github.com/horde/horde/commit/ecea895075319793c034a30a625be1bc5db60a6c

Git Commit <commits@lists.horde.org> 2014-09-17 07:38:25
Changes have been made in Git (master):

commit 77acd38f939d3b1642ea72d210790466320f9a48
Author: Michael M Slusarz <slusarz@horde.org>
Date:   Tue Sep 16 14:36:50 2014 -0600

     [mms] IMAP driver needs to explicitly search for UTF-8 text, 
since we don't restrict charset input at the UI level (Bug #13554).

     Conflicts:
             ingo/docs/CHANGES
             ingo/package.xml

  ingo/docs/CHANGES        |    3 +++
  ingo/lib/Script/Imap.php |    5 +++++
  ingo/package.xml         |    3 ++-
  3 files changed, 10 insertions(+), 1 deletions(-)

http://github.com/horde/horde/commit/77acd38f939d3b1642ea72d210790466320f9a48