6.0.0-git
2019-03-19

[#11930] Horde_String::validUtf8 fails to validate valid UTF8
Summary Horde_String::validUtf8 fails to validate valid UTF8
Queue Horde Framework Packages
Queue Version Git master
Type Bug
State Resolved
Priority 1. Low
Owners slusarz (at) horde (dot) org
Requester samuel (at) sheepflock (dot) de
Created 2013-01-02 (2267 days ago)
Due
Updated 2013-01-09 (2260 days ago)
Assigned 2013-01-04 (2265 days ago)
Resolved 2013-01-09 (2260 days ago)
Milestone
Patch No

History
2013-01-09 22:40:02 Michael Rubinsky Comment #22 Reply to this comment
From what i have seen  if this function returns false  the code 
strips all non 7 bit characters
This works in english but strips the whole text in other 
languages.Maybe a less heavy handed approach is possible striping 
only offending characters and replacing with equal byte symbols by 
calling this logic .
This is only true for ActiveSync. In ActiveSync, the stripping of non 
7 bit characters is a last ditch effort when we can't determine what 
the encoding is. If we don't know what encoding the text is in, how 
are we to know what equal byte symbols are?

The reason we have to strip the non 7 bit characters is because if we 
send invalid UTF-8 data over wbxml, it can completely break the sync 
and even crash clients like iOS. Again, we only due this as a last 
ditch effort when the incoming email contains inproper character 
encoding information.
2013-01-09 21:17:24 Michael Slusarz Comment #21 Reply to this comment
a quick reference  i found without reading the whole paper is table 3.7 in
http://www.unicode.org/versions/Unicode6.0.0/ch03.pdf
a implementation of the table i crafted is  bellow
Uhh... that's what I already implemented several months ago.
2013-01-09 21:12:50 tassoskalyvas (at) gmail (dot) com Comment #20 Reply to this comment
From what i have seen  if this function returns false  the code 
strips all non 7 bit characters
This works in english but strips the whole text in other 
languages.Maybe a less heavy handed approach is possible striping only 
offending characters and replacing with equal byte symbols by calling 
this logic .
2013-01-09 21:01:58 tassoskalyvas (at) gmail (dot) com Comment #19 Reply to this comment
a quick reference  i found without reading the whole paper is table 3.7 in
http://www.unicode.org/versions/Unicode6.0.0/ch03.pdf
a implementation of the table i crafted is  bellow

   static public function validUtf8($text)
     {
     $text = strval($text);
        $len = strlen($text);
         for ($i = 0; $i < $len;  $i++) {
             $c = ord($text[$i]);
             if ($c >= 128) {
                 if ($c > 244) return false;
                         elseif ($c > 239)  {$bytes = 4;
                                 if ($c = 240)  {$c1 = ord($text[$i+1]);
                                         if (($c1 < 144)) return false;}
                                 if ($c = 244)  {$c1 = ord($text[$i+1]);
                                         if (($c1 > 144)) return false;}}
                         elseif ($c > 223)  {$bytes = 3;
                                                if ( $c = 237) {$c1 = ord($text[$i+1]);
                                                        if (($c1 > 159)) return false;}
                                                if ( $c = 224) {$c1 = ord($text[$i+1]);
                                                        if (($c1 < 160)) return false;}}
                         elseif ($c > 193)  $bytes = 2;
                                else  return false;
                 if (($i + $bytes) > $len)   return false;
                 while ($bytes >  1) {
                                    $i++;
                     $c = ord($text[$i]);
                     if (($c < 128) || ($c > 191))    return false;
                                        $bytes--;

                 }
             }
         }

         return true;
     }

Warning i am not a programmer
2013-01-09 14:52:38 Michael Rubinsky Comment #18 Reply to this comment
New Horde_Util package released.
2013-01-09 14:43:42 Michael Rubinsky Comment #17
Assigned to Michael Slusarz
State ⇒ Resolved
Reply to this comment
I can verify that commit e18b2c3595fbabda2cf9a3d4a348b1db837dda7a is 
what fixes this issue. Reverting that commit causes the test to fail 
for me and reapplying the commit causes it to pass.
2013-01-09 14:14:57 vilius (at) lnk (dot) lt Comment #16 Reply to this comment
To everyone struggling with this issue. Download Horde/String.php 
file from git and see if fixes the issue. The released package have 
an older one. Not sure why, but git version works without issues for 
me, but released version won't. Probably a bug somewhere in the 
validUtf8 logic.
This would also explain why Horde developers don't see any problems.
2013-01-09 14:14:08 vilius (at) lnk (dot) lt Comment #15 Reply to this comment
To everyone struggling with this issue. Download Horde/String.php file 
from git and see if fixes the issue. The released package have an 
older one. Not sure why, but git version works without issues for me, 
but released version won't. Probably a bug somewhere in the validUtf8 
logic.
2013-01-09 07:33:28 Michael Slusarz Deleted Original Message
 
2013-01-09 07:33:09 Michael Slusarz Deleted Original Message
 
2013-01-09 07:13:30 Michael Slusarz Comment #14 Reply to this comment
Somebody suggested that the 'mbstring.func_overload' php.ini setting 
might be at issue.  For obvious reasons, this needs to be 0.  If this 
is 1, all sorts of things are going to be broken in Horde.
2013-01-08 22:04:08 samuel (at) sheepflock (dot) de Comment #13 Reply to this comment
Sorry, I select the wrong version. :-(
I am not on git, only horde stable packages.


2013-01-08 21:35:13 samuel (at) sheepflock (dot) de Comment #12
New Attachment: boolean_result.png
Reply to this comment
$test = 'ö ä
Grüßen';
var_dump(Horde_String::validUtf8($test));
=> boolean false
--------------------------------------%<--------------------------------------
$test = 'ö ä
Grü ßen';
var_dump(Horde_String::validUtf8($test));
=> boolean true
--------------------------------------%<--------------------------------------
$test = 'öä
Grü ßen';
var_dump(Horde_String::validUtf8($test));
=> boolean false


root@wds:/usr/share/php/tests/Horde_Util/Horde/Util# phpunit StringTest.php
PHPUnit 3.7.10 by Sebastian Bergmann.

Configuration read from /usr/share/php/tests/Horde_Util/Horde/Util/phpunit.xml

.S.S..S.........

Time: 0 seconds, Memory: 3.50Mb

OK, but incomplete or skipped tests!
Tests: 16, Assertions: 90, Skipped: 3.
root@wds:/usr/share/php/tests/Horde_Util/Horde/Util#
2013-01-07 22:38:49 Jan Schneider Comment #11 Reply to this comment

[Show Quoted Text - 14 lines]
You didn't install Horde_Test, or your PEAR installation is not in 
your PHP include_path.
2013-01-07 15:56:07 boris (at) fouc (dot) de Comment #10 Reply to this comment
Hi,

I tried this on my installation to:
PHP Code (Horde PHP Shell):
$test = 'ö ä ü ß

Mit freundlichen Grüßen';

var_dump(Horde_String::validUtf8($test));
The Result: bool(false)


If I change this: Grüßen to Grü ßen

the result is true ?

Then I changed "ö ä" to "öä"  and it fails again.
Are  there two utf8 values not allowed after each-other ?








2013-01-06 10:01:30 samuel (at) sheepflock (dot) de Comment #9 Reply to this comment
In this case close the bug report, I use PHP 5.3.3 on debian squeeze.
This is not helpful.
I upgrade today to debian wheezy with php version 5.4.4-10, result is 
false as well but different format:
boolean false

I am not sure phpunit work in my setup, here is the result:

root@wds:/usr/share/php/tests/Horde_Util/Horde/Util# phpunit StringTest.php
PHP Warning:  require_once(Horde/Test/Bootstrap.php): failed to open 
stream: No such file or directory in 
/usr/share/php/tests/Horde_Util/Horde/Util/bootstrap.php on line 2
PHP Stack trace:
PHP   1. {main}() /usr/bin/phpunit:0
PHP   2. PHPUnit_TextUI_Command::main() /usr/bin/phpunit:46
PHP   3. PHPUnit_TextUI_Command->run() 
/usr/share/php/PHPUnit/TextUI/Command.php:130
PHP   4. PHPUnit_TextUI_Command->handleArguments() 
/usr/share/php/PHPUnit/TextUI/Command.php:139
PHP   5. PHPUnit_TextUI_Command->handleBootstrap() 
/usr/share/php/PHPUnit/TextUI/Command.php:620
PHP   6. PHPUnit_Util_Fileloader::checkAndLoad() 
/usr/share/php/PHPUnit/TextUI/Command.php:867
PHP   7. PHPUnit_Util_Fileloader::load() 
/usr/share/php/PHPUnit/Util/Fileloader.php:79
PHP   8. include_once() /usr/share/php/PHPUnit/Util/Fileloader.php:95
PHP Fatal error:  require_once(): Failed opening required 
'Horde/Test/Bootstrap.php' 
(include_path='.:/usr/share/php:/usr/share/pear') in 
/usr/share/php/tests/Horde_Util/Horde/Util/bootstrap.php on line 2
PHP Stack trace:
PHP   1. {main}() /usr/bin/phpunit:0
PHP   2. PHPUnit_TextUI_Command::main() /usr/bin/phpunit:46
PHP   3. PHPUnit_TextUI_Command->run() 
/usr/share/php/PHPUnit/TextUI/Command.php:130
PHP   4. PHPUnit_TextUI_Command->handleArguments() 
/usr/share/php/PHPUnit/TextUI/Command.php:139
PHP   5. PHPUnit_TextUI_Command->handleBootstrap() 
/usr/share/php/PHPUnit/TextUI/Command.php:620
PHP   6. PHPUnit_Util_Fileloader::checkAndLoad() 
/usr/share/php/PHPUnit/TextUI/Command.php:867
PHP   7. PHPUnit_Util_Fileloader::load() 
/usr/share/php/PHPUnit/Util/Fileloader.php:79
PHP   8. include_once() /usr/share/php/PHPUnit/Util/Fileloader.php:95
root@wds:/usr/share/php/tests/Horde_Util/Horde/Util#
2013-01-05 19:49:34 Michael Slusarz Comment #8 Reply to this comment
You also need to make sure you are running a somewhat recent version
of PHP.  Versions of PHP distributed via a package (i.e. Debian) is
not acceptable.
In this case close the bug report, I use PHP 5.3.3 on debian squeeze.
This is not helpful.

2013-01-05 18:52:39 samuel (at) sheepflock (dot) de Comment #7 Reply to this comment
You also need to make sure you are running a somewhat recent version 
of PHP.  Versions of PHP distributed via a package (i.e. Debian) is 
not acceptable.
In this case close the bug report, I use PHP 5.3.3 on debian squeeze.

2013-01-05 17:00:37 Michael Slusarz Deleted Original Message
 
2013-01-05 16:57:23 Michael Slusarz Comment #6 Reply to this comment
After updating with the latest horde package just now, my test still
also show fail.
Same here, after upgrade of:
This doesn't help at all.  You need to run the unit tests.  See: 
http://wiki.horde.org/Doc/Dev/TestH5#toc8

You also need to make sure you are running a somewhat recent version 
of PHP.  Versions of PHP distributed via a package (i.e. Debian) is 
not acceptable.
2013-01-05 14:41:24 samuel (at) sheepflock (dot) de Comment #5 Reply to this comment
After updating with the latest horde package just now, my test still 
also show fail.
Same here, after upgrade of:
upgrade-all ok: channel://pear.horde.org/Horde_Mime-2.0.2
upgrade-all ok: channel://pear.horde.org/Horde_Imap_Client-2.4.2
upgrade-all ok: channel://pear.horde.org/Horde_Core-2.1.4
upgrade-all ok: channel://pear.horde.org/Horde_ActiveSync-2.1.1

--> bool(false)
2013-01-05 04:36:00 busywater (at) gmail (dot) com Comment #4
New Attachment: Screen Shot 2013-01-05 at 12.33.52 PM.png
Reply to this comment
After updating with the latest horde package just now, my test still 
also show fail.

Even quoting (single and double) the string, it still fails.

Kinglok, Fong

[Show Quoted Text - 15 lines]
2013-01-04 21:11:40 Michael Slusarz Comment #3
State ⇒ Feedback
Reply to this comment
Cannot reproduce.  Unit test added with this test string and it 
validates correctly:

slusarz@bigworm % phpunit StringTest.php
PHPUnit 3.7.10 by Sebastian Bergmann.

Configuration read from 
/disk2/src/horde/framework/Util/test/Horde/Util/phpunit.xml

.S.S..S.........

Time: 0 seconds, Memory: 5.50Mb

OK, but incomplete or skipped tests!
Tests: 16, Assertions: 91, Skipped: 3.
2013-01-04 21:11:19 Git Commit Comment #2 Reply to this comment
Changes have been made in Git (master):

commit c29e62131c9582c59890a8031f909be7d8e4ccbb
Author: Michael M Slusarz <slusarz@horde.org>
Date:   Fri Jan 4 14:09:43 2013 -0700

     Validation test for Bug #11930

  framework/Util/test/Horde/Util/StringTest.php |    3 ++-
  1 files changed, 2 insertions(+), 1 deletions(-)

http://git.horde.org/horde-git/-/commit/c29e62131c9582c59890a8031f909be7d8e4ccbb
2013-01-02 19:45:11 samuel (at) sheepflock (dot) de Comment #1
Type ⇒ Bug
State ⇒ Unconfirmed
Priority ⇒ 1. Low
Summary ⇒ Horde_String::validUtf8 fails to validate valid UTF8
Queue ⇒ Horde Framework Packages
Milestone ⇒
Patch ⇒ No
New Attachment: horde_php_shell_utf8.png
Reply to this comment
PHP Code (Horde PHP Shell):
$test = 'ö ä ü ß

Mit freundlichen Grüßen';

var_dump(Horde_String::validUtf8($test));

Result:
bool(false)

Debian Squeeze Server with php5  5.3.3-7+squeeze14

INSTALLED PACKAGES, CHANNEL PEAR.HORDE.ORG:
===========================================
PACKAGE                   VERSION    STATE
Horde_ActiveSync          2.0.14     stable
Horde_Alarm               2.0.2      stable
Horde_Argv                2.0.2      stable
Horde_Auth                2.0.1      stable
Horde_Autoloader          2.0.1      stable
Horde_Browser             2.0.2      stable
Horde_Cache               2.0.1      stable
Horde_Cli                 2.0.1      stable
Horde_Compress            2.0.1      stable
Horde_Constraint          2.0.1      stable
Horde_Controller          2.0.1      stable
Horde_Core                2.1.3      stable
Horde_Crypt               2.1.0      stable
Horde_Crypt_Blowfish      1.0.1      stable
Horde_Data                2.0.1      stable
Horde_Date                2.0.1      stable
Horde_Date_Parser         2.0.1      stable
Horde_Db                  2.0.1      stable
Horde_Editor              2.0.1      stable
Horde_ElasticSearch       1.0.1      stable
Horde_Exception           2.0.1      stable
Horde_Feed                2.0.1      stable
Horde_Form                2.0.1      stable
Horde_Group               2.0.1      stable
Horde_History             2.0.1      stable
Horde_Http                2.0.1      stable
Horde_Icalendar           2.0.1      stable
Horde_Image               2.0.1      stable
Horde_Imap_Client         2.4.1      stable
Horde_Imsp                2.0.1      stable
Horde_Injector            2.0.1      stable
Horde_Itip                2.0.1      stable
Horde_Kolab_Format        2.0.1      stable
Horde_Kolab_Server        2.0.1      stable
Horde_Kolab_Session       2.0.1      stable
Horde_Kolab_Storage       2.0.2      stable
Horde_ListHeaders         1.0.1      stable
Horde_Lock                2.0.1      stable
Horde_Log                 2.0.1      stable
Horde_LoginTasks          2.0.1      stable
Horde_Mail                2.0.3      stable
Horde_Memcache            2.0.1      stable
Horde_Mime                2.0.1      stable
Horde_Mime_Viewer         2.0.1      stable
Horde_Nls                 2.0.1      stable
Horde_Notification        2.0.1      stable
Horde_Oauth               2.0.1      stable
Horde_Perms               2.0.1      stable
Horde_Prefs               2.0.1      stable
Horde_Rdo                 2.0.1      stable
Horde_Role                1.0.1      stable
Horde_Routes              2.0.1      stable
Horde_Rpc                 2.0.2      stable
Horde_Scribe              2.0.1      stable
Horde_Secret              2.0.2      stable
Horde_Serialize           2.0.1      stable
Horde_Service_Facebook    2.0.1      stable
Horde_Service_Twitter     2.0.1      stable
Horde_Service_Weather     2.0.1      stable
Horde_SessionHandler      2.0.1      stable
Horde_Share               2.0.1      stable
Horde_SpellChecker        2.0.1      stable
Horde_Stream              1.2.0      stable
Horde_Stream_Filter       2.0.1      stable
Horde_Stream_Wrapper      2.0.1      stable
Horde_Support             2.0.2      stable
Horde_SyncMl              2.0.1      stable
Horde_Template            2.0.1      stable
Horde_Text_Diff           2.0.1      stable
Horde_Text_Filter         2.0.3      stable
Horde_Text_Filter_Csstidy 2.0.1      stable
Horde_Text_Flowed         2.0.1      stable
Horde_Thrift              2.0.1      stable
Horde_Timezone            1.0.1      stable
Horde_Token               2.0.1      stable
Horde_Translation         2.0.1      stable
Horde_Tree                2.0.1      stable
Horde_Url                 2.0.1      stable
Horde_Util                2.0.2      stable
Horde_Vfs                 2.0.3      stable
Horde_View                2.0.1      stable
Horde_Xml_Element         2.0.1      stable
Horde_Xml_Wbxml           2.0.1      stable
content                   2.0.1      stable
horde                     5.0.2      stable
imp                       6.0.2      stable
ingo                      3.0.1      stable
kronolith                 4.0.2      stable
mnemo                     4.0.1      stable
nag                       4.0.1      stable
trean                     1.0.0beta2 beta
turba                     4.0.1      stable

Saved Queries