charset pb replying to message

using dynamic view compose_html=1 reply_format=1 attach

Wed, 09 Feb 2011 14:47:50 +0000

using dynamic view compose_html=1 reply_format=1 attached is the message from ticket #9189 and #9190. replying to this message gives : "prÃ©parer Ã vendre dâaoÃ»tÂ ; " expected : "préparer à vendre d'août ;" Funny thing is (this probably is related to ticket #9549) : using dynamic view, - select message and reply - click on "HTML composition" (do not click or modify the body of the message) output :

ronan@maison.com a écrit :

> Bonjour,
>
>
>
> préparer à vendre d?août ;

the output is html source, but the text (accents) looks ok.

Duplicate of Bug #9549.

Wed, 09 Feb 2011 19:12:50 +0000

Duplicate of Bug #9549.

> Duplicate of Bug #9549. I agree that my first comment c

Thu, 10 Feb 2011 08:08:47 +0000

> Duplicate of Bug #9549. I agree that my first comment can be a bit confusing, and that the part with "HTML composition" is can be a duplicate of Bug #9549. But the first part is not a duplicate of Bug #9549. When I reply to the attached message, accents are showing OK.

>> Duplicate of Bug #9549. > > I agree that my first comme

Fri, 11 Feb 2011 09:12:13 +0000

>> Duplicate of Bug #9549. > > I agree that my first comment can be a bit confusing, and that the > part with "HTML composition" is can be a duplicate of Bug #9549. > > But the first part is not a duplicate of Bug #9549. When I reply to > the attached message, accents are showing OK. So there is no longer an issue, correct? I can't reproduce.

>> But the first part is not a duplicate of Bug #9549. When

Fri, 11 Feb 2011 14:03:46 +0000

>> But the first part is not a duplicate of Bug #9549. When I reply to >> the attached message, accents are showing OK. > > So there is no longer an issue, correct? I can't reproduce. Arrgh, really bad week. I meant "accents are not showing OK". I can't debug today. I updated from git this morning and I can't use dynamic imp now (see below). I have either missed something or there's something wrong in git repo. I'll wait monday... A fatal error has occurred "Array" is not configured in the Horde Registry. 1. require() /var/www/html/hordetest/imp/index.php:19 2. IMP_Dimp::header() /var/www/html/hordetest/imp/index-dimp.php:38 3. include() /var/www/html/hordetest/imp/lib/Dimp.php:78 4. include() /var/www/html/hordetest/imp/templates/common-header.inc:9 5. Horde_Registry->getInitialPage() /var/www/html/hordetest/imp/templates/dimp/javascript_defs.php:22

> A fatal error has occurred > "Array" is not configured in

Fri, 11 Feb 2011 17:32:56 +0000

> A fatal error has occurred > "Array" is not configured in the Horde Registry. You did not update imp/config/portal.php for changes made last night.

Restarting from scratch to explain the issue I'm having when

Mon, 28 Feb 2011 08:43:57 +0000

Restarting from scratch to explain the issue I'm having when replying to messages. using the message attached to this ticket, traditional view mode, - $_prefs['compose_html']['value'] = 0; - $_prefs['reply_format']['value'] = 0; => reply Ok - $_prefs['compose_html']['value'] = 1; - $_prefs['reply_format']['value'] = 1; => reply *NOK* see attached screenshot, accents look like rubbish.

Still working perfectly here.

Thu, 03 Mar 2011 23:00:04 +0000

Still working perfectly here.

> Still working perfectly here. body (variable $msg) look

Fri, 04 Mar 2011 16:09:30 +0000

> Still working perfectly here. body (variable $msg) looks fine up to line 2504 in imp/lib/Compose.php if ($mode == 'html') { $msg = $GLOBALS['injector']->getInstance('Horde_Core_Factory_TextFilter')->filter($msg, array('Cleanhtml', 'Xss'), array(array('body_only' => true), array('strip_styles' => true, 'strip_style_attributes' => false))); } elseif ($type == 'text/html') { $msg = $GLOBALS['injector']->getInstance('Horde_Core_Factory_TextFilter')->filter($msg, 'Html2text'); $type = 'text/plain'; } but after line 2511, msg looks wrong (accent are screwed). If I change array('Cleanhtml', 'Xss') to array(), accents look OK (but the reply message is a bit screwed :-)).

I got further. problem is coming from Domhtml.php. DOMDoc

Fri, 04 Mar 2011 16:37:35 +0000

I got further. problem is coming from Domhtml.php. DOMDocument thinks that $text is iso-8859-1, but it is UTF-8 as it has been converted earlier on. The text message gets screwed after the following call (line 83) : $doc->loadHTML(Horde_String::convertCharset($text, $charset, $doc->encoding)); $charset = utf-8 $doc->encoding = iso-8859-1 I don't know what to do from there. Anyway I can help ? I use: Mozilla/5.0 (X11; U; Linux i686; fr; rv:1.9.2.13) Gecko/20110103 Fedora/3.6.13-1.fc14 Firefox/3.6.13 php-5.3.5-1.el5.remi.1

> I got further. > > problem is coming from Domhtml.php. D

Fri, 04 Mar 2011 17:59:13 +0000

> I got further. > > problem is coming from Domhtml.php. DOMDocument thinks that $text is > iso-8859-1, but it is UTF-8 as it has been converted earlier on. > > The text message gets screwed after the following call (line 83) : > $doc->loadHTML(Horde_String::convertCharset($text, > $charset, $doc->encoding)); > > $charset = utf-8 > $doc->encoding = iso-8859-1 So what you are saying is that BEFORE line 83, $charset is utf-8 and $doc->encoding is iso-8859-1? If that is the case, I don't see why this isn't working... we are converting $text to ISO-8859-1 (from UTF-8) and then sending to loadHTML. So things should be fine. Maybe check what the value of $doc->encoding is AFTER line 83? Or try creating a new DOMDocument object - e.g.: /* If libxml can't auto-detect encoding, convert to what it * *thinks* the encoding should be. */ $doc = new DOMDocument(); $doc->loadHTML(Horde_String::convertCharset($text, $charset, $doc->encoding));

> So what you are saying is that BEFORE line 83, $charset is

Mon, 07 Mar 2011 09:22:05 +0000

> So what you are saying is that BEFORE line 83, $charset is utf-8 and > $doc->encoding is iso-8859-1? If that is the case, I don't see why > this isn't working... we are converting $text to ISO-8859-1 (from > UTF-8) and then sending to loadHTML. So things should be fine. I got mislead with editors charset. Depending on which one I was using (vi, nedit) I wasn't getting (seeing) the same output, and I just realised that now. So, this got me up to framework/Text_Filter/lib/Horde/Text/Filter/Xss.php I've attached the log patch and horde log file. The log file is trace of replying to the message. It looks like dom->saveXML returns UTF-8 characters. If I change the last 'return' of function postProcess($text) like this - return Horde_String::convertCharset($text, $dom->encoding, $this->_params['charset']); + return $text; Then accents look Ok!

I've attached the wrong file in the last comment.

Mon, 07 Mar 2011 09:37:30 +0000

I've attached the wrong file in the last comment.

with the patch attached (and using the message originally at

Wed, 09 Mar 2011 08:46:06 +0000

with the patch attached (and using the message originally attached to this ticket), here is the output of charset detection : 2011-03-09T09:33:34+01:00 INFO: HORDE [imp] 777777777777777777ASCII [pid 15410 on line 130 of "/var/www/html/hordetest/libs/Horde/Text/Filter/Xss.php"] 2011-03-09T09:33:34+01:00 INFO: HORDE [imp] 777777777777777777UTF-8 [pid 15410 on line 130 of "/var/www/html/hordetest/libs/Horde/Text/Filter/Xss.php"] 2011-03-09T09:33:34+01:00 INFO: HORDE [imp] 777777777777777777ASCII [pid 15410 on line 130 of "/var/www/html/hordetest/libs/Horde/Text/Filter/Xss.php"] 2011-03-09T09:33:34+01:00 INFO: HORDE [imp] 777777777777777777UTF-8 [pid 15410 on line 133 of "/var/www/html/hordetest/libs/Horde/Text/Filter/Xss.php"] 2011-03-09T09:33:34+01:00 INFO: HORDE [imp] 777777777777777777iso-8859-1 [pid 15410 on line 134 of "/var/www/html/hordetest/libs/Horde/Text/Filter/Xss.php"] So, this confirm that dom->saveXML returns UTF-8 characters, but $doc->encoding is iso-8859-1. I'm having this issue not only with the attached message, but pretty much with all messages in my inbox (as a matter of fact all messages containing accents). Just in case this was related to libxml, I've update the lib to libxml2-2.7.8

Changes have been made in Git for this ticket: Add test for

Thu, 10 Mar 2011 20:24:25 +0000

Changes have been made in Git for this ticket: Add test for Bug #9567 1 files changed, 0 insertions(+), 0 deletions(-) http://git.horde.org/horde-git/-/commit/9bc38e1452ccb11d2c709175b65d705243df1ad0

I have added a test case in Text_Filter that passes for me.

Thu, 10 Mar 2011 20:26:10 +0000

I have added a test case in Text_Filter that passes for me. See if it passes for you. Easiest way to run is to go to horde/framework/Text_Filter/test/Horde/Text/Filter and run 'php AllTests.php'

> I have added a test case in Text_Filter that passes for me

Sat, 12 Mar 2011 16:24:26 +0000

> I have added a test case in Text_Filter that passes for me. See if > it passes for you. after installing PHPUnit (thanks to remi's repo) I get 2 failures running the test, but not related to this bug I guess. see attatched log file. I can provide you another message as an example if you want, but I can't attach it to this ticket as it is a non public message. I tried to remove private information, but no matter what editor I was using, I always ended up altering the charset of the message when saving.

>> I have added a test case in Text_Filter that passes for m

Tue, 15 Mar 2011 03:06:08 +0000

>> I have added a test case in Text_Filter that passes for me. See if >> it passes for you. > > after installing PHPUnit (thanks to remi's repo) I get 2 failures > running the test, but not related to this bug I guess. Yes - those are not related to this bug. A user had submitted this patch awhile back. Maybe this fixes things for you?

Patch originally appeared in this message: http://lists.h

Tue, 15 Mar 2011 03:06:51 +0000

Patch originally appeared in this message: http://lists.horde.org/archives/dev/Week-of-Mon-20101115/025488.html

> A user had submitted this patch awhile back. Maybe this f

Tue, 15 Mar 2011 09:54:46 +0000

> A user had submitted this patch awhile back. Maybe this fixes things > for you? Nope, it does fix anything, although I use the same OS Centos 5.4, but probably not the same libxml/php version Googling a bit, I ran into this article http://devzone.zend.com/article/8855, 5. Save/dumping operations and encoding : "Node or XML subtree dumping using the DOMDocument::saveXML($node) method is always performed in UTF-8." This is the issue I'm having, $dom->encoding = iso-8859-1 and $dom->dom->saveXML($child) returns utf-8. The following patch works for me for all messages read, reply, forward... (for whatever I've tested so far) : --- Xss.php.org 2011-03-15 10:41:22.000000000 +0100 +++ Xss.php 2011-03-15 10:41:24.000000000 +0100 @@ -130,7 +130,7 @@ } } - return Horde_String::convertCharset($text, $dom->encoding, $this->_params['charset']); + return $text; } /**

> --- Xss.php.org 2011-03-15 10:41:22.000000000 +0100 > +++

Tue, 15 Mar 2011 10:15:13 +0000

> --- Xss.php.org 2011-03-15 10:41:22.000000000 +0100 > +++ Xss.php 2011-03-15 10:41:24.000000000 +0100 > - return Horde_String::convertCharset($text, $dom->encoding, $this->_params['charset']); > + return $text; with this patch, testBug9567 fails. I've attached the output.

>> --- Xss.php.org 2011-03-15 10:41:22.000000000 +0100 >> +

Tue, 15 Mar 2011 16:56:30 +0000

>> --- Xss.php.org 2011-03-15 10:41:22.000000000 +0100 >> +++ Xss.php 2011-03-15 10:41:24.000000000 +0100 >> - return Horde_String::convertCharset($text, $dom->encoding, >> $this->_params['charset']); >> + return $text; > > with this patch, testBug9567 fails. > I've attached the output. OK - I am *totally confused*. The test runs successfully. Why should we be patching? Of course the test is going to fail - you are altering the output for a successful test.

> --- Xss.php.org 2011-03-15 10:41:22.000000000 +0100 > +++

Tue, 15 Mar 2011 17:04:01 +0000

> --- Xss.php.org 2011-03-15 10:41:22.000000000 +0100 > +++ Xss.php 2011-03-15 10:41:24.000000000 +0100 > @@ -130,7 +130,7 @@ > } > } > > - return Horde_String::convertCharset($text, $dom->encoding, > $this->_params['charset']); > + return $text; > } This isn't correct. Xss filter needs to return text in whatever charset it was provided in, which is why the convertCharset() call is necessary. The question is why $dom->encoding is 'ISO-8859-1' for you and 'UTF-8' for *everybody* else.

> http://devzone.zend.com/article/8855 This may be useful

Tue, 15 Mar 2011 17:12:46 +0000

> http://devzone.zend.com/article/8855 This may be useful, specifically Section 4 about loading/encoding. This is the starting point in trying to figure this out.

>>> --- Xss.php.org 2011-03-15 10:41:22.000000000 +0100 >>>

Tue, 15 Mar 2011 22:06:58 +0000

>>> --- Xss.php.org 2011-03-15 10:41:22.000000000 +0100 >>> +++ Xss.php 2011-03-15 10:41:24.000000000 +0100 >>> - return Horde_String::convertCharset($text, $dom->encoding, >>> $this->_params['charset']); >>> + return $text; >> >> with this patch, testBug9567 fails. >> I've attached the output. > > OK - I am *totally confused*. The test runs successfully. Why > should we be patching? Of course the test is going to fail - you are > altering the output for a successful test. I'm lost as well, and desperately trying to find a way out of this. Forget about this *patch*

> This isn't correct. Xss filter needs to return text in wh

Tue, 15 Mar 2011 22:12:21 +0000

> This isn't correct. Xss filter needs to return text in whatever > charset it was provided in, which is why the convertCharset() call is > necessary. The question is why $dom->encoding is 'ISO-8859-1' for > you and 'UTF-8' for *everybody* else. Hmm wait, weird, the message attached to this ticket looks wrong. re-attaching the message. To the question as why $dom->encoding is 'ISO-8859-1', the answer is in the message : encoding = ISO-8859-15 If I remove the meta tag from the message, everything works fine. I've checked other messages I'm having issue with, and they all have the same charset meta tag So this behaviour is expected according to http://devzone.zend.com/article/8855, Section 4. I don't understand why this doesn't work here and works for you. What version of libxml2 and php are you using just in case this is related to the version I'm using. As I'm using the latest (or close enough), I'll try to downgrade to whatever version you are using.

> To the question as why $dom->encoding is 'ISO-8859-1', the

Tue, 15 Mar 2011 22:41:22 +0000

> To the question as why $dom->encoding is 'ISO-8859-1', the answer is > in the message : >

> I don't understand why this doesn't work here and works fo

Tue, 15 Mar 2011 23:01:39 +0000

> I don't understand why this doesn't work here and works for you. What > version of libxml2 and php are you using just in case this is related > to the version I'm using. As I'm using the latest (or close enough), > I'll try to downgrade to whatever version you are using. PHP 5.3.4 (cli) (built: Feb 12 2011 00:26:56) libxml2 Version => 2.7.8

>> To the question as why $dom->encoding is 'ISO-8859-1', th

Wed, 16 Mar 2011 07:50:26 +0000

>> To the question as why $dom->encoding is 'ISO-8859-1', the answer is >> in the message : >> > Maybe one problem is that this message is NOT iso-8859-1; it is windows-1252. If I change encoding = windows-1252 which is I my case expected and the same result as before : rubbish.

>> I don't understand why this doesn't work here and works f

Wed, 16 Mar 2011 15:50:12 +0000

>> I don't understand why this doesn't work here and works for you. What >> version of libxml2 and php are you using just in case this is related >> to the version I'm using. As I'm using the latest (or close enough), >> I'll try to downgrade to whatever version you are using. > > PHP 5.3.4 (cli) (built: Feb 12 2011 00:26:56) > libxml2 Version => 2.7.8 I recompiled both libxml2 and php, still having the same issue :-(

>> --- Xss.php.org 2011-03-15 10:41:22.000000000 +0100 >> +

Wed, 16 Mar 2011 16:41:57 +0000

>> --- Xss.php.org 2011-03-15 10:41:22.000000000 +0100 >> +++ Xss.php 2011-03-15 10:41:24.000000000 +0100 >> - return Horde_String::convertCharset($text, $dom->encoding, >> $this->_params['charset']); >> + return $text; > This isn't correct. Xss filter needs to return text in whatever > charset it was provided in, which is why the convertCharset() call is > necessary. The question is why $dom->encoding is 'ISO-8859-1' for > you and 'UTF-8' for *everybody* else. If we assume that information from this link http://devzone.zend.com/article/8855 are right, specifically section 5 : "DOMDocument::saveXML($node) method is always performed in UTF-8" Then, no matter what $doc->encoding is set to, the following code will *always* return a UTF-8 encoded string : if ($body && $body->hasChildNodes()) { foreach ($body->childNodes as $child) { $text .= $dom->dom->saveXML($child); } } So, I think that Horde_Text_Filter_Xss::postProcess should be patch like this : - return Horde_String::convertCharset($text, $dom->encoding, $this->_params['charset']); + return Horde_String::convertCharset($text, 'UTF-8', $this->_params['charset']); Now, why $dom->encoding is different on my machine than yours, I don't have the answer (and I tried a lot of things). but according to http://devzone.zend.com/article/8855, Section 4, DOMDocument::loadHTML() should detect meta tag 'charset', and on my system, it does (I guess) and this should explain as why $dom->encoding=iso-8859-1 (or whatever charset the meta tag is set to, see other comments). As I think that the above small patch is right, I don't mind if some of the other dev can try to reply to the message 'email_charset.eml' to see if I'm really alone on this one. Thanks.

> As I think that the above small patch is right, I don't mi

Wed, 16 Mar 2011 16:53:29 +0000

> As I think that the above small patch is right, I don't mind if some > of the other dev can try to reply to the message 'email_charset.eml' > to see if I'm really alone on this one. I didn't follow the complete ticket history, but I don't see anything wrong when replying to this message in traditional mode, using the HTML editor.

This is driving mad. For the last 20 minutes, I can't rep

Wed, 16 Mar 2011 17:28:45 +0000

This is driving mad. For the last 20 minutes, I can't reproduce this issue any more. But I don't know why, I don't know what I did or changed :-( Anyway, since I can't reproduce this issue any more, I guess there is no need to keep this ticket open. But I think the patch from comment #31 still applies. Thanks for you patience.

I still can't reproduce the behaviour I was having using IMP

Thu, 17 Mar 2011 08:29:11 +0000

I still can't reproduce the behaviour I was having using IMP, but I found a way to reproduce using the test case in Text_Filter you originally created. See attached file, I think that this is how the test case in Text_Filter should work. The test passes OK with the patch from comment #31. What do you think ?

Changes have been made in Git for this ticket: Add another

Thu, 17 Mar 2011 17:08:41 +0000

Changes have been made in Git for this ticket: Add another test for Bug #9567 1 files changed, 0 insertions(+), 0 deletions(-) http://git.horde.org/horde-git/-/commit/423b29e9ae9e6b4227f7d8870aa8a2b6d823f120

> Changes have been made in Git for this ticket: > > Add a

Thu, 17 Mar 2011 17:19:19 +0000

> Changes have been made in Git for this ticket: > > Add another test for Bug #9567 typos : it is content="text/html; charset=iso-8859-1" instead of content="text/html; charset="iso8859-1"

> See attached file, I think that this is how the test case

Thu, 17 Mar 2011 17:24:19 +0000

> See attached file, I think that this is how the test case in > Text_Filter should work. Nope. Works perfect for me.

> typos : it is > content="text/html; charset=iso-8859-1" i

Thu, 17 Mar 2011 17:26:56 +0000

> typos : it is > content="text/html; charset=iso-8859-1" instead of > content="text/html; charset="iso8859-1" This shouldn't (and doesn't) make a difference.

Changes have been made in Git for this ticket: Bug #9567: I

Thu, 17 Mar 2011 18:34:32 +0000

Changes have been made in Git for this ticket: Bug #9567: Improve loading of HTML documents 6 files changed, 123 insertions(+), 46 deletions(-) http://git.horde.org/horde-git/-/commit/cf8cb46f13765b88342477735f7bbb473727fffd

Rewrote HTML document loading to better match what the URL b

Thu, 17 Mar 2011 18:35:07 +0000

Rewrote HTML document loading to better match what the URL below describes. See if this helps.

Changes have been made in Git for this ticket: Bug #9567: X

Thu, 17 Mar 2011 21:29:08 +0000

Changes have been made in Git for this ticket: Bug #9567: XML encoding tag may not appear at beginning of output 1 files changed, 6 insertions(+), 3 deletions(-) http://git.horde.org/horde-git/-/commit/ba922deff269a9a6a35427610ddb6eb3adf9282f

> Rewrote HTML document loading to better match what the URL

Fri, 18 Mar 2011 08:21:43 +0000

> Rewrote HTML document loading to better match what the URL below > describes. See if this helps. Replying seems to be working just fine. I tried with a few messages and they all looked OK. I ran the new test case, and it fails. See attached log file.

> I ran the new test case, and it fails. See attached log fi

Fri, 18 Mar 2011 09:27:34 +0000

> I ran the new test case, and it fails. See attached log file. I add a problem pulling up to date code from git, and I didn't notice until now. The test passes OK.

>> I ran the new test case, and it fails. See attached log f

Fri, 18 Mar 2011 09:30:17 +0000

>> I ran the new test case, and it fails. See attached log file. > I add a problem pulling up to date code from git, and I didn't notice > until now. I meant, I *had* a problem! Everything looks good now.