Check for ISO-8859-1/Windows-1252 improper charset labeling

Placing in IMP queue for now. Not sure if this is somethi

Wed, 25 Aug 2010 18:58:45 +0000

Placing in IMP queue for now. Not sure if this is something we should do in Horde_String or in IMP. Possible ideas: 1. Always treat ISO-8859-1 data as windows-1252 2. Look at X-mailer (or equivalent) and if it looks like Outlook, do #1. 3. For ISO-8859-1 parts, check for 0x80 to 0x9F characters and, if found, change charset representation to windows-1252

> 3. For ISO-8859-1 parts, check for 0x80 to 0x9F characters

Wed, 25 Aug 2010 19:34:34 +0000

> 3. For ISO-8859-1 parts, check for 0x80 to 0x9F characters and, if > found, change charset representation to windows-1252 This sounds like the most stable solution to me.

Changes have been made in Git for this ticket: Ticket #9201

Wed, 25 Aug 2010 19:36:58 +0000

Changes have been made in Git for this ticket: Ticket #9201: Better to convert things to UTF-8, to prevent lossy conversion. http://git.horde.org/diff.php/framework/Support/lib/Horde/Support/Domhtml.php?rt=horde-git&r1=6149c84e973f3fb5c61760834c148bae4cbf04b8&r2=699d059d4fa0faeed9273862ce3e19474bb8fd2d

This change seems to fix the conversion issues for me. Al

Wed, 25 Aug 2010 19:39:15 +0000

This change *seems* to fix the conversion issues for me. Although that could just be fortune that we are not doing any ISO-8859-1 conversions in the codepaths I have tested.

> This change seems to fix the conversion issues for me.

Thu, 26 Aug 2010 07:38:01 +0000

> This change *seems* to fix the conversion issues for me. Although > that could just be fortune that we are not doing any ISO-8859-1 > conversions in the codepaths I have tested. I still have problems with the euro sign. Using IMP : compose a new text message and set charset to UTF-8. Send the following string : "azerty " the received message looks fine in IMP, Thunderbird and Outlook. compose a new HTML message and set charset to UTF-8. Send the following string : "azerty " the HTML part looks ok and render just fine in Outlook and Thunderbird, but not IMP. I get this with IMP/FF : "azerty â¬" Looking at the source of the message, the encoded string doesn't look the same in the text part : --=_MgggdEx2InJNFCIGJXUlY0k0 Content-Type: text/plain; charset=UTF-8; format=flowed; DelSp=Yes Content-Description: Plaintext Version Content-Disposition: inline Content-Transfer-Encoding: quoted-printable azerty =C3=A2=C2=82=C2=AC --=_MgggdEx2InJNFCIGJXUlY0k0 Content-Type: text/html; charset=UTF-8 Content-Description: HTML Version Content-Disposition: inline Content-Transfer-Encoding: quoted-printable

azerty =E2=82=AC

--=_MgggdEx2InJNFCIGJXUlY0k0--

> compose a new HTML message and set charset to UTF-8. Send

Thu, 26 Aug 2010 07:52:54 +0000

> compose a new HTML message and set charset to UTF-8. Send the following string : "azerty " > the HTML part looks ok and render just fine in Outlook and > Thunderbird, but not IMP. I get this with IMP/FF : "azerty â¬" Actually, any accent aren't render correctly at all.

>> compose a new HTML message and set charset to UTF-8. Send

Thu, 26 Aug 2010 16:19:57 +0000

>> compose a new HTML message and set charset to UTF-8. Send the >> following string : "azerty " >> the HTML part looks ok and render just fine in Outlook and >> Thunderbird, but not IMP. I get this with IMP/FF : "azerty â¬" > > Actually, any accent aren't render correctly at all. Nope - works perfectly here. You are going to have to trace down where the text is being mangled on your system.

>> This change seems to fix the conversion issues for me.

Thu, 26 Aug 2010 16:53:10 +0000

>> This change *seems* to fix the conversion issues for me. Although >> that could just be fortune that we are not doing any ISO-8859-1 >> conversions in the codepaths I have tested. > > I still have problems with the euro sign. > > Using IMP : > compose a new text message and set charset to UTF-8. Send the following string : "azerty " > the received message looks fine in IMP, Thunderbird and Outlook. > > compose a new HTML message and set charset to UTF-8. Send the following string : "azerty " > the HTML part looks ok and render just fine in Outlook and > Thunderbird, but not IMP. I get this with IMP/FF : "azerty â¬" > > Looking at the source of the message, the encoded string doesn't look > the same in the text part : > > --=_MgggdEx2InJNFCIGJXUlY0k0 > Content-Type: text/plain; charset=UTF-8; format=flowed; DelSp=Yes > Content-Description: Plaintext Version > Content-Disposition: inline > Content-Transfer-Encoding: quoted-printable > > azerty =C3=A2=C2=82=C2=AC > > > --=_MgggdEx2InJNFCIGJXUlY0k0 > Content-Type: text/html; charset=UTF-8 > Content-Description: HTML Version > Content-Disposition: inline > Content-Transfer-Encoding: quoted-printable > > "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> > > > > > >

azerty =E2=82=AC

> > > --=_MgggdEx2InJNFCIGJXUlY0k0-- I see exactly the same behavior. And there are actually two errors. 1) the plain text is double encoded, i.e. the euro sign is turned into =C3=A2=C2=82=C2=AC while it's still correct in the html part (=E2=82=AC) 2) even though it's correct in the mail part, it's not displayed correctly, i.e. as "â¬"

> I see exactly the same behavior. And there are actually tw

Thu, 26 Aug 2010 17:56:17 +0000

> I see exactly the same behavior. And there are actually two errors. > 1) the plain text is double encoded, i.e. the euro sign is turned > into =C3=A2=C2=82=C2=AC while it's still correct in the html part > (=E2=82=AC) > 2) even though it's correct in the mail part, it's not displayed > correctly, i.e. as "â¬" Still works for me. Here's what my test message looks like. Sent via dimp/HTML compose: ---- Subject: Test Content-Type: multipart/alternative; boundary="=_GSd1CTVMdqosdDl3SYlomXUo" MIME-Version: 1.0 This message is in MIME format. --=_GSd1CTVMdqosdDl3SYlomXUo Content-Type: text/plain; charset=UTF-8; format=flowed; DelSp=Yes Content-Description: Plaintext Version Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Euro Character: =E2=82=AC =E2=82=AC --=_GSd1CTVMdqosdDl3SYlomXUo Content-Type: text/html; charset=UTF-8 Content-Description: HTML Version Content-Disposition: inline Content-Transfer-Encoding: quoted-printable

Euro Character: =E2=82=AC =E2=82=AC

--=_GSd1CTVMdqosdDl3SYlomXUo--

My guess is that there is something weird going on with the

Thu, 26 Aug 2010 18:03:10 +0000

My guess is that there is something weird going on with the DOM encoding/loading. It seems to be working perfect on my system - but that could be because I am using en_US.UTF-8. It might not be working properly on, e.g., de or fr locales. I would suggest playing around with charsets in Horde_Domhtml (located in the horde/Util package).

> My guess is that there is something weird going on with th

Thu, 26 Aug 2010 18:08:47 +0000

> My guess is that there is something weird going on with the DOM > encoding/loading. It seems to be working perfect on my system - but > that could be because I am using en_US.UTF-8. It might not be > working properly on, e.g., de or fr locales. > > I would suggest playing around with charsets in Horde_Domhtml > (located in the horde/Util package). For reference... when I view the HTML part in a new window, Horde_Domhtml is called once. The initial loadHTML() call fails as the encoding is not auto-determined. It then moves into the forced loadHTML() call after converting to UTF-8. The charset passed into the constructor is UTF-8. Pseudocode: public function __construct($text, 'UTF-8) { $doc = new DOMDocument(); $doc->loadHTML($text); // $doc->encoding is empty $this->encoding = $doc->encoding; if (!is_null($charset)) { if (!$doc->encoding) { $doc->loadHTML('' . Horde_String::convertCharset($text, $charset, 'UTF-8')); $this->encoding = 'UTF-8'; } } }

FWIW I sent the message with IMP, not DIMP.

Thu, 26 Aug 2010 20:31:38 +0000

FWIW I sent the message with IMP, not DIMP.

> FWIW I sent the message with IMP, not DIMP. No differen

Thu, 26 Aug 2010 20:43:41 +0000

> FWIW I sent the message with IMP, not DIMP. No difference for me - still works fine.

>> FWIW I sent the message with IMP, not DIMP. > > No diff

Fri, 27 Aug 2010 12:38:28 +0000

>> FWIW I sent the message with IMP, not DIMP. > > No difference for me - still works fine. Everything is fine now. I tried so many things that I'm not sure what did it for me, but my guess is the update of php from 5.3.2 to 5.3.3 Can this ticket be related to php bug #50661 ? I now run php-5.3.3-1.el5.remi on CentOS release 5.4

>>> FWIW I sent the message with IMP, not DIMP. >> >> No d

Sun, 29 Aug 2010 00:48:22 +0000

>>> FWIW I sent the message with IMP, not DIMP. >> >> No difference for me - still works fine. > > Everything is fine now. Jan - looks like you are the only one still seeing this. Have you tried upgrading PHP?

Changes have been made in Git for this ticket: Ticket #9201

Sun, 29 Aug 2010 00:51:09 +0000

Changes have been made in Git for this ticket: Ticket #9201: Treat ISO-8859-1 as windows-1252 http://git.horde.org/diff.php/imp/lib/Contents.php?rt=horde-git&r1=a2e63c9945413bc8d0487ded4a6f505ad6d20386&r2=74cc881c526c261d9acfc7ccfbaf3a4e7009141e

> Changes have been made in Git for this ticket: > > Ticke

Sun, 29 Aug 2010 00:53:34 +0000

> Changes have been made in Git for this ticket: > > Ticket #9201: Treat ISO-8859-1 as windows-1252 I've gone ahead and committed this - doing a bit of research, a bunch of mailers do a similar thing. I think that scanning for the unused 8859-1 codepoints is too much overhead.

Changes have been made in Git for this ticket: Ticket #9201

Mon, 30 Aug 2010 17:41:13 +0000

Changes have been made in Git for this ticket: Ticket #9201: part might not exist http://git.horde.org/diff.php/imp/lib/Contents.php?rt=horde-git&r1=74cc881c526c261d9acfc7ccfbaf3a4e7009141e&r2=fad70e02a52d0c9d80172c6c61b28f8765856e48

Things seem to be working well with this fix. Resolving tic

Thu, 21 Oct 2010 06:09:00 +0000

Things seem to be working well with this fix. Resolving ticket.