Tickets :: [#14841] Send mail over ActiveSync - lose content

6.0.0-alpha10

5/15/25

Summary	Send mail over ActiveSync - lose content
Queue	Synchronization
Queue Version	FRAMEWORK_5_2
Type	Bug
State	Resolved
Priority	1. Low
Owners	mrubinsk (at) horde (dot) org
Requester	samuel.wolf (at) wolf-maschinenbau (dot) de
Created	07/27/2018 (2484 days ago)
Due
Updated	03/17/2019 (2251 days ago)
Assigned	07/29/2018 (2482 days ago)
Resolved	08/11/2018 (2469 days ago)
Milestone
Patch	No

03/17/2019 06:10:46 PM	samuel (dot) wolf (at) wolf-maschinenbau (dot) de	Comment #36	Reply to this comment
Please let me know if this solves the problem on your end, as it solved the test cases on my end. Yes this solved my problem, awesome! Thank you very much.

03/17/2019 02:45:23 AM	Git Commit	Comment #35	Reply to this comment
Changes have been made in Git (master): commit 11cea561c9d92b2316d43527ee69136543186f1e Author: Michael J Rubinsky <mrubinsk@horde.org> Date: Sat, 16 Mar 2019 22:45:16 -0400 [mjr] Fix issue causing the loss of reply text in SMARTREPLY requests (~~Bug #14841~~). M doc/Horde/ActiveSync/changelog.yml https://github.com/horde/ActiveSync/commit/11cea561c9d92b2316d43527ee69136543186f1e

03/17/2019 02:43:11 AM	Git Commit	Comment #34	Reply to this comment
Changes have been made in Git (FRAMEWORK_5_2): commit 68847766db72772ebf3c362d84ca9af21943b70e Author: Michael J Rubinsky <mrubinsk@horde.org> Date: Sat, 16 Mar 2019 22:43:00 -0400 [mjr] Fix issue causing the loss of reply text in SMARTREPLY requests (~~Bug #14841~~). M doc/Horde/ActiveSync/CHANGES M package.xml https://github.com/horde/ActiveSync/commit/68847766db72772ebf3c362d84ca9af21943b70e

03/17/2019 02:43:10 AM	Git Commit	Comment #33	Reply to this comment
Changes have been made in Git (FRAMEWORK_5_2): commit 83961ff9091935b51375bbf0987a0a62a0397aa3 Author: Michael J Rubinsky <mrubinsk@horde.org> Date: Sat, 16 Mar 2019 22:43:00 -0400 [mjr] Fix issue causing the loss of reply text in SMARTREPLY requests (~~Bug #14841~~). M doc/Horde/ActiveSync/changelog.yml https://github.com/horde/ActiveSync/commit/83961ff9091935b51375bbf0987a0a62a0397aa3

03/16/2019 10:35:48 PM	Michael Rubinsky	Comment #32	Reply to this comment
Thanks to information and a test device, sent by Samuel off-list, I believe this is now finally fixed/worked around. Essentially, the GMail client is assuming that the MIME type of the email body is always HTML because that's what it requests when synchronizing. This obviously isn't always the case, emails can have text/plain parts with no text/html parts. In these cases we are correctly setting the TYPE property to indicate it is, indeed, a plain text/plain body. Unfortunately, GMail appears to ignore this. Since the specification isn't 100% clear as to what to do if the MIME type requested isn't available, I'm now converting the plain text part into HTML in the most minimalistic way possible. Please let me know if this solves the problem on your end, as it solved the test cases on my end.

03/05/2019 09:44:24 PM	samuel (dot) wolf (at) wolf-maschinenbau (dot) de	Comment #31	Reply to this comment
Still same error with Android 9 / Patchlevel February 2019 and GMail App Android-Mail/9.2.3.235065705.release

08/11/2018 09:30:56 PM	Michael Rubinsky	State ⇒ Resolved

08/11/2018 09:29:52 PM	Michael Rubinsky	Comment #30	Reply to this comment
When you decode the base64 in the composemail request, do you see all of the text that YOU wrote in the reply, or some of that missing too? If you mean the base64 at <ComposeMail:SmartReply>, no some text still missing. Look like to me the text missing after the second answer line / line break. Then unfortunately, there is nothing Horde can do about this. The client is truncating your reply for some reason. That's broken behavior. All Horde can do is try to sanitize any badly formed HTML after it receives the data from the client, but we can't do anything if we never get the full data. From the sample you sent me earlier, it almost looks like it's trying to remove your signature or something. Also, do you have the tidy extension installed? Oh wasn't, it's now installed and apache2 restart, nothing changed. php7.0-tidy (7.0.30-0+deb9u1)

08/11/2018 09:19:00 PM	samuel (dot) wolf (at) wolf-maschinenbau (dot) de	Comment #29	Reply to this comment
When you decode the base64 in the composemail request, do you see all of the text that YOU wrote in the reply, or some of that missing too? If you mean the base64 at <ComposeMail:SmartReply>, no some text still missing. Look like to me the text missing after the second answer line / line break. Also, do you have the tidy extension installed? Oh wasn't, it's now installed and apache2 restart, nothing changed. php7.0-tidy (7.0.30-0+deb9u1)

08/11/2018 09:03:31 PM	Michael Rubinsky	Comment #28	Reply to this comment
[Show Quoted Text - 12 lines][Hide Quoted Text] M lib/Horde/Core/ActiveSync/Mail.php https://github.com/horde/Core/commit/81e0160a5319a4b26deb5545995b1cfe1eb3e217 I make this changes in my local code, double check and restart apache2. Unfortunately it did not work, I can reproduce it. Is there something else to change? When you decode the base64 in the composemail request, do you see all of the text that YOU wrote in the reply, or some of that missing too? Also, do you have the tidy extension installed?

08/11/2018 08:55:57 PM	Michael Rubinsky	Comment #27	Reply to this comment
M lib/Horde/Core/ActiveSync/Mail.php https://github.com/horde/Core/commit/81e0160a5319a4b26deb5545995b1cfe1eb3e217 I make this changes in my local code, double check and restart apache2. Unfortunately it did not work, I can reproduce it. Is there something else to change? When you decode the base64 in the composemail request, do you see all of the text that YOU wrote in the reply, or some of that missing too?

08/11/2018 08:53:54 PM	Michael Rubinsky	Comment #26	Reply to this comment
When the content is base64 decoded, you can see that the result is incorrect HTML (actual content removed for privacy): <div dir='auto'>blah blah blah<div I see this open end, but you mean the client "stop sending" in middle of the html code? The way "SMART" replies work is the client only sends the content of the reply to the server. The server then fetches the content of the original email (using the ItemId/FolderId sent in the SmartReply) and appends it to the reply text that was sent from the client. I.e., it only sends the new text that you compose, not the original content that you are replying to. So, the text you see when you decode the base64 in the <ComposeMail:SmartReply> element is expected to only contain the text that YOU composed. There is no filtering that is done by Horde to the content of the SmartReply that you see in the log - this is raw data from the client. If it's broken here, it's broken by the client. If I decode the base64 I see only my broken answer, but was there no filter before? I do not see the original reply text in the base64 decode (but in IMP), where is this text appended? This is correct. It's how SMART REPLY works. It's designed to save bandwidth between the EAS client and server. The server is responsible for grabbing the content of the original email that you are replying to and quoting/appending it to the message body that is sent from the eas client. The result is that when the message you are replying to is concatenated to your reply text, the result is badly formed HTML, with most of the text looking like it's part of an unclosed div tag. This is the question, who drop this text/html. The content of the ComposeMail:SmartReply is expected to contain only the reply text. I'm not sure what your full text was when you wrote it so I don't know what is being truncated there, if anything but I can tell you that the text being sent is broken html. Either way, whatever is causing that badly formed HTML is definitely coming from the client.

08/11/2018 08:37:20 PM	samuel (dot) wolf (at) wolf-maschinenbau (dot) de	Comment #25	Reply to this comment
M lib/Horde/Core/ActiveSync/Mail.php https://github.com/horde/Core/commit/81e0160a5319a4b26deb5545995b1cfe1eb3e217 I make this changes in my local code, double check and restart apache2. Unfortunately it did not work, I can reproduce it. Is there something else to change?

08/11/2018 08:30:59 PM	samuel (dot) wolf (at) wolf-maschinenbau (dot) de	Comment #24	Reply to this comment
When the content is base64 decoded, you can see that the result is incorrect HTML (actual content removed for privacy): <div dir='auto'>blah blah blah<div I see this open end, but you mean the client "stop sending" in middle of the html code? If I decode the base64 I see only my broken answer, but was there no filter before? I do not see the original reply text in the base64 decode (but in IMP), where is this text appended? The result is that when the message you are replying to is concatenated to your reply text, the result is badly formed HTML, with most of the text looking like it's part of an unclosed div tag. This is the question, who drop this text/html.

08/11/2018 08:17:32 PM	Git Commit	Comment #23	Reply to this comment
Changes have been made in Git (FRAMEWORK_5_2): commit 565cc1f1f98c0d0758eca79911482b5d9a01cacf Author: Michael J Rubinsky <mrubinsk@horde.org> Date: Sat, 11 Aug 2018 16:08:35 -0400 [mjr] Work around buggy EAS clients that send badly formed HTML in SMARTREPLY (~~Bug #14841~~). M doc/Horde/Core/changelog.yml https://github.com/horde/Core/commit/565cc1f1f98c0d0758eca79911482b5d9a01cacf

08/11/2018 08:11:38 PM	Git Commit	Comment #22	Reply to this comment
Changes have been made in Git (master): commit 639f18cc888e2fb5c7b85229241701a7233a3d0f Author: Michael J Rubinsky <mrubinsk@horde.org> Date: Sat, 11 Aug 2018 16:03:44 -0400 [mjr] Work around buggy EAS clients that send badly formed HTML in SMARTREPLY (~~Bug #14841~~). M doc/Horde/Core/CHANGES M package.xml https://github.com/horde/Core/commit/639f18cc888e2fb5c7b85229241701a7233a3d0f

08/11/2018 08:11:37 PM	Git Commit	Comment #21	Reply to this comment
Changes have been made in Git (master): commit ee7d2b38aa44c9f5603b96422a1477bd1c142c26 Author: Michael J Rubinsky <mrubinsk@horde.org> Date: Sat, 11 Aug 2018 16:03:44 -0400 [mjr] Work around buggy EAS clients that send badly formed HTML in SMARTREPLY (~~Bug #14841~~). M doc/Horde/Core/changelog.yml https://github.com/horde/Core/commit/ee7d2b38aa44c9f5603b96422a1477bd1c142c26

08/11/2018 08:02:03 PM	Git Commit	Comment #20	Reply to this comment
Changes have been made in Git (master): commit 0903c9cb13467fad144d13308a9e7d6066face8a Author: Michael J Rubinsky <mrubinsk@horde.org> Date: Sat, 11 Aug 2018 16:01:45 -0400 ~~Bug: 14841~~ Attempt to sanitize badly formed HTML sent by broken EAS clients. M lib/Horde/Core/ActiveSync/Mail.php https://github.com/horde/Core/commit/0903c9cb13467fad144d13308a9e7d6066face8a

08/11/2018 08:01:01 PM	Git Commit	Comment #19	Reply to this comment
Changes have been made in Git (FRAMEWORK_5_2): commit 81e0160a5319a4b26deb5545995b1cfe1eb3e217 Author: Michael J Rubinsky <mrubinsk@horde.org> Date: Sat, 11 Aug 2018 16:00:17 -0400 ~~Bug: 14841~~ Attempt to sanitize badly formed HTML sent by broken EAS clients. M lib/Horde/Core/ActiveSync/Mail.php https://github.com/horde/Core/commit/81e0160a5319a4b26deb5545995b1cfe1eb3e217

08/11/2018 07:47:04 PM	Michael Rubinsky	Comment #18 Assigned to Michael Rubinsky	Reply to this comment
The bad news is that this is a client issue. The good news is, we can probably sanitize the badly formed HTML it is sending... From the latest log you sent me you can see the data that is being sent FROM your client TO the server. This represents YOUR reply to an email. When the content is base64 decoded, you can see that the result is incorrect HTML (actual content removed for privacy): <div dir='auto'>blah blah blah<div The result is that when the message you are replying to is concatenated to your reply text, the result is badly formed HTML, with most of the text looking like it's part of an unclosed div tag. I'm going to add code to run the incoming reply text through some tidy filters to see if that helps your case.

08/08/2018 04:05:47 PM	samuel (dot) wolf (at) wolf-maschinenbau (dot) de	Comment #17	Reply to this comment
I get weird double folder names for all my mail folders. I know about that issue, this happens when you open GMail to early (?) after add an EAS account. Reset states on the Horde side and wait for the first complete sync. ;) Either way, we don't add any html when replying to plain text and don't use 'data-smartm' anywhere either. I understand this, I did not want to point you in the wrong direction with my typo (google,...)

08/08/2018 03:55:44 PM	Michael Rubinsky	Comment #16	Reply to this comment
version of the protocol. Unless you are talking about the native Android Mail app, and not the Gmail app? There is no native Android Mail App (AFAIK), it's all in the GMail App. I'll be darned, you're right, I forgot that they rolled those into one app. I was thinking about the Mail app on my Samsung devices. I tried setting up a Gmail account on another, more recent, phone and it sort-of worked. I get weird double folder names for all my mail folders. I gave up on the Gmail/Android apps long ago during development of our EAS code due to them being lacking in modern EAS features and buggy. I'll have to put this back on my radar when I have time... client issue - we don't add that data-smartms attribute Sorry typo, the message ends with data-smartm (s was from samuel in the terminal). Either way, we don't add any html when replying to plain text and don't use 'data-smartm' anywhere either. [Show Quoted Text - 10 lines][Hide Quoted Text] NINE reply it with multipart/alternative, GMail with text/html. NINE: MIME-Version: 1.0 Content-Type: multipart/alternative; boundary=--_com.ninefolders.hd3.email_2288528803058_alt GMail: MIME-Version: 1.0 Content-Type: text/html; charset=utf-8

08/08/2018 03:36:22 PM	samuel (dot) wolf (at) wolf-maschinenbau (dot) de	Comment #15	Reply to this comment
version of the protocol. Unless you are talking about the native Android Mail app, and not the Gmail app? There is no native Android Mail App (AFAIK), it's all in the GMail App. client issue - we don't add that data-smartms attribute Sorry typo, the message ends with data-smartm (s was from samuel in the terminal). NINE reply it with multipart/alternative, GMail with text/html. NINE: MIME-Version: 1.0 Content-Type: multipart/alternative; boundary=--_com.ninefolders.hd3.email_2288528803058_alt GMail: MIME-Version: 1.0 Content-Type: text/html; charset=utf-8

08/08/2018 03:20:59 PM	Michael Rubinsky	Comment #14	Reply to this comment
Maybe the last PHP7 upgrade on Debian 9 is the issue? php7.0:amd64 (7.0.27-0+deb9u1, 7.0.30-0+deb9u1) See in my mail history this happens also with 7.0.27, but unsure it happen since upgrade from php5 to php7. Seriously doubt it's a php version issue. Interesting, reply to plain text email and decode the base64 from the ActiveSync log show the text, end missing: [...]<br><div data-smartms This gives me somewhere else to look, but this honestly sounds like a client issue - we don't add that data-smartms attribute, that's the client. Though, since it's a plain-text reply that apparently has html in it, the filters on our end might be doing something to it. I'll look. Reply to html email and decode the base64 from the ActiveSync log is ok: [...]<br type="attribution"> How can drop this end? Is there something wrong with plain text <-> html reply?

08/08/2018 03:15:58 PM	Michael Rubinsky	Comment #13	Reply to this comment
Wait, you are using the GMAIL app for activesync to Horde's email account? I didn't think that worked. Like at all? Works fine, contacts, calendar, mail (only this issue now). You mean the GMail APP/ActiveSync is the issue? That's surprising. The Gmail app doesn't even have a exchange/eas account option on ios, and in Android, it does, but uses an ancient version of the protocol. Unless you are talking about the native Android Mail app, and not the Gmail app?

08/08/2018 11:45:18 AM	samuel (dot) wolf (at) wolf-maschinenbau (dot) de	Comment #12	Reply to this comment
Maybe the last PHP7 upgrade on Debian 9 is the issue? php7.0:amd64 (7.0.27-0+deb9u1, 7.0.30-0+deb9u1) See in my mail history this happens also with 7.0.27, but unsure it happen since upgrade from php5 to php7. Interesting, reply to plain text email and decode the base64 from the ActiveSync log show the text, end missing: [...]<br><div data-smartms Reply to html email and decode the base64 from the ActiveSync log is ok: [...]<br type="attribution"> How can drop this end? Is there something wrong with plain text <-> html reply?

08/08/2018 11:13:54 AM	samuel (dot) wolf (at) wolf-maschinenbau (dot) de	Comment #11	Reply to this comment
Same mail works fine with NINE. One difference, NINE write a line under every reply: ________________________________ Can GMail/Android so faulty?

08/08/2018 09:49:53 AM	samuel (dot) wolf (at) wolf-maschinenbau (dot) de	Comment #10	Reply to this comment
Wait, you are using the GMAIL app for activesync to Horde's email account? Can reproduce the issue now, test today with: Android-Mail/8.3.25.192150130.release (Android 8.0) Android-Mail/8.5.6.197055160.release (Android 8.1) Android-Mail/8.7.15.206199545.release (Android 8.1) Maybe the last PHP7 upgrade on Debian 9 is the issue? php7.0:amd64 (7.0.27-0+deb9u1, 7.0.30-0+deb9u1)

08/06/2018 02:57:55 AM	samuel (dot) wolf (at) wolf-maschinenbau (dot) de	Comment #9	Reply to this comment
Wait, you are using the GMAIL app for activesync to Horde's email account? I didn't think that worked. Like at all? Works fine, contacts, calendar, mail (only this issue now). You mean the GMail APP/ActiveSync is the issue? Sorry, I need the log showing the email being sent FROM the device, when you reply to the original email, not the log showing the server sending the received email. You said it happens when replying to an My fault, send you the right part of the log, sorry. email using activesync, right? Yes, only on reply using ActiveSync.

08/05/2018 11:55:50 PM	Michael Rubinsky	Comment #8	Reply to this comment
Wait, you are using the GMAIL app for activesync to Horde's email account? I didn't think that worked. Like at all?

08/05/2018 10:04:04 PM	Michael Rubinsky	Comment #7	Reply to this comment
It will look similar to: [9597][2018-08-05T17:58:17-04:00] I: <ComposeMail:SmartReply> [9597][2018-08-05T17:58:17-04:00] I: <ComposeMail:ClientId> [9597][2018-08-05T17:58:17-04:00] I: ca3f70cc-5570-4bfa-9207-c9603cce93b7-0 [9597][2018-08-05T17:58:17-04:00] I: </ComposeMail:ClientId> [9597][2018-08-05T17:58:17-04:00] I: <ComposeMail:Source> [9597][2018-08-05T17:58:17-04:00] I: <ComposeMail:FolderId> [9597][2018-08-05T17:58:17-04:00] I: F266c6478 [9597][2018-08-05T17:58:17-04:00] I: </ComposeMail:FolderId> [9597][2018-08-05T17:58:17-04:00] I: <ComposeMail:ItemId> [9597][2018-08-05T17:58:17-04:00] I: 391845 [9597][2018-08-05T17:58:17-04:00] I: </ComposeMail:ItemId> [9597][2018-08-05T17:58:17-04:00] I: </ComposeMail:Source> [9597][2018-08-05T17:58:17-04:00] I: <ComposeMail:SaveInSentItems /> [9597][2018-08-05T17:58:17-04:00] I: <ComposeMail:MIME> [9597][2018-08-05T17:58:17-04:00] I: [2357 bytes of content] [9597][2018-08-05T17:58:17-04:00] I: </ComposeMail:MIME> [9597][2018-08-05T17:58:17-04:00] I: </ComposeMail:SmartReply>

08/05/2018 09:51:30 PM	Michael Rubinsky	Comment #6	Reply to this comment
Would you be able to provide a activesync log showing the message being sent from the device? Yes, send you by mail. Look like I can reproduce it with this mail. Sorry, I need the log showing the email being sent FROM the device, when you reply to the original email, not the log showing the server sending the received email. You said it happens when replying to an email using activesync, right?

08/05/2018 05:13:06 PM	samuel (dot) wolf (at) wolf-maschinenbau (dot) de	Comment #5	Reply to this comment
Would you be able to provide a activesync log showing the message being sent from the device? Yes, send you by mail. Look like I can reproduce it with this mail.

08/03/2018 02:08:12 PM	samuel (dot) wolf (at) wolf-maschinenbau (dot) de	Comment #4	Reply to this comment
Impossible to write a log file at the moment ~~#14842~~ Unfortunately also GAL search over ActiveSync doesn't work anymore...

07/29/2018 08:05:48 PM	samuel (dot) wolf (at) wolf-maschinenbau (dot) de	Comment #3	Reply to this comment
Would you be able to provide a activesync log showing the message being sent from the device? There are quite a number of things that the client can set that changes the behavior of the activesync server. Not at the moment, logging is normally disabled :-/ Also, does this by any chance happen only on forwards/replies? Yes only on replies (forwards not tested). See this on replies to emails from Airmail iOS, that's why I think first it's a GMail App problem. As far I know was every time we see this issue a URL in the text (not sure this is an important information). Trying today send and reply mails to myself (GMail App), but can not reproduce the error in this combination.

07/29/2018 04:32:18 PM	Michael Rubinsky	Comment #2 State ⇒ Feedback	Reply to this comment
Would you be able to provide a activesync log showing the message being sent from the device? There are quite a number of things that the client can set that changes the behavior of the activesync server. Also, does this by any chance happen only on forwards/replies?

07/27/2018 07:41:55 PM	samuel (dot) wolf (at) wolf-maschinenbau (dot) de	Comment #1 Priority ⇒ 1. Low State ⇒ Unconfirmed New Attachment: send_mail_activesync.png Patch ⇒ No Milestone ⇒ Queue ⇒ Synchronization Type ⇒ Bug Summary ⇒ Send mail over ActiveSync - lose content	Reply to this comment
From time to time some of my mails send over ActiveSync lose content. I have no idea why this happens, it is not reproducible. Search for PHP/Mail errors, upgrade my GMail App but still the same issue. Attached a screenshot, it's a mail in my send folder with missing some content (red arrow). Interesting the quote arrow is behind my text. Any idea whats going wrong here? php7.0-common 7.0.30-0+deb9u1 Horde_Core 2.31.3 stable Horde_ActiveSync 2.39.3 stable