6.0.0-git
2019-10-16

[#14841] Send mail over ActiveSync - lose content
Summary Send mail over ActiveSync - lose content
Queue Synchronization
Queue Version FRAMEWORK_5_2
Type Bug
State Resolved
Priority 1. Low
Owners mrubinsk (at) horde (dot) org
Requester samuel.wolf (at) wolf-maschinenbau (dot) de
Created 2018-07-27 (446 days ago)
Due
Updated 2019-03-17 (213 days ago)
Assigned 2018-07-29 (444 days ago)
Resolved 2018-08-11 (431 days ago)
Milestone
Patch No

History
2019-03-17 18:10:46 samuel (dot) wolf (at) wolf-maschinenbau (dot) de Comment #36 Reply to this comment
Please let me know if this solves the problem on your end, as it 
solved the test cases on my end.
Yes this solved my problem, awesome!
Thank you very much.
2019-03-17 02:45:23 Git Commit Comment #35 Reply to this comment
Changes have been made in Git (master):

commit 11cea561c9d92b2316d43527ee69136543186f1e
Author: Michael J Rubinsky <mrubinsk@horde.org>
Date:   Sat, 16 Mar 2019 22:45:16 -0400

[mjr] Fix issue causing the loss of reply text in SMARTREPLY requests 
(Bug #14841).

  M doc/Horde/ActiveSync/changelog.yml

https://github.com/horde/ActiveSync/commit/11cea561c9d92b2316d43527ee69136543186f1e
2019-03-17 02:43:11 Git Commit Comment #34 Reply to this comment
Changes have been made in Git (FRAMEWORK_5_2):

commit 68847766db72772ebf3c362d84ca9af21943b70e
Author: Michael J Rubinsky <mrubinsk@horde.org>
Date:   Sat, 16 Mar 2019 22:43:00 -0400

[mjr] Fix issue causing the loss of reply text in SMARTREPLY requests 
(Bug #14841).

  M doc/Horde/ActiveSync/CHANGES
  M package.xml

https://github.com/horde/ActiveSync/commit/68847766db72772ebf3c362d84ca9af21943b70e
2019-03-17 02:43:10 Git Commit Comment #33 Reply to this comment
Changes have been made in Git (FRAMEWORK_5_2):

commit 83961ff9091935b51375bbf0987a0a62a0397aa3
Author: Michael J Rubinsky <mrubinsk@horde.org>
Date:   Sat, 16 Mar 2019 22:43:00 -0400

[mjr] Fix issue causing the loss of reply text in SMARTREPLY requests 
(Bug #14841).

  M doc/Horde/ActiveSync/changelog.yml

https://github.com/horde/ActiveSync/commit/83961ff9091935b51375bbf0987a0a62a0397aa3
2019-03-16 22:35:48 Michael Rubinsky Comment #32 Reply to this comment
Thanks to information and a test device, sent by Samuel off-list, I 
believe this is now finally fixed/worked around.

Essentially, the GMail client is assuming that the MIME type of the 
email body is *always* HTML because that's what it requests when 
synchronizing. This obviously isn't always the case, emails can have 
text/plain parts with no text/html parts. In these cases we are 
correctly setting the TYPE property to indicate it is, indeed, a plain 
text/plain body.
  Unfortunately, GMail appears to ignore this.

Since the specification isn't 100% clear as to what to do if the MIME 
type requested isn't available, I'm now converting the plain text part 
into HTML in the most minimalistic way possible.

Please let me know if this solves the problem on your end, as it 
solved the test cases on my end.

2019-03-05 21:44:24 samuel (dot) wolf (at) wolf-maschinenbau (dot) de Comment #31 Reply to this comment
Still same error with Android 9 / Patchlevel February 2019 and GMail 
App Android-Mail/9.2.3.235065705.release
2018-08-11 21:30:56 Michael Rubinsky State ⇒ Resolved
 
2018-08-11 21:29:52 Michael Rubinsky Comment #30 Reply to this comment
When you decode the base64 in the composemail request, do you see all
of the text that YOU wrote in the reply, or some of that missing too?
If you mean the base64 at <ComposeMail:SmartReply>, no some text 
still missing.
Look like to me the text missing after the second answer line / line break.
Then unfortunately, there is nothing Horde can do about this. The 
client is truncating your reply for some reason. That's broken 
behavior. All Horde can do is try to sanitize any badly formed HTML 
after it receives the data from the client, but we can't do anything 
if we never get the full data.

From the sample you sent me earlier, it almost looks like it's trying 
to remove your signature or something.
Also, do you have the tidy extension installed?
Oh wasn't, it's now installed and apache2 restart, nothing changed.
php7.0-tidy (7.0.30-0+deb9u1)
2018-08-11 21:19:00 samuel (dot) wolf (at) wolf-maschinenbau (dot) de Comment #29 Reply to this comment
When you decode the base64 in the composemail request, do you see 
all of the text that YOU wrote in the reply, or some of that missing 
too?
If you mean the base64 at <ComposeMail:SmartReply>, no some text still 
missing.
Look like to me the text missing after the second answer line / line break.
Also, do you have the tidy extension installed?
Oh wasn't, it's now installed and apache2 restart, nothing changed.
php7.0-tidy (7.0.30-0+deb9u1)

2018-08-11 21:03:31 Michael Rubinsky Comment #28 Reply to this comment

[Show Quoted Text - 12 lines]
Also, do you have the tidy extension installed?
2018-08-11 20:55:57 Michael Rubinsky Comment #27 Reply to this comment
I make this changes in my local code, double check and restart apache2.
Unfortunately it did not work, I can reproduce it.

Is there something else to change?
When you decode the base64 in the composemail request, do you see all 
of the text that YOU wrote in the reply, or some of that missing too?

2018-08-11 20:53:54 Michael Rubinsky Comment #26 Reply to this comment
When the content is base64 decoded, you can see that the
result is incorrect HTML (actual content removed for privacy):

<div dir='auto'>blah blah blah<div
I see this open end, but you mean the client "stop sending" in 
middle of the html code?
The way "SMART" replies work is the client only sends the content of 
the reply to the server. The server then fetches the content of the 
original email (using the ItemId/FolderId sent in the SmartReply) and 
appends it to the reply text that was sent from the client.  I.e., it 
only sends the new text that you compose, not the original content 
that you are replying to. So, the text you see when you decode the 
base64 in the <ComposeMail:SmartReply> element is expected to only 
contain the text that YOU composed. There is no filtering that is done 
by Horde to the content of the SmartReply that you see in the log - 
this is raw data from the client. If it's broken here, it's broken by 
the client.
If I decode the base64 I see only my broken answer, but was there no 
filter before?
I do not see the original reply text in the base64 decode (but in 
IMP), where is this text appended?
This is correct. It's how SMART REPLY works. It's designed to save 
bandwidth between the EAS client and server. The server is responsible 
for grabbing the content of the original email that you are replying 
to and quoting/appending it to the message body that is sent from the 
eas client.
The result is that when the message you are replying to is
concatenated to your reply text, the result is badly formed HTML,
with most of the text looking like it's part of an unclosed div tag.
This is the question, who drop this text/html.
The content of the ComposeMail:SmartReply is expected to contain only 
the reply text. I'm not sure what your full text was when you wrote it 
so I don't know what is being truncated there, if anything but I can 
tell you that the text being sent is broken html. Either way, whatever 
is causing that badly formed HTML is definitely coming from the client.
2018-08-11 20:37:20 samuel (dot) wolf (at) wolf-maschinenbau (dot) de Comment #25 Reply to this comment
I make this changes in my local code, double check and restart apache2.
Unfortunately it did not work, I can reproduce it.

Is there something else to change?
2018-08-11 20:30:59 samuel (dot) wolf (at) wolf-maschinenbau (dot) de Comment #24 Reply to this comment
When the content is base64 decoded, you can see that the result is 
incorrect HTML (actual content removed for privacy):

<div dir='auto'>blah blah blah<div
I see this open end, but you mean the client "stop sending" in middle 
of the html code?
If I decode the base64 I see only my broken answer, but was there no 
filter before?
I do not see the original reply text in the base64 decode (but in 
IMP), where is this text appended?
The result is that when the message you are replying to is 
concatenated to your reply text, the result is badly formed HTML, 
with most of the text looking like it's part of an unclosed div tag.
This is the question, who drop this text/html.
2018-08-11 20:17:32 Git Commit Comment #23 Reply to this comment
Changes have been made in Git (FRAMEWORK_5_2):

commit 565cc1f1f98c0d0758eca79911482b5d9a01cacf
Author: Michael J Rubinsky <mrubinsk@horde.org>
Date:   Sat, 11 Aug 2018 16:08:35 -0400

[mjr] Work around buggy EAS clients that send badly formed HTML in 
SMARTREPLY (Bug #14841).

  M doc/Horde/Core/changelog.yml

https://github.com/horde/Core/commit/565cc1f1f98c0d0758eca79911482b5d9a01cacf
2018-08-11 20:11:38 Git Commit Comment #22 Reply to this comment
Changes have been made in Git (master):

commit 639f18cc888e2fb5c7b85229241701a7233a3d0f
Author: Michael J Rubinsky <mrubinsk@horde.org>
Date:   Sat, 11 Aug 2018 16:03:44 -0400

[mjr] Work around buggy EAS clients that send badly formed HTML in 
SMARTREPLY (Bug #14841).

  M doc/Horde/Core/CHANGES
  M package.xml

https://github.com/horde/Core/commit/639f18cc888e2fb5c7b85229241701a7233a3d0f
2018-08-11 20:11:37 Git Commit Comment #21 Reply to this comment
Changes have been made in Git (master):

commit ee7d2b38aa44c9f5603b96422a1477bd1c142c26
Author: Michael J Rubinsky <mrubinsk@horde.org>
Date:   Sat, 11 Aug 2018 16:03:44 -0400

[mjr] Work around buggy EAS clients that send badly formed HTML in 
SMARTREPLY (Bug #14841).

  M doc/Horde/Core/changelog.yml

https://github.com/horde/Core/commit/ee7d2b38aa44c9f5603b96422a1477bd1c142c26
2018-08-11 20:02:03 Git Commit Comment #20 Reply to this comment
Changes have been made in Git (master):

commit 0903c9cb13467fad144d13308a9e7d6066face8a
Author: Michael J Rubinsky <mrubinsk@horde.org>
Date:   Sat, 11 Aug 2018 16:01:45 -0400

Bug: 14841 Attempt to sanitize badly formed HTML sent by broken EAS clients.

  M lib/Horde/Core/ActiveSync/Mail.php

https://github.com/horde/Core/commit/0903c9cb13467fad144d13308a9e7d6066face8a
2018-08-11 20:01:01 Git Commit Comment #19 Reply to this comment
Changes have been made in Git (FRAMEWORK_5_2):

commit 81e0160a5319a4b26deb5545995b1cfe1eb3e217
Author: Michael J Rubinsky <mrubinsk@horde.org>
Date:   Sat, 11 Aug 2018 16:00:17 -0400

Bug: 14841 Attempt to sanitize badly formed HTML sent by broken EAS clients.

  M lib/Horde/Core/ActiveSync/Mail.php

https://github.com/horde/Core/commit/81e0160a5319a4b26deb5545995b1cfe1eb3e217
2018-08-11 19:47:04 Michael Rubinsky Comment #18
Assigned to Michael Rubinsky
Reply to this comment
The bad news is that this is a client issue. The good news is, we can 
probably sanitize the badly formed HTML it is sending...

From the latest log you sent me you can see the data that is being 
sent FROM your client TO the server. This represents YOUR reply to an 
email. When the content is base64 decoded, you can see that the result 
is incorrect HTML (actual content removed for privacy):

<div dir='auto'>blah blah blah<div


The result is that when the message you are replying to is 
concatenated to your reply text, the result is badly formed HTML, with 
most of the text looking like it's part of an unclosed div tag.

I'm going to add code to run the incoming reply text through some tidy 
filters to see if that helps your case.


2018-08-08 16:05:47 samuel (dot) wolf (at) wolf-maschinenbau (dot) de Comment #17 Reply to this comment
I get weird double folder names for all my mail folders.
I know about that issue, this happens when you open GMail to early (?) 
after add an EAS account.
Reset states on the Horde side and wait for the first complete sync. ;)
Either way, we don't add any html when replying to plain text and 
don't use 'data-smartm' anywhere either.
I understand this, I did not want to point you in the wrong direction 
with my typo (google,...)
2018-08-08 15:55:44 Michael Rubinsky Comment #16 Reply to this comment
version of the protocol. Unless you are talking about the native
Android Mail app, and not the Gmail app?
There is no native Android Mail App (AFAIK), it's all in the GMail App.
I'll be darned, you're right, I forgot that they rolled those into one 
app. I was thinking about the Mail app on my Samsung devices.  I tried 
setting up a Gmail account on another, more recent, phone and it 
sort-of worked. I get weird double folder names for all my mail 
folders. I gave up on the Gmail/Android apps long ago during 
development of our EAS code due to them being lacking in modern EAS 
features and buggy. I'll have to put this back on my radar when I have 
time...
client issue - we don't add that data-smartms attribute
Sorry typo, the message ends with data-smartm (s was from samuel in 
the terminal).
Either way, we don't add any html when replying to plain text and 
don't use 'data-smartm' anywhere either.

[Show Quoted Text - 10 lines]
2018-08-08 15:36:22 samuel (dot) wolf (at) wolf-maschinenbau (dot) de Comment #15 Reply to this comment
version of the protocol. Unless you are talking about the native 
Android Mail app, and not the Gmail app?
There is no native Android Mail App (AFAIK), it's all in the GMail App.
client issue - we don't add that data-smartms attribute
Sorry typo, the message ends with data-smartm (s was from samuel in 
the terminal).

NINE reply it with multipart/alternative, GMail with text/html.

NINE:
MIME-Version: 1.0
Content-Type: multipart/alternative; 
boundary=--_com.ninefolders.hd3.email_2288528803058_alt

GMail:
MIME-Version: 1.0
Content-Type: text/html; charset=utf-8
2018-08-08 15:20:59 Michael Rubinsky Comment #14 Reply to this comment
Maybe the last PHP7 upgrade on Debian 9 is the issue?
php7.0:amd64 (7.0.27-0+deb9u1, 7.0.30-0+deb9u1)
See in my mail history this happens also with 7.0.27, but unsure it 
happen since upgrade from php5 to php7.
Seriously doubt it's a php version issue.
Interesting, reply to plain text email and decode the base64 from 
the ActiveSync log show the text, end missing:
[...]<br><div data-smartms
This gives me somewhere else to look, but this honestly sounds like a 
client issue - we don't add that data-smartms attribute, that's the 
client. Though, since it's a plain-text reply that apparently has html 
in it, the filters on our end might be doing something to it. I'll look.
Reply to html email and decode the base64 from the ActiveSync log is ok:
[...]<br type="attribution">

How can drop this end?
Is there something wrong with plain text <-> html reply?
2018-08-08 15:15:58 Michael Rubinsky Comment #13 Reply to this comment
Wait, you are using the GMAIL app for activesync to Horde's email account?
I didn't think that worked. Like at all?
Works fine, contacts, calendar, mail (only this issue now).
You mean the GMail APP/ActiveSync is the issue?
That's surprising. The Gmail app doesn't even have a exchange/eas 
account option on ios, and in Android, it does, but uses an ancient 
version of the protocol. Unless you are talking about the native 
Android Mail app, and not the Gmail app?

2018-08-08 11:45:18 samuel (dot) wolf (at) wolf-maschinenbau (dot) de Comment #12 Reply to this comment
Maybe the last PHP7 upgrade on Debian 9 is the issue?
php7.0:amd64 (7.0.27-0+deb9u1, 7.0.30-0+deb9u1)
See in my mail history this happens also with 7.0.27, but unsure it 
happen since upgrade from php5 to php7.

Interesting, reply to plain text email and decode the base64 from the 
ActiveSync log show the text, end missing:
[...]<br><div data-smartms

Reply to html email and decode the base64 from the ActiveSync log is ok:
[...]<br type="attribution">

How can drop this end?
Is there something wrong with plain text <-> html reply?
2018-08-08 11:13:54 samuel (dot) wolf (at) wolf-maschinenbau (dot) de Comment #11 Reply to this comment
Same mail works fine with NINE.
One difference, NINE write a line under every reply:
________________________________

Can GMail/Android so faulty?
2018-08-08 09:49:53 samuel (dot) wolf (at) wolf-maschinenbau (dot) de Comment #10 Reply to this comment
Wait, you are using the GMAIL app for activesync to Horde's email account?
Can reproduce the issue now, test today with:
Android-Mail/8.3.25.192150130.release (Android 8.0)
Android-Mail/8.5.6.197055160.release (Android 8.1)
Android-Mail/8.7.15.206199545.release (Android 8.1)

Maybe the last PHP7 upgrade on Debian 9 is the issue?
php7.0:amd64 (7.0.27-0+deb9u1, 7.0.30-0+deb9u1)

2018-08-06 02:57:55 samuel (dot) wolf (at) wolf-maschinenbau (dot) de Comment #9 Reply to this comment
Wait, you are using the GMAIL app for activesync to Horde's email account?
I didn't think that worked. Like at all?
Works fine, contacts, calendar, mail (only this issue now).
You mean the GMail APP/ActiveSync is the issue?
Sorry, I need the log showing the email being sent FROM the device, 
when you reply to the original email, not the log showing the server 
sending the received email.  You said it happens when replying to an
My fault, send you the right part of the log, sorry.
email using activesync, right?
Yes, only on reply using ActiveSync.

2018-08-05 23:55:50 Michael Rubinsky Comment #8 Reply to this comment
Wait, you are using the GMAIL app for activesync to Horde's email account?

I didn't think that worked. Like at all?
2018-08-05 22:04:04 Michael Rubinsky Comment #7 Reply to this comment
It will look similar to:

[9597][2018-08-05T17:58:17-04:00] I: <ComposeMail:SmartReply>
[9597][2018-08-05T17:58:17-04:00] I:  <ComposeMail:ClientId>
[9597][2018-08-05T17:58:17-04:00] I:    ca3f70cc-5570-4bfa-9207-c9603cce93b7-0
[9597][2018-08-05T17:58:17-04:00] I:  </ComposeMail:ClientId>
[9597][2018-08-05T17:58:17-04:00] I:  <ComposeMail:Source>
[9597][2018-08-05T17:58:17-04:00] I:   <ComposeMail:FolderId>
[9597][2018-08-05T17:58:17-04:00] I:     F266c6478
[9597][2018-08-05T17:58:17-04:00] I:   </ComposeMail:FolderId>
[9597][2018-08-05T17:58:17-04:00] I:   <ComposeMail:ItemId>
[9597][2018-08-05T17:58:17-04:00] I:     391845
[9597][2018-08-05T17:58:17-04:00] I:   </ComposeMail:ItemId>
[9597][2018-08-05T17:58:17-04:00] I:  </ComposeMail:Source>
[9597][2018-08-05T17:58:17-04:00] I:  <ComposeMail:SaveInSentItems />
[9597][2018-08-05T17:58:17-04:00] I:  <ComposeMail:MIME>
[9597][2018-08-05T17:58:17-04:00] I:    [2357 bytes of content]
[9597][2018-08-05T17:58:17-04:00] I:  </ComposeMail:MIME>
[9597][2018-08-05T17:58:17-04:00] I: </ComposeMail:SmartReply>
2018-08-05 21:51:30 Michael Rubinsky Comment #6 Reply to this comment
Would you be able to provide a activesync log showing the message
being sent from the device?
Yes, send you by mail.
Look like I can reproduce it with this mail.
Sorry, I need the log showing the email being sent FROM the device, 
when you reply to the original email, not the log showing the server 
sending the received email.  You said it happens when replying to an 
email using activesync, right?
2018-08-05 17:13:06 samuel (dot) wolf (at) wolf-maschinenbau (dot) de Comment #5 Reply to this comment
Would you be able to provide a activesync log showing the message 
being sent from the device?
Yes, send you by mail.
Look like I can reproduce it with this mail.
2018-08-03 14:08:12 samuel (dot) wolf (at) wolf-maschinenbau (dot) de Comment #4 Reply to this comment
Impossible to write a log file at the moment #14842
Unfortunately also GAL search over ActiveSync doesn't work anymore...
2018-07-29 20:05:48 samuel (dot) wolf (at) wolf-maschinenbau (dot) de Comment #3 Reply to this comment
Would you be able to provide a activesync log showing the message 
being sent from the device? There are quite a number of things that 
the client can set that changes the behavior of the activesync server.
Not at the moment, logging is normally disabled :-/
Also, does this by any chance happen only on forwards/replies?
Yes only on replies (forwards not tested).

See this on replies to emails from Airmail iOS, that's why I think 
first it's a GMail App problem.
As far I know was every time we see this issue a URL in the text (not 
sure this is an important information).

Trying today send and reply mails to myself (GMail App), but can not 
reproduce the error in this combination.
2018-07-29 16:32:18 Michael Rubinsky Comment #2
State ⇒ Feedback
Reply to this comment
Would you be able to provide a activesync log showing the message 
being sent from the device? There are quite a number of things that 
the client can set that changes the behavior of the activesync server.

Also, does this by any chance happen only on forwards/replies?
2018-07-27 19:41:55 samuel (dot) wolf (at) wolf-maschinenbau (dot) de Comment #1
Type ⇒ Bug
State ⇒ Unconfirmed
Priority ⇒ 1. Low
Summary ⇒ Send mail over ActiveSync - lose content
Queue ⇒ Synchronization
Milestone ⇒
Patch ⇒ No
New Attachment: send_mail_activesync.png Download
Reply to this comment
From time to time some of my mails send over ActiveSync lose content.
I have no idea why this happens, it is not reproducible.
Search for PHP/Mail errors, upgrade my GMail App but still the same issue.

Attached a screenshot, it's a mail in my send folder with missing some 
content (red arrow).
Interesting the quote arrow is behind my text.

Any idea whats going wrong here?

php7.0-common                   7.0.30-0+deb9u1
Horde_Core             2.31.3  stable
Horde_ActiveSync       2.39.3  stable

Saved Queries