Tickets :: [#14086] Image Attachments via ActiveSync: Last Byte Missing?

6.0.0-beta1

8/17/25

Summary	Image Attachments via ActiveSync: Last Byte Missing?
Queue	Synchronization
Queue Version	FRAMEWORK_5_2
Type	Bug
State	Resolved
Priority	1. Low
Owners	mrubinsk (at) horde (dot) org
Requester	guenter (at) vista-rasch (dot) com
Created	08/15/2015 (3655 days ago)
Due
Updated	08/22/2015 (3648 days ago)
Assigned	08/20/2015 (3650 days ago)
Resolved	08/20/2015 (3650 days ago)
Github Issue Link
Github Pull Request
Milestone
Patch	No

08/22/2015 09:03:19 AM	guenter (at) vista-rasch (dot) com	Comment #24	Reply to this comment
Fixed with pear package Horde_ActiveSync-2.29.0 I can confirm this update also fixes the issue on my setup (Server running CentOS 7.1, client Archos 45c Platinum running Android 4.4.2) Excellent work! :)

08/21/2015 05:51:04 PM	samuel (dot) wolf (at) wolf-maschinenbau (dot) de	Comment #23	Reply to this comment
Fixed with pear package Horde_ActiveSync-2.29.0

08/20/2015 02:30:44 PM	Michael Rubinsky	Comment #22 Taken from Jan Schneider State ⇒ Resolved Taken from slusarz@horde.org	Reply to this comment
yay!

08/20/2015 05:05:43 AM	samuel (dot) wolf (at) wolf-maschinenbau (dot) de	Comment #21	Reply to this comment
Can you test that this commit fixes things on your system as well? Confirm this commit fixed the issue on Debian 8. The mails with attachment load now really fast, like never before on the smartphone. Thank you for the workaround.

08/20/2015 03:01:03 AM	Michael Rubinsky	Comment #20	Reply to this comment
Can you test that this commit fixes things on your system as well? I've tested on all of my test VMs including 5.5.x and 5.6.x with and without the Debian package and they all seem to work with the test email that was broken earlier.

08/20/2015 02:55:59 AM	Git Commit	Comment #19	Reply to this comment
Changes have been made in Git (master): commit 97ad57bc3568286a14c41bc28d99f94bfce83d97 Author: Michael J Rubinsky <mrubinsk@horde.org> Date: Wed Aug 19 22:52:00 2015 -0400 ~~Bug: 14086~~ Work around various broken PHP behaviours regarding stream filters. It appears to be safe to use the base64-encode filter when it's attached as a STREAM_FILTER_WRITE. The behavior is only broken when the stream is being read through the filter, so perform the filtering in this method instead of just attaching the filter to the stream that is passed to the wbxml encoder. Adds a small memory overhead but this now works reliably with packages such as Debian's that have removed the BC breaking PHP patch as well as with PHP binaries that have the change included. .../Message/AirSyncBaseFileAttachment.php \| 16 ++++++++++++++-- 1 files changed, 14 insertions(+), 2 deletions(-) http://github.com/horde/horde/commit/97ad57bc3568286a14c41bc28d99f94bfce83d97

08/20/2015 01:59:20 AM	Michael Rubinsky	Comment #18 Assigned to slusarz@horde.org Assigned to Jan Schneider State ⇒ Feedback	Reply to this comment
To summarize: This commit: https://github.com/php/php-src/pull/936 which was an attempt to fix the base64-encode stream filter was BC breaking against PHP 5.x. This discussion: https://bugs.php.net/bug.php?id=68948 explains the issue in detail. In fact, I remember this discussion and I'm surprised I didn't put 2 and 2 together for THIS bug report.... At this point, I'm really not sure what to do. Debian reverted the BC breaking commit - which IMO is probably the best course of action. However, this means that we are screwed either way, as in Debian we have the original broken behavior of the base64 filter, but in pure PHP we have completely broken stream handling in some pretty critical places. Adding Michael S. and Jan to this ticket, as they were the main participants in the above mentioned ticket.

08/19/2015 09:54:58 PM	samuel (dot) wolf (at) wolf-maschinenbau (dot) de	Comment #17	Reply to this comment
Can't believe how deep this bug going and I'm surprised there's not more traffic in the horde mailing list on this issue. Open a Debian bug: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=796169

08/19/2015 08:12:51 PM	Michael Rubinsky	Comment #16	Reply to this comment
...and apparently this is a bug in the Debian/Ubuntu packages. Reported: https://github.com/oerdnj/deb.sury.org/issues/103

08/19/2015 07:35:48 PM	Michael Rubinsky	Comment #15	Reply to this comment
The original bug report's test case fails on 5.6.12. Updated the php ticket: https://bugs.php.net/bug.php?id=68532

08/19/2015 07:27:55 PM	Michael Rubinsky	Comment #14	Reply to this comment
Just verified this locally using the php 5.6 ppa in ubuntu which currently installs php 5.6.12. Not sure why the changelog for php says it's fixed in 5.6.5, unless it is a regression. Either way, verified that the issue is with the base64 stream filter as described below.

08/19/2015 07:14:25 PM	Michael Rubinsky	Comment #13	Reply to this comment
I'm not sure what to tell you about that. Can you try the fix, or something similar, to what I posted? If that "fixes" it for you, it's definitely a PHP issue with the stream filter.

08/19/2015 06:23:55 PM	samuel (dot) wolf (at) wolf-maschinenbau (dot) de	Comment #12	Reply to this comment
Was fixed in 5.5.21 and 5.6.5, which explains why my test apache vm exhibited this behavior but my main dev box did not. samuel@wds:~/php5$ php5 test.php memoryStream = dGVz fileStream = dGVzdA== samuel@wds:~/php5$ dpkg -l \| grep php ii dh-php5 0.2 all debhelper add-on to handle PHP PECL extensions ii libapache2-mod-php5 5.6.9+dfsg-0+deb8u1 amd64 server-side, HTML-embedded scripting language (Apache 2 module) ii php-pear 5.6.9+dfsg-0+deb8u1 all PEAR - PHP Extension and Application Repository ii php5 5.6.9+dfsg-0+deb8u1 all server-side, HTML-embedded scripting language (metapackage) ii php5-cgi 5.6.9+dfsg-0+deb8u1 amd64 server-side, HTML-embedded scripting language (CGI binary) ii php5-cli 5.6.9+dfsg-0+deb8u1 amd64 command-line interpreter for the php5 scripting language ii php5-common 5.6.9+dfsg-0+deb8u1 amd64 Common files for packages built from the php5 source ii php5-dev 5.6.9+dfsg-0+deb8u1 amd64 Files for PHP5 module development ii php5-gd 5.6.9+dfsg-0+deb8u1 amd64 GD module for php5 ii php5-imagick 3.2.0~rc1-1 amd64 Provides a wrapper to the ImageMagick library ii php5-imap 5.6.9+dfsg-0+deb8u1 amd64 IMAP module for php5 ii php5-json 1.3.6-1 amd64 JSON module for php5 ii php5-mcrypt 5.6.9+dfsg-0+deb8u1 amd64 MCrypt module for php5 ii php5-mysql 5.6.9+dfsg-0+deb8u1 amd64 MySQL module for php5 ii php5-readline 5.6.9+dfsg-0+deb8u1 amd64 Readline module for php5 ii pkg-php-tools 1.28 all various packaging tools and scripts for PHP packages samuel@wds:~/php5$

08/19/2015 04:33:08 PM	Michael Rubinsky	Comment #11	Reply to this comment
would the option to enable the workaround only on those platforms that do not have the fix in place (e.g. by checking the php version number) be worth considering? Not really, no. That would be a good amount of work to percolate an option like that down the call stack, only to work around a bug that isn't our doing, and is already fixed.

08/19/2015 04:22:40 PM	guenter (at) vista-rasch (dot) com	Comment #10	Reply to this comment
very cool find! guess we will try the workaround soon, will report about our results as soon as we have them. To get ahead of the "I can't upgrade PHP" and "this is the version that comes with <insert some distro here>" comments that always come after a bug like this, I should say we won't work around this in code since it would require holding the entire attachment in memory, thus negating just about all of the work we do to minimize memory usage by using streams. This could also very well hit your local php memory limit. would the option to enable the workaround only on those platforms that do not have the fix in place (e.g. by checking the php version number) be worth considering?

08/19/2015 03:56:19 PM	Michael Rubinsky	Comment #9	Reply to this comment
To get ahead of the "I can't upgrade PHP" and "this is the version that comes with <insert some distro here>" comments that always come after a bug like this, I should say we won't work around this in code since it would require holding the entire attachment in memory, thus negating just about all of the work we do to minimize memory usage by using streams. This could also very well hit your local php memory limit. HOWEVER, if someone wants to work around this locally on their own install knowing the consequences of doing so, what you need to do is modify Horde_ActiveSync_Message_AirSyncBaseFileAttachment::_checkEncoding() method to pull the stream contents into memory, perform a base64_encode() on them, and then write it back out to another stream to return: So replace the inner block in that method with something like this: if (is_resource($data)) { rewind($data); $temp_data = stream_get_contents($data); $data = fopen('php://temp/', 'r+'); fwrite($data, base64_encode($temp_data)); } else { $data = base64_encode($data); }

08/19/2015 03:35:50 PM	Michael Rubinsky	Comment #8 State ⇒ Not A Bug	Reply to this comment
This is a bug in PHP's base64_encode stream filter. https://bugs.php.net/bug.php?id=68532 Was fixed in 5.5.21 and 5.6.5, which explains why my test apache vm exhibited this behavior but my main dev box did not.

08/19/2015 02:11:46 PM	Michael Rubinsky	Comment #7	Reply to this comment
Has this issue some parallels with ~~ticket #12486~~ Not the same issue, but it got me thinking to try this on an Apache based server. I can reproduce this when using Apache, but not on lighttpd. It's definitely not a header or a chunked content issue though. At least I can reproduce this now.

08/18/2015 11:04:33 PM	samuel (dot) wolf (at) wolf-maschinenbau (dot) de	Comment #6	Reply to this comment
Has this issue some parallels with ~~ticket #12486~~

08/18/2015 10:33:06 PM	samuel (dot) wolf (at) wolf-maschinenbau (dot) de	Comment #5	Reply to this comment
Can you send me an email containing an image that this occurs with and attach an activesync log contianing at least the ITEMOPERATIONS command that fetches the attachment? -rw-r--r-- 1 samuel samuel 58831 Aug 19 00:09 test_horde.png <ItemOperations:Range> 0-58830 </ItemOperations:Range> <ItemOperations:Total> 58831 </ItemOperations:Total> [...] 2015-08-18T22:25:02+00:00 INFO: [ANDROIDXXXXXXXXX] Handling ITEMOPERATIONS command. 2015-08-18T22:25:02+00:00 DEBUG: [32523] I <ItemOperations:ItemOperations> 2015-08-18T22:25:02+00:00 DEBUG: [32523] I <ItemOperations:Fetch> 2015-08-18T22:25:02+00:00 DEBUG: [32523] I <ItemOperations:Store> 2015-08-18T22:25:02+00:00 DEBUG: [32523] I Mailbox 2015-08-18T22:25:02+00:00 DEBUG: [32523] I </ItemOperations:Store> 2015-08-18T22:25:02+00:00 DEBUG: [32523] I <AirSyncBase:FileReference> 2015-08-18T22:25:02+00:00 DEBUG: [32523] I INBOX:3:2 2015-08-18T22:25:02+00:00 DEBUG: [32523] I </AirSyncBase:FileReference> 2015-08-18T22:25:02+00:00 DEBUG: [32523] I </ItemOperations:Fetch> 2015-08-18T22:25:02+00:00 DEBUG: [32523] I </ItemOperations:ItemOperations> 2015-08-18T22:25:02+00:00 DEBUG: [32523] O <ItemOperations:ItemOperations> 2015-08-18T22:25:02+00:00 DEBUG: [32523] O <ItemOperations:Status> 2015-08-18T22:25:02+00:00 DEBUG: [32523] O 1 2015-08-18T22:25:02+00:00 DEBUG: [32523] O </ItemOperations:Status> 2015-08-18T22:25:02+00:00 INFO: [32523] Creating new Horde_ActiveSync_SyncCache. 2015-08-18T22:25:02+00:00 DEBUG: [32523] O <ItemOperations:Response> 2015-08-18T22:25:02+00:00 DEBUG: [32523] O <ItemOperations:Fetch> 2015-08-18T22:25:02+00:00 DEBUG: [32523] O <ItemOperations:Status> 2015-08-18T22:25:02+00:00 DEBUG: [32523] O 1 2015-08-18T22:25:02+00:00 DEBUG: [32523] O </ItemOperations:Status> 2015-08-18T22:25:02+00:00 DEBUG: [32523] O <AirSyncBase:FileReference> 2015-08-18T22:25:02+00:00 DEBUG: [32523] O INBOX:3:2 2015-08-18T22:25:02+00:00 DEBUG: [32523] O </AirSyncBase:FileReference> 2015-08-18T22:25:02+00:00 DEBUG: [32523] O <ItemOperations:Properties> 2015-08-18T22:25:02+00:00 DEBUG: [32523] O <ItemOperations:Range> 2015-08-18T22:25:02+00:00 DEBUG: [32523] O 0-58830 2015-08-18T22:25:02+00:00 DEBUG: [32523] O </ItemOperations:Range> 2015-08-18T22:25:02+00:00 DEBUG: [32523] O <ItemOperations:Total> 2015-08-18T22:25:02+00:00 DEBUG: [32523] O 58831 2015-08-18T22:25:02+00:00 DEBUG: [32523] O </ItemOperations:Total> 2015-08-18T22:25:02+00:00 DEBUG: [32523] O <AirSyncBase:ContentType> 2015-08-18T22:25:02+00:00 DEBUG: [32523] O image/png 2015-08-18T22:25:02+00:00 DEBUG: [32523] O </AirSyncBase:ContentType> 2015-08-18T22:25:02+00:00 DEBUG: [32523] O <ItemOperations:Data> 2015-08-18T22:25:02+00:00 DEBUG: [32523] O [STREAM] 2015-08-18T22:25:02+00:00 DEBUG: [32523] O </ItemOperations:Data> 2015-08-18T22:25:02+00:00 DEBUG: [32523] O </ItemOperations:Properties> 2015-08-18T22:25:02+00:00 DEBUG: [32523] O </ItemOperations:Fetch> 2015-08-18T22:25:02+00:00 DEBUG: [32523] O </ItemOperations:Response> 2015-08-18T22:25:02+00:00 DEBUG: [32523] O </ItemOperations:ItemOperations> 2015-08-18T22:25:02+00:00 INFO: [32523] Maximum memory usage for ActiveSync request: 18087936 bytes. [...]

08/18/2015 08:50:48 PM	Michael Rubinsky	Comment #4 State ⇒ Feedback	Reply to this comment
I can't reproduce this. I've tried 6 different images of varying sized on multiple android devices. Can you send me an email containing an image that this occurs with and attach an activesync log contianing at least the ITEMOPERATIONS command that fetches the attachment?

08/18/2015 02:33:35 PM	Michael Rubinsky	State ⇒ Assigned

08/18/2015 12:16:35 PM	samuel (dot) wolf (at) wolf-maschinenbau (dot) de	Comment #3	Reply to this comment
Send mail from one Horde server to a second Horde server with IMP. Sent Mail Folder (IMP): md5sum attachment: aa2d147b50c6eadb92d498942fb47cb8 size: 64750 Inbox Folder (IMP second server): md5sum attachment: aa2d147b50c6eadb92d498942fb47cb8 size: 64750 Inbox Folder (Android): Android can not open the attachment, but I can save it to the downloads folder, connect the smartphone to my pc and copy the image: md5sum attachment: 5cce577cb27835a7b80d381d06c914d3 size: 64749

08/18/2015 10:58:11 AM	samuel (dot) wolf (at) wolf-maschinenbau (dot) de	Comment #2 New Attachment: Screenshot_2015-08-18-12-54-3311111.png	Reply to this comment
(2) view email on android client connected via ActiveSync - client will not display image (image loading error) see attachment

08/15/2015 01:08:40 PM	Michael Rubinsky	Assigned to Michael Rubinsky Priority ⇒ 1. Low

08/15/2015 10:29:05 AM	guenter (at) vista-rasch (dot) com	Comment #1 Priority ⇒ 2. Medium State ⇒ Unconfirmed Patch ⇒ No Milestone ⇒ Summary ⇒ Image Attachments via ActiveSync: Last Byte Missing? Type ⇒ Bug Queue ⇒ Synchronization	Reply to this comment
While investigating why my android device would sometimes refuse to display image attachments in EMails I discovered that apparently the last byte is (sometimes/often) missing from those attachments, verified by this experiment: (1) send email to my horde account with image attached (2) view email on android client connected via ActiveSync - client will not display image (image loading error) (3) save image attachment on android client to downloads folder on the device (4) send email back with saved image attached to it (5) compare size of original and received image: -rw-r----- 1 guenter guenter 659625 Aug 14 22:33 speech_and_language (1).jpg -rw-r----- 1 guenter guenter 659626 Aug 8 12:19 speech_and_language.jpg effect is reproducible accross multiple devices and two separate horde installations (one freshly installed, one in production use for years). I am happy to provide further details (version numbers, avticesync log) if that seems useful, just drop me a line.