6.0.0-git
2021-05-07

[#14199] Decoding of base64 encoded Kolab (.xml) objects is broken
Summary Decoding of base64 encoded Kolab (.xml) objects is broken
Queue Kolab
Type Bug
State Resolved
Priority 1. Low
Owners mrubinsk (at) horde (dot) org, thomas.jarosch (at) intra2net (dot) com
Requester mike.gabriel (at) das-netzwerkteam (dot) de
Created 2015-12-16 (1969 days ago)
Due
Updated 2016-01-02 (1952 days ago)
Assigned 2016-01-02 (1952 days ago)
Resolved 2016-01-02 (1952 days ago)
Milestone
Patch Yes

History
2016-01-02 20:10:32 Michael Rubinsky Comment #17
State ⇒ Resolved
Reply to this comment
Fixed in Git. Release on it's way.
2016-01-02 19:56:42 Michael Rubinsky Comment #16 Reply to this comment
This commit actually breaks the unit tests. Need to track this down 
to determine if it's a broken test or broken logic before releasing.
Ugh. This is because there is no consistency in how the 
Horde_Kolab_Storage_Driver_* classes return data from the 
::getBodyPart() method. Some return it always decoded (like 
Horde_Imap_Client), some not decoded (like CClient and the mock driver 
used in the test). This is probably the reason we didn't assume 8Bit 
encoded data when setting the mime part contents. Anyway, this 
reiterates the need to drop the non-maintained drivers going forward 
and standardize on the Horde_Imap_Client driver only.


2016-01-02 19:11:05 Michael Rubinsky State ⇒ Assigned
 
2016-01-02 19:10:47 Michael Rubinsky Comment #15 Reply to this comment

[Show Quoted Text - 14 lines]
This commit actually breaks the unit tests. Need to track this down to 
determine if it's a broken test or broken logic before releasing.
2015-12-28 22:42:34 Michael Rubinsky Comment #14
Assigned to Michael Rubinsky
Taken from Michael J Rubinsky <mrubinsk@horde.org>
State ⇒ Resolved
Reply to this comment
https://github.com/horde/horde/commit/9d4c9323bca19cd8d722dbbb882b72390109f3a7#diff-0e5242e7e792d6a3dbcde340e95cff1e

The above commit looked like it could fix the observed double 
decoding issue on reading events, tasks, etc.

It does not. My provided patch is still valid and fixes *reading* 
base64 decoded MIME types.
Correct. That patch is just to avoid the overhead of having Horde_Mime 
perform decoding if the IMAP server (or the imap client library) was 
able to do this already.

Your patch does make sense and was applied, thanks!

2015-12-28 22:40:09 Git Commit Comment #13 Reply to this comment
Changes have been made in Git (master):

commit 4922efa0e79f5a3d954a7425f076610734e8cba9
Author: Mike Gabriel <mike.gabriel@das-netzwerkteam.de>
Date:   Mon Dec 28 17:38:46 2015 -0500

     Bug: 14199 Fix decoding of base64 encoded Kolab (.xml) objects

     Signed-off-by: Michael J Rubinsky <mrubinsk@horde.org>

  .../lib/Horde/Kolab/Storage/Object.php             |    2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)

http://github.com/horde/horde/commit/4922efa0e79f5a3d954a7425f076610734e8cba9
2015-12-18 12:11:20 mike (dot) gabriel (at) das-netzwerkteam (dot) de Comment #12 Reply to this comment
I haven't really tested this with turba. Only with kronolith and nag.
But turba should blow up, as well, as the issue is in the Kolab
Storage backend.
one more detail: Which IMAP server do you use for your tests?

I'm testing on cyrus-imapd 2.4.18, maybe dovecot or others behave 
differently regarding 8bit support.
I use kolab-cyrus-imapd as found in Debian squeeze-lts: 2.2.13-9.1.

I know, it is old, but things worked well in Horde3.

The issue only occurs if the mail attachments are stored in base64 
transfer encoding.

2015-12-18 11:46:17 mike (dot) gabriel (at) das-netzwerkteam (dot) de Comment #11 Reply to this comment
https://github.com/horde/horde/commit/9d4c9323bca19cd8d722dbbb882b72390109f3a7#diff-0e5242e7e792d6a3dbcde340e95cff1e

The above commit looked like it could fix the observed double decoding 
issue on reading events, tasks, etc.

It does not. My provided patch is still valid and fixes *reading* 
base64 decoded MIME types.

However, after some more testing, the patch provided below does *not* 
fix saving groupware items.

This is fixed / worked around by the patch provided in:
https://bugs.horde.org/ticket/14201


2015-12-18 11:23:30 Thomas Jarosch Comment #10 Reply to this comment
I haven't really tested this with turba. Only with kronolith and 
nag. But turba should blow up, as well, as the issue is in the Kolab 
Storage backend.
one more detail: Which IMAP server do you use for your tests?

I'm testing on cyrus-imapd 2.4.18, maybe dovecot or others behave 
differently regarding 8bit support.

2015-12-18 09:49:14 mike (dot) gabriel (at) das-netzwerkteam (dot) de Comment #9 Reply to this comment
The Horde Imap_Client backend is the only real maintained one, the
others haven't been supported for years. In fact I don't see much
point in keeping this extra maintenance burden in the future.
+1. These should be deprecated and removed for Horde 6.
+1 from here too... (the Kolab backend drivers consist of enough code 
already, unmaintained parts should really be dropped from the 
Horde_Kolab backend code).
2015-12-17 15:41:38 Michael Rubinsky State ⇒ Assigned
 
2015-12-17 15:37:14 Michael Rubinsky Assigned to Michael J Rubinsky <mrubinsk@horde.org>
 
2015-12-17 15:35:38 Michael Rubinsky Comment #8 Reply to this comment
The Horde Imap_Client backend is the only real maintained one, the 
others haven't been supported for years. In fact I don't see much 
point in keeping this extra maintenance burden in the future.
+1. These should be deprecated and removed for Horde 6.

2015-12-17 10:53:29 mike (dot) gabriel (at) das-netzwerkteam (dot) de Comment #7 Reply to this comment
Hi Thomas,
So basically having a contact picture + umlaut + disabled caching 
should trigger the problem in turba.
Hmm, really odd.
I haven't really tested this with turba. Only with kronolith and nag. 
But turba should blow up, as well, as the issue is in the Kolab 
Storage backend.

Mike
2015-12-17 10:36:19 Thomas Jarosch Comment #6 Reply to this comment

[Show Quoted Text - 15 lines]
thanks for the reproduction steps. I can take a look at this in the 
middle / end of January.
(currently busy with a time critical project that needs to be finished 
before the end of the year)

So basically having a contact picture + umlaut + disabled caching 
should trigger the problem in turba.
Hmm, really odd.

2015-12-17 09:11:20 mike (dot) gabriel (at) das-netzwerkteam (dot) de Comment #5 Reply to this comment
Btw. the issue is 100% reproducible with Horde caching disabled.

With caching enabled, things behave rather unpredictable.

For tasks in Nag, the situation was this:

   o caching enabled
   o create tasks with Umlauts (works)
   o tasks are shown in Nag and Kronolith

Then...

   o run horde-clear-cache
   o logout/login
   o those tasks with special chars are gone now...

2015-12-17 09:07:12 mike (dot) gabriel (at) das-netzwerkteam (dot) de Comment #4 Reply to this comment

[Show Quoted Text - 10 lines]
Ok...
@Mike: Which IMAP backend are you using for Kolab? I've seen you 
sent a patch for the Pear backend in bug entry #14200.
I use the Horde_Imap_Client (i.e., Horde_Kolab_Storage_Driver_Imap).

The patch in #14200 was a side catch when testing other 
Kolab_Storage_Driver_* implementations.
The Horde Imap_Client backend is the only real maintained one, the 
others haven't been supported for years. In fact I don't see much 
point in keeping this extra maintenance burden in the future. IIRC 
Horde Imap_Client even outperforms the others...
Yeah, I noticed that.

Please check with your test cases if your attachments are really 
stored with base64 transfer encoding.

I have seen Kolab XML objects stored in quoted-printable with base64 
encoded Umlauts / special chars (single letters).

The issue only occurs, when Kolab XML objects have transfer encoding 
base64. However, that is the default way of Horde_Kolab_Storage 
storing attachments (at least here).
2015-12-17 08:34:43 Thomas Jarosch Comment #3 Reply to this comment
This sounds familiar, like something that was fixed previously - 
though maybe the fix itself wasn't in the Kolab library.

Thomas, do you remember this, or can you even reproduce it? With all 
the torture testing we did recently, I'd be very surprised if this 
somehow escaped our notice. I'm sure you have 8-bit characters in 
your data ... :)
yes, even the automate test suite controlling the browser specifially 
uses umlauts in the Kolab data.

@Mike: Which IMAP backend are you using for Kolab? I've seen you sent 
a patch for the Pear backend in bug entry #14200.

The Horde Imap_Client backend is the only real maintained one, the 
others haven't been supported for years. In fact I don't see much 
point in keeping this extra maintenance burden in the future. IIRC 
Horde Imap_Client even outperforms the others...

2015-12-16 23:13:19 Michael Rubinsky Comment #2
Assigned to Thomas Jarosch
Priority ⇒ 1. Low
Reply to this comment
This sounds familiar, like something that was fixed previously - 
though maybe the fix itself wasn't in the Kolab library.

Thomas, do you remember this, or can you even reproduce it? With all 
the torture testing we did recently, I'd be very surprised if this 
somehow escaped our notice. I'm sure you have 8-bit characters in your 
data ... :)
2015-12-16 15:03:51 mike (dot) gabriel (at) das-netzwerkteam (dot) de Comment #1
Type ⇒ Bug
State ⇒ Unconfirmed
Priority ⇒ 3. High
Summary ⇒ Decoding of base64 encoded Kolab (.xml) objects is broken
Queue ⇒ Kolab
Milestone ⇒
Patch ⇒ Yes
New Attachment: 1001_Kolab_fix-decoding-of-base64-encoded-kolab-objects.patch Download
Reply to this comment
  Kolab groupware objects are stored as XML files / MIME attachments 
in mails on Kolab servers. Retrieving Kolab groupware objects means: 
retrieving message from an IMAP server, finding the Kolab XML file in 
the MIME attachment(s) of that mail and parse the contained (probably 
transfer-encoded) XML DOM tree.

The code in Kolab_Storage_Object decodes MIME attachments containing 
Kolab XML objects twice(!). The first decoding step (by a call to 
Horde_Imap_Data_Fetch::getBodyPart()) decodes from the MIME 
attachment's transfer-encoding to 8bit encoding. The second decoding 
step is--by design--actually a non-decoding step, and rather happens 
by accident in Horde_Mime_Part::setContents(). Even worse, it seems 
that the second encoding step defaults to the MIME attachments 
transfer encoding (base64), which has already been decoded to 8bit 
encoding.

So what happens with Kolab MIME attachments with base64 transfer 
encoding: they get base64-decoded twice. Which, of course, fails.

  Without the provided patch applied, this results in:

    * Saving failures of events, tasks, etc. if they contain umlauts
      and already contained umlauts when opening them.

    * Events, tasks, contacts, etc. not being shown on Kolab shared
      folders. People that a certain Kolab container (calendar, address book,
      task list) is shared with, cannot see events, tasks, contacts, if
      the underlying Kolab XML object's MIME attachment is stored with base64
      transfer encoding.

   * Reproduce:
      - create task with title "TEST" (saving works ok)
      - modify task, rename title to TÄST (saving works ok)
      - modify task again: rename title to TEST (or TÖST, does not 
matter): saving fails
      - the task named "TÄST" is now immutable and not visible anymore 
to other people
        how share the task list that task item is in

The error message, btw., is very similar to what has been reported here:
https://bugs.horde.org/ticket/13865

"""
Failed writing the Kolab object: Failed parsing Kolab
object input data of type string! Input was: <some-weird-characters>
"""


Saved Queries