6.0.0-git
2019-04-21

[#5564] Memcache data size limitations
Summary Memcache data size limitations
Queue IMP
Queue Version HEAD
Type Bug
State Resolved
Priority 3. High
Owners slusarz (at) horde (dot) org
Requester adrieder (at) sbox (dot) tugraz (dot) at
Created 2007-07-18 (4295 days ago)
Due 07/18/2007 (4295 days ago)
Updated 2007-08-03 (4279 days ago)
Assigned 2007-07-25 (4288 days ago)
Resolved 2007-08-03 (4279 days ago)
Milestone
Patch No

History
2007-08-03 15:04:21 Chuck Hagenbuch Comment #20
State ⇒ Resolved
Reply to this comment
Resolving, unless we hear otherwise.
2007-08-03 07:29:54 adrieder (at) sbox (dot) tugraz (dot) at Comment #19 Reply to this comment
I've tested it for a while now (single user only, for now) and it 
works fine. Maybe there are also some other users who can confirm it.
2007-08-01 04:33:42 Michael Slusarz Comment #18
Summary ⇒ Memcache data size limitations
Reply to this comment
found some more small problems:
Committed.  Thanks again.  Hopefully, this will allow us to close this ticket.


2007-07-25 08:51:48 adrieder (at) sbox (dot) tugraz (dot) at Comment #17
New Attachment: Memcache-0725.php.patch Download
Reply to this comment
found some more small problems:

- Horde_Memcache::set() is still called from within set(), which 
doesn't work (at least here)

- when using Horde_Memcache::get() there is no need for prepend the 
'prefix' parameter to

   the key

- in order to avoid compression for the memcache oversize lookup 
table, the entries should

   not be set using Horde_Memcache::set(), but rather by using Memcache::set



The attached patch (Memcache-0725.php.patch) solves those points and 
works fine here.
2007-07-25 05:32:40 Michael Slusarz Comment #16
State ⇒ Feedback
Reply to this comment
Let's see where these changes get us.  Thanks for helping debug this code.
2007-07-25 05:32:10 Michael Slusarz Comment #15 Reply to this comment
... I think this patch to framework/Cache/Cache/memcache.php solves
the array() key problem.
Yep - looks like this was a remnant from when we were directly calling 
the memcache methods from within this object.
2007-07-25 05:29:01 Michael Slusarz Comment #14 Reply to this comment
it is still seqfaulting with your latest changes, and I think found
also some additional problems in the code. The attached patch works
for me, and with this I'm also able to correctly open/download big
attachments from gpg signed messages.
Added all changes except the compression for memcache oversize lookup 
table - this table will be very small (a couple of entries at best) 
and since it needs to be loaded on every memcache lookup, no need to 
have this overhead.


2007-07-25 05:18:17 Michael Slusarz Comment #13 Reply to this comment
Just from looking at it:

I think line 161 should read:
  $os = $this->_memcache->get($this->_params['prefix'] . 'horde_memcache_os');
instead of:
  $os = $this->_memcache->get('horde_memcache_os');

right?
No - this was by design.  I wanted a single lookup table for all 
prefixes.  But thinking about this a bit more... it doesn't really 
hurt to split this table into several tables.
2007-07-24 14:15:34 adrieder (at) sbox (dot) tugraz (dot) at Comment #12
New Attachment: memcache.php.patch Download
Reply to this comment
... I think this patch to framework/Cache/Cache/memcache.php solves 
the array() key problem.
2007-07-24 12:00:31 adrieder (at) sbox (dot) tugraz (dot) at Comment #11
New Attachment: Memcache.php.patch Download
Reply to this comment
it is still seqfaulting with your latest changes, and I think found 
also some additional problems in the code. The attached patch works 
for me, and with this I'm also able to correctly open/download big 
attachments from gpg signed messages.



Please not the "TODO" comment. I found that sometimes we get arrays 
passed as cache keys, e.g. from horde/services/portal/sidebar.php or 
imp/mailbox.php. I added an error logging code to the patch to see 
what is in the arrays. Here some examples:



a:2:{i:0;s:27:"perm_exists_imp:max_folders";i:1;s:29:"perm_exists_imp:max_folders_e";

a:2:{i:0;s:34:"perm_exists_turba:sources:localsql";i:1;s:36:"perm_exists_turba:sources:localsql_e";}

a:2:{i:0;s:30:"perm_exists_imp:create_folders";i:1;s:32:"perm_exists_imp:create_folders_e";}



It seems that all of them are deriving from the perms system.



I'm not sure if erverything is correct in this patch, but I hope this 
helps to further optimize the memcache driver.
2007-07-23 18:48:28 adrieder (at) sbox (dot) tugraz (dot) at Comment #10 Reply to this comment
Just from looking at it:



I think line 161 should read:

  $os = $this->_memcache->get($this->_params['prefix'] . 'horde_memcache_os');

instead of:

  $os = $this->_memcache->get('horde_memcache_os');



right?
2007-07-23 17:18:12 Michael Slusarz Comment #9 Reply to this comment
...and shouldn't the substring start be something like "$i *
MEMCACHE_MAX_SIZE" instead of just "$i" ? (Line 196)
Correct.  This has been fixed.
2007-07-23 17:17:59 Michael Slusarz Comment #8 Reply to this comment
:-(
With this patch my apache starts to seqfault....
The attached, patch at least solves the segfault problem,
Turns out that we were calling Horde_Memcache::get() from within get() 
instead of calling the function in the memcache object - this is most 
likely where the segfault was occuring.  This has been fixed.


2007-07-23 17:15:01 Michael Slusarz Deleted Original Message
 
2007-07-23 15:48:49 adrieder (at) sbox (dot) tugraz (dot) at Comment #7 Reply to this comment
...and shouldn't the substring start be something like "$i * 
MEMCACHE_MAX_SIZE" instead of just "$i" ? (Line 196)
2007-07-23 13:13:12 adrieder (at) sbox (dot) tugraz (dot) at Comment #6
New Attachment: Memcache.php.patch Download
Reply to this comment
:-(
With this patch my apache starts to seqfault....
The attached, patch at least solves the segfault problem, but I still 
can not open the pdf attachment (or other bigger attachemnts) when   
the message is pgp signed.

I also noted the following log in the error log, when opening the 
message with the pdf attachments:



PHP Notice:  Array to string conversion in 
/tmp/include/Horde/Memcache.php on line 190, referer: 
https://webmail.tugraz.at/cvs_dev/horde/imp/mailbox.php?mailbox=INBOX.sent-mail&start=67


2007-07-19 07:31:43 adrieder (at) sbox (dot) tugraz (dot) at Comment #5 Reply to this comment
:-(

With this patch my apache starts to seqfault....
2007-07-18 21:11:29 Michael Slusarz Comment #4 Reply to this comment
Ironically enough, I committed some code relating to this same problem 
today, but for different reasons.  It has not yet been tested, so if 
you could provide feedback that would be great:

  http://lists.horde.org/archives/cvs/Week-of-Mon-20070716/069425.html
2007-07-18 19:43:16 Chuck Hagenbuch Assigned to Michael Slusarz
State ⇒ Assigned
 
2007-07-18 15:39:27 adrieder (at) sbox (dot) tugraz (dot) at Comment #3 Reply to this comment

[Show Quoted Text - 12 lines]
As I guessed, it is indeed a memcache max item size problem. The 
default memcached allows only for a max item size of 1Mb.

When dirty hacking the memached server to allow 2Mb items the gpg 
signed message including the attachment opens correcly, but that's not 
a clean solution.



BTW:

When the message is not gpg-signed, then it's no problem to open and 
download the attachment.
2007-07-18 09:26:16 adrieder (at) sbox (dot) tugraz (dot) at Comment #2
New Attachment: 3d-structure-chromatin-rev-16075283.pdf Download
Reply to this comment
The pdf file, when attached and signed via IMP, can't be displayed or 
downloaded via IMP anymore. I get a blank message frame when trying to 
click on the download (diskette) item.

When clicking on the filename link I get  a randomly blank new window 
or a horde login page, or a horde portal page.

I guess it is again a memcache item size problem. This fle is about 
500kb, smaller files seem to work. I tried to disable the mailbox 
cache by stetting $conf['mboxcache']['use_mboxcache'] = false; but 
this didn't help.

I always get errors of this kind when opening the message:



PHP Notice:  Memcache::set() [<a 
href='function.Memcache-set'>function.Memcache-set</a>]: marked server 
'xxx.xxx.xxx.xxx:11211' as failed in /tmp/include/Horde/Memcache.php 
on line 151, referer: 
https://webmail.tugraz.at/cvs_dev/horde/imp/mailbox.php?mailbox=INBOX&start=3


2007-07-18 09:14:49 adrieder (at) sbox (dot) tugraz (dot) at Comment #1
Type ⇒ Bug
State ⇒ Unconfirmed
Priority ⇒ 3. High
Summary ⇒ problems attaching files when gpg-signing
Due ⇒ 2007-07-18
Queue ⇒ IMP
New Attachment: Similaun-2007.kmz Download
Reply to this comment
I discovered several problems with adding attachments when the message 
is gpg-signed.

The attached file (Similaun-2007.kmz, wich is a zipped xml for 
google-earth) here will not be attached to the message at all.


Saved Queries