6.0.0-beta1
7/8/25

[#10072] Memcache storage is duplicating the key prefix
Summary Memcache storage is duplicating the key prefix
Queue Horde Framework Packages
Queue Version Git master
Type Bug
State Resolved
Priority 1. Low
Owners slusarz (at) horde (dot) org
Requester goncalo.queiros (at) portugalmail (dot) net
Created 05/12/2011 (5171 days ago)
Due
Updated 05/24/2011 (5159 days ago)
Assigned 05/24/2011 (5159 days ago)
Resolved 05/24/2011 (5159 days ago)
Github Issue Link
Github Pull Request
Milestone
Patch Yes

History
05/24/2011 04:07:31 PM Michael Slusarz Comment #17
State ⇒ Resolved
Reply to this comment
We have been thinking about a fix for this problem, but since its 
not related with the original bug, i will post to the dev the 
solution we've gotten to.
Thanks for your help in this ticket.

05/24/2011 03:39:59 PM goncalo (dot) queiros (at) portugalmail (dot) net Comment #16 Reply to this comment
Memcache will have a mediocre hit rate because we do locking within 
memcache itself.  So there will be a lot of cache misses - every 
write will necessarily have a miss before.
Investigating this further we found out that it isn't related with 
locking, since setting $conf['memcache']['large_items'] to false will 
increase memcache hits to around 96%. So the problem is due to the get 
misses generated when trying to get the respective _os keys of items 
that are not large items.

We have been thinking about a fix for this problem, but since its not 
related with the original bug, i will post to the dev the solution 
we've gotten to.

05/24/2011 07:17:21 AM Michael Slusarz Comment #15
Assigned to Michael Slusarz
State ⇒ Feedback
Reply to this comment
We are back to this issue, and discovered a problem on the existing 
code (patch attached)
Slightly modified and committed (we shouldn't be changing the function 
signature because we are extending the method from the base class).   
Thanks for tracking this down.

[Show Quoted Text - 9 lines]
Memcache will have a mediocre hit rate because we do locking within 
memcache itself.  So there will be a lot of cache misses - every write 
will necessarily have a miss before.
05/24/2011 07:13:18 AM Git Commit Comment #14 Reply to this comment
Changes have been made in Git for this ticket:

Bug #10072: Fix memcache duplicate keys

  2 files changed, 7 insertions(+), 8 deletions(-)
http://git.horde.org/horde-git/-/commit/b28f5232ab1193cbd008a710d024de58c9b688df
05/23/2011 06:27:18 PM goncalo (dot) queiros (at) portugalmail (dot) net Comment #13
New Attachment: 0001-Pass-the-correct-key-to-expire-exists-methods-of-Mem.patch Download
Reply to this comment
We are back to this issue, and discovered a problem on the existing 
code (patch attached)

We agree that this is a conceptual problem, but it also doesn't make 
sense having the Horde_Cache_Storage_Memcache + Horde_Memcache 
duplicating the key prefix since one always calls the other.

We could maybe pass an empty prefix when we build the 
Horde_Cache_Storage_Memcache object based on the already prefixed 
Horde_Memcache in Horde_Core_Factory_Cache. We are only looking to the 
cache keys because we have a cache hit around 50% (which seems very 
low). Is this the expected hit rate?
05/16/2011 09:52:13 PM Michael Slusarz Comment #12 Reply to this comment
I don't say that memcache is not working correctly, im just saying 
that i think there's some problem with it. The way i see it having 
in the memcache something like:
PREFIX_number_of_session
This key would be created by something that *directly* calls Horde_Memcache.
PREFIX_PREFIX_number_of_session
This key would be created by something that calls Horde_Cache->memcache.

So these are two completely different calls.

If you are using something like memcache caching of session 
information, this is completely expected.  The session handler caches 
using direct queries to Horde_Memcache.  But that same data may be 
cached elsewhere via Horde_Cache.  So although the data happens to be 
the same, it is not a duplicate key - they are two independent keys 
that happen to have the same data.  As mentioned previously, this is 
an inevitable limitation of OO programming (and caching in general).
05/16/2011 09:39:12 PM Michael Slusarz Comment #11 Reply to this comment

[Show Quoted Text - 22 lines]
If this is happening then this *might* be an error.  You will have to 
track down where these alleged duplicate prefixes are being generated 
(Horde::debug() might be useful for this).

Note that just because the data value is the same, that does NOT mean 
that the key is being duplicated.  That is a dangerous assumption and 
is simply wrong.

FWIW, your patch doesn't fix this problem at all.  All you are doing 
is removing the prefix feature, which is not the correct solution.
05/14/2011 10:33:04 AM goncalo (dot) queiros (at) portugalmail (dot) net Comment #10 Reply to this comment
I still don't understand why this is an issue.  Please don't tell me 
you are looking at key names in your memcache (or any) backend.   
Because that is completely irrelevant.
Yes, i was debugging memcache get() and set() methods.
All that matters is that for a given key input to Cache::get() (or 
Cache::set()), a single, unique data value is associated with it.   
If a key has 2, 3 (or 10) prefixes tacked on to the front, why does 
that matter?
I think it matters, because without the patch after a single login i 
have (ex) 30 keys being created on cache. With the patch i only have 
like 12 or 13.. this is because without the patch every key is created 
twice in the database (one with one prefix the other with the prefix 2 
times) and some are even created 3 times...

I don't say that memcache is not working correctly, im just saying 
that i think there's some problem with it. The way i see it having in 
the memcache something like:
PREFIX_number_of_session
PREFIX_PREFIX_number_of_session

doens't seem correct
05/13/2011 10:30:39 PM Michael Slusarz Comment #9 Reply to this comment

[Show Quoted Text - 17 lines]
I still don't understand why this is an issue.  Please don't tell me 
you are looking at key names in your memcache (or any) backend.   
Because that is completely irrelevant.

All that matters is that for a given key input to Cache::get() (or 
Cache::set()), a single, unique data value is associated with it.  If 
a key has 2, 3 (or 10) prefixes tacked on to the front, why does that 
matter?
05/13/2011 12:57:54 AM goncalo (dot) queiros (at) portugalmail (dot) net Comment #8 Reply to this comment
The default value for memcache caching prefix 
($conf['cache']['params']['prefix']) is empty, so a duplicate prefix 
would be an issue with your local installation.
I have setted $conf['cache']['params']['prefix'] because i have 
multiple horde instances (vhosts) in the same memcache...
If the value of $conf['cache']['params']['prefix'] is something like 
'ABC' then i will have on my memcache duplicate keys like:
ABC_horde_perms_sql
ABC_ABC_horde_perms_sql
and for some others i even have triplicates!!!
ABC_ABC_ABC_horde_perms_sql

Picking up a practical example:
Horde_Cache_Storage_Memcache method set(), adds the prefix to the key 
and then calls $this->_memcache->set().. and this method will also 
append the prefix to the $key, thus making it duplicate
05/12/2011 10:20:39 PM Michael Slusarz Comment #7 Reply to this comment
If this is happening in practice, the solution would be to ensure 
that the prefix given to the cache object is empty.  But removing 
prefixing in the cache object accomplishes nothing.
The default value for memcache caching prefix 
($conf['cache']['params']['prefix']) is empty, so a duplicate prefix 
would be an issue with your local installation.
05/12/2011 10:14:30 PM Michael Slusarz Comment #6 Reply to this comment
Because after setting that $key, the class will call 
$this->_memcache->xxxmethod
In this class, $this->_memcache that is a Horde_Memcache object...
This object always calls the function _key that prefixes the 
memcache prefix..thus, duplicating it.
But how is a prefix configuration setting in the Horde_Cache->Memcache 
object relevant to what is going in Horde_Memcache?  Basic object 
oriented principle that classes are blind to the implementation 
details of other classes.

If this is happening in practice, the solution would be to ensure that 
the prefix given to the cache object is empty.  But removing prefixing 
in the cache object accomplishes nothing.
05/12/2011 09:39:23 PM goncalo (dot) queiros (at) portugalmail (dot) net Comment #5 Reply to this comment
Because after setting that $key, the class will call 
$this->_memcache->xxxmethod
In this class, $this->_memcache that is a Horde_Memcache object...
This object always calls the function _key that prefixes the memcache 
prefix..thus, duplicating it.


05/12/2011 08:48:20 PM Michael Slusarz Comment #4 Reply to this comment
How is this duplicating the prefix?  The prefix is added once per 
method.  Where else would the prefix be added?
05/12/2011 01:48:42 PM goncalo (dot) queiros (at) portugalmail (dot) net Comment #3
New Attachment: 0001-Memcache-storage-is-duplicating-key-prefix.patch Download
Reply to this comment
True story :-p
05/12/2011 12:22:20 PM rsalmon (at) mbpgroup (dot) com Comment #2 Reply to this comment
Attached patch.
no file attached.
05/12/2011 11:49:29 AM goncalo (dot) queiros (at) portugalmail (dot) net Comment #1
Priority ⇒ 1. Low
State ⇒ Unconfirmed
Patch ⇒ Yes
Milestone ⇒
Summary ⇒ Memcache storage is duplicating the key prefix
Type ⇒ Bug
Queue ⇒ Horde Framework Packages
Reply to this comment
Attached patch.

Saved Queries