Changes have been made in Git (master):

commit 521b2258f947c4f6fac7472df649c1366416aca2
Author: Thomas Jarosch <thomas.jarosch@intra2net.com>
Date:   Wed Nov 23 15:58:10 2016 +0100

     Fix UIDVALIDITY change detection

     The 'is_reset' flag was lost during a
     INFO -> DEBUG log level change in
     commit 395635111ddde15dd9f0183d75d9be079cc728a5

     Probably a botched merge since there were
     two corrections to this code later on, too.

     -> restore it. Originally introduced by bugs.horde.org entry #11807.

  framework/Kolab_Storage/lib/Horde/Kolab/Storage/Data/Cached.php | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)

http://github.com/horde/horde/commit/521b2258f947c4f6fac7472df649c1366416aca2

Changes have been made in Git (master):

commit 8e9463e42526b5617b465a12fbc4ad565e016d08
Author: Thomas Jarosch <thomas.jarosch@intra2net.com>
Date:   Wed Mar 20 14:38:06 2013 +0100

     Improve IMAP folder UIDVALIDITY change detection. Fixes #11807.

     We stored the whole list of IMAP UIDs of a folder next
     to every object. This turned out to be a major memory
     and CPU time hog (see #11807).

     Instead we detect a change of UIDVALIDITY at the
     IMAP folder cache level (where it belongs) and pass
     the information about a reset situation to the
     history 'query' as additional parameter.

  .../lib/Horde/Kolab/Storage/Data/Cached.php        |   13 +++++++--
  .../Kolab/Storage/Data/Query/History/Base.php      |   27 
+++++++++++---------
  2 files changed, 25 insertions(+), 15 deletions(-)

http://github.com/horde/horde/commit/8e9463e42526b5617b465a12fbc4ad565e016d08

Review kolab-horde-history pull request? ;)

I'm downstream and send my changes upstream.

Upstream?

This issue has been resolved, the code is waiting to be merged upstream.

Regarding the history driver, the SQL based history backend is 
automatically enabled if a SQL database is configured (sqlite in my 
case).

The Horde_Kolab_Storage_Factory always uses "Horde_History_Mock".

Or was your intention that I should replace the "mock" driver with the 
SQL backend
in Horde/Kolab/Storage/Factory.php?

Do we really need to serialize the whole UID list for every processed message?
Can't we do it just once or at least update something like a 
"last_known_uids" entry?

Do you see the same behavior with any of the SQL based history driver? 
After all the "Mock" driver simply keeps the history in memory. It 
forgets all its information after each and every request. Meaning that 
it rebuilds the history log after each request. So for the mock driver 
this is the "expected" behavior.

Alright, I traced this with the xhprof profiler.

The function Horde_History_Mock::_log() calls 
Horde_Kolab_Storage_Folder_Stamp_Uids::serialize()
for every Kolab object in that folder.

So for every Kolab object of a total of 5,544,
we call serialize() of the complete IMAP UID list of a folder...

For the fun of it, I've added a return statement at the beginning of 
the _log() function.
The peak memory usage dropped to 100mb and the page load is fast, too.

Attached is the relevant profiler output as HTML page.

Hi,

I've done some stress testing with 6.000+ contacts on a Kolab backend.
Tests are done with H4 and H5.

H4 consumes about 150MB memory during IMAP retrieval + parse + display.

H5 consumes 140MB memory during IMAP retrieval and 750MB+ during parse. Argh.
Once the parsed contacts are cached, simple clicks in turba consume 
about 150MB.

We probably do something very memory unfriendly either in the Kolab 
driver or in the Kolab_Storage framework.

This is just for the record, the issue is of low priority for now.

Thomas