Tue, 07 Oct 2008 11:53:06 -0400 http://bugs.horde.org/ticket/6295 Ansel flat shares with hierarchies Finally I had some time to merge my flat share driver from request # with some hierarchies logic from DataTree. The result is a new share driver for Ansel galleries speeding listing of my 86k galleries to an usable stage. 84k galleries, just counting and selecting galleries on the first page (9 galleries as default pref) current Ansel : 11,17 sec, memory used 13,37 MB. applied bug #6290: 7,67 sec, memory used 6,55 MB **using this driver: 0,12 sec, memory used 4,98 MB** The main performance benefit is of removing nested joins from datatree attributes table to a flat table. Where we have every attribute in a dedicated indexed column. So every gallery has only one row. I called it share driver. I am going to investigate more. As we still select galleries with two queries. Wed, 20 Feb 2008 04:10:10 -0500 http://bugs.horde.org/ticket/6295#t42647 Going forward. Modifying listGalleries use a direct join so we can get galleries in one db query we get a little more time 0,095 sec. but a much smaller memory footprint, only 1,572,864. Modifying table stracture to merge shares and galleries table I fall down to 0,089 execution time. But uses a little more memory, 1,835,008. The last is not logic, but I don't have time to dig in now. I must go back and revision the code to clean up it and test other functionalities other then listing ( I can already CRUD sub-galleries). But before going foward I will like to get your comments/ideas. Wed, 20 Feb 2008 05:01:30 -0500 http://bugs.horde.org/ticket/6295#t42648 > Modifying table stracture to merge shares and galleries table I fall > down to 0,089 execution time. But uses a little more memory, > 1,835,008. The last is not logic, but I don't have time to dig in now. I haven't looked at your patch yet, but I will say that I'd prefer to keep the shares and galleries table separate. IMO it makes for a cleaner implementation and keeps the share implementation/driver decoupled from the gallery objects. But I'm definitely open to other's thoughts to convince me otherwise. Wed, 20 Feb 2008 12:08:05 -0500 http://bugs.horde.org/ticket/6295#t42660 > I haven't looked at your patch yet, but I will say that I'd prefer to > keep the shares and galleries table separate. IMO it makes for a > cleaner implementation and keeps the share implementation/driver > decoupled from the gallery objects. But I'm definitely open to > other's thoughts to convince me otherwise. By combining we can remove 3 columns and have one join less. It will make data more compact and much faster and memory friendly which is always a strong point for large systems. But your wish is not a problem for me, beside data being proccessed. Moving the storage mechanism on driver bases, will allow me to have my own performance driver and combined tables. When someone can still use the classic DT implementation or this share driver. So the next step will be defenitly be to separate the storega classes with driver abstraction. Making my work easier possible to have different testing drivers without affecting Ansel core. Wed, 20 Feb 2008 14:21:44 -0500 http://bugs.horde.org/ticket/6295#t42672 Agreed, and abstracting the storage has been on my TODO for some time now... Wed, 20 Feb 2008 14:55:28 -0500 http://bugs.horde.org/ticket/6295#t42675 > Agreed, and abstracting the storage has been on my TODO for some time now... So a part of my patch is what you are looking for. You can omit the share driver itself as is not finished yet. Thu, 21 Feb 2008 03:56:23 -0500 http://bugs.horde.org/ticket/6295#t42705 Fix share driver. Should work in most cases. I am going to import my image files and test in deeply. Thu, 21 Feb 2008 13:49:04 -0500 http://bugs.horde.org/ticket/6295#t42726 Fixed criteria. Now seems to work all actions I tried. I am going to create a demo install for my coworkers to try Ansel and create a bug and usability enhancement list. Tue, 26 Feb 2008 10:59:16 -0500 http://bugs.horde.org/ticket/6295#t43108 I am unable to successfully apply the Ansel.php patch. Could you please upload an updated diff? Thanks. Thu, 28 Feb 2008 14:43:03 -0500 http://bugs.horde.org/ticket/6295#t43255 > I am unable to successfully apply the Ansel.php patch. Could you Sure Fri, 29 Feb 2008 07:58:39 -0500 http://bugs.horde.org/ticket/6295#t43269 Thanks for the updated patch, I'm not ignoring this, but I'm waiting until I have more solid dedicated time to look at it, as it looks like there is still more work to do to abstract out storage...in particular, there are still queries being executed directly in Ansel_Gallery (and maybe Image, don't remember) and I'd prefer to have *all* the storage access abstracted out if we are going to do this... Sun, 02 Mar 2008 10:28:41 -0500 http://bugs.horde.org/ticket/6295#t43312 Ok. I looked at this a little closer this afternoon. Maybe I misunderstood your intentions, but I thought one of the reasons you wanted to do this was to enable you to use a custom "performance" driver that would combine the share and gallery data into one table...with the way your splitting the various classes, your leaving a lot of database specific code behind in the original Ansel.php file (in Ansel_Gallery and Ansel_Storage, and Ansel_Image - although the Ansel_Image code is probably ok as I don't think it touches the galleries table). I think if we're going to go through the process of moving this to a "true" driver structure, we need to abstract out *all* the data storage for the vaious drivers. Otherwise any drivers that use a different table structure (as you say your will) will need to override various Ansel_* methods. Also, it seems like the Ansel_Share classes in the two different Storage "drivers" have different interfaces - some of which, at least, is due to the category methods no being present in the datatree version of the driver. I *do* think it's worth moving to a more modular driver structure - but I think we need to rethink the design, maybe? As always, I'm open to other's feedback / suggestions / slaps etc... Sun, 02 Mar 2008 15:38:09 -0500 http://bugs.horde.org/ticket/6295#t43320 ...forgot to mention, with the new share driver code you've included here, I don't see why we would even need to keep the datatree driver if everything works well with the new share driver you've provided and a conversion script can be written. I just didn't want to merge the gallery and share tables. Thanks again! Sun, 02 Mar 2008 15:42:51 -0500 http://bugs.horde.org/ticket/6295#t43321 I intentionally take in observations only the shares part. Not just because shares was the main performance problem but also was clean and so easier to understand and implement. I thought that going directly to a "true" storage driver will be probably to much for the beginning, as this driver does not need any Ansel core modifications. And as the share driver is now, can be even easily ported to other applications using shares and hierarchies. I like to see a general driver too. In it, we can freely combine the list/countGalleries to one query (now we need two runs for each of them). Sun, 02 Mar 2008 17:11:25 -0500 http://bugs.horde.org/ticket/6295#t43324 Another issue that I came across while trying to see the best way to split out the storage was that with your share driver, it is not possible to set a scope other than Ansel unless a SQL table already exists for the scope you want to use. This might not be too big of an issue if the scope is another Horde application, where the table might exist (if the new share driver get's adopted) but it's not a great general solution. Yes, the table can always be created, but I think one of the strong points of Ansel is that you can use the API to create an image storage backend for any custom application, not just Horde applications - and having to create a new storage table in the Horde database for any new "application" is not a very good solution IMO. Fri, 07 Mar 2008 18:30:46 -0500 http://bugs.horde.org/ticket/6295#t43565 If we are trying to be to general again, it will be a step backward. We can still create a driver from this one to use a single table. Then the administrator can choose which one to use. Sun, 09 Mar 2008 06:59:26 -0400 http://bugs.horde.org/ticket/6295#t43611 I split just the Horde_Shares class into a driver pattern and included Ansel_Gallery_Share class definitions in each one of those files. I excluded the Ansel_Storage subclasses that you had originally included as I don't see the reason for those changes... I attempted to make small changes to your original 'share' driver code to fit into this structure but haven't tested yet. Wed, 19 Mar 2008 14:39:49 -0400 http://bugs.horde.org/ticket/6295#t44021 So is time I post the last version of the share driver. Contains some parents fixes and avoids _fromDriver loop. It should work in all cases, but I don't use it any more. Is much better than DT but is just a was between a pure sql driver in order to leave the data structure as much as possible common to the original structure. I even left the idea of having a join based driver. Yes it reducses the number of queries and ram usage on the php side, but the time spent by the database server to join two large tables is longer than selecting ids and then the data as in the share driver. The best option is actually an sql driver I am working on right now. Combines the two share and gallery tables in one, while still having separate the shares and gallery objects side. Wed, 19 Mar 2008 15:32:05 -0400 http://bugs.horde.org/ticket/6295#t44024 OK. I'm merging the needed changes from the overridden methods in Ansel_Storage_share into the main-line driver (Since Ansel_Storage *creates* Ansel_Shares_* it would be impossible to get the sub class with the current code structure). Then I'll make the few needed changes in the share driver and drop into lib/Shares. Regarding your "best bet" sql driver - is this something that will require other api changes or would it be another "drop in" Ansel_Share driver? If the former, unless the performance is earth shatteringly different, I'd propose to save that one for Horde 4 and we can finally "lock in" on a driver architecture for Ansel 1.0 Any other thoughts on this from anyone? Thu, 20 Mar 2008 10:55:40 -0400 http://bugs.horde.org/ticket/6295#t44071 Ok, new driver is merged in. Could you possibly either add some documentation to the driver file, or even better, provide a sql script to create the database table(s) needed for your driver? Thu, 20 Mar 2008 12:14:33 -0400 http://bugs.horde.org/ticket/6295#t44072 > Ok, new driver is merged in. > Could you possibly either add some documentation to the driver file, > or even better, provide a sql script to create the database table(s) > needed for your driver? The structure contains the structure for the share and future native sql driver. Thu, 20 Mar 2008 13:02:15 -0400 http://bugs.horde.org/ticket/6295#t44076 > Regarding your "best bet" sql driver - is this something that will > require other api changes or would it be another "drop in" > Ansel_Share driver? If the former, unless the performance is earth > shatteringly different, I'd propose to save that one for Horde 4 and > we can finally "lock in" on a driver architecture for Ansel 1.0 It does not require special structure changes, only the Ansel::createGallery() and Ansel_Gallery::save() methods should be corrected. But can be moved to the storage driver itself. Structural changes soul be done after DT will be left behind and we will be able to avoid some ft logic. Thu, 20 Mar 2008 13:07:01 -0400 http://bugs.horde.org/ticket/6295#t44077 Resolved by porting the sql share driver to handle hierarchical shares. Hopefully, we can migrate Trean to use the same driver. Thanks to Duck for all the work on these drivers. Bye-Bye Datatree :) Fri, 09 May 2008 10:53:45 -0400 http://bugs.horde.org/ticket/6295#t45207