Resolved by porting the sql share driver to handle hierarchical 
shares. Hopefully, we can migrate Trean to use the same driver.



Thanks to Duck for all the work on these drivers.



Bye-Bye Datatree :)

It does not require special structure changes, only the 
Ansel::createGallery() and Ansel_Gallery::save() methods should be 
corrected. But can be moved to the storage driver itself.



Structural changes soul be done after DT will be left behind and we 
will be able to avoid some ft logic.

The structure contains the structure for the share and future native 
sql driver.

Ok, new driver is merged in.

Could you possibly either add some documentation to the driver file, 
or even better, provide a sql script to create the database table(s) 
needed for your driver?

OK. I'm merging the needed changes from the overridden methods in 
Ansel_Storage_share into the main-line driver (Since Ansel_Storage 
*creates* Ansel_Shares_* it would be impossible to get the sub class 
with the current code structure). Then I'll make the few needed 
changes in the share driver and drop into lib/Shares.



Regarding your "best bet" sql driver - is this something that will 
require other api changes or would it be another "drop in" Ansel_Share 
driver?  If the former, unless the performance is earth shatteringly 
different, I'd propose to save that one for Horde 4 and we can finally 
"lock in" on a driver architecture for Ansel 1.0



Any other thoughts on this from anyone?

So is time I post the last version of the share driver. Contains some 
parents fixes and avoids _fromDriver loop. It should work in all 
cases, but I don't use it any more. Is much better than DT but is just 
a was between a pure sql driver in order to leave the data structure 
as much as possible common to the original structure.



I even left the idea of having a join based driver. Yes it reducses 
the number of queries and ram usage on the php side,  but the time 
spent by the database server to join two large tables is longer than 
selecting ids and then the data as in the share driver.



The best option is actually an sql driver I am working on right now. 
Combines the two share and gallery tables in one, while still having 
separate the shares and gallery objects side.

I split just the Horde_Shares class into a driver pattern and included 
Ansel_Gallery_Share class definitions in each one of those files. I 
excluded the Ansel_Storage subclasses that you had originally included 
as I don't see the reason for those changes...



I attempted to make small changes to your original 'share' driver code 
to fit into this structure but haven't tested yet.

If we are trying to be to general again, it will be a step backward. 
We can still create a driver from this one to use a single table. Then 
the administrator can choose which one to use.

Another issue that I came across while trying to see the best way to 
split out the storage was that with your share driver, it is not 
possible to set a scope other than Ansel unless a SQL table already 
exists for the scope you want to use.   This might not be too big of 
an issue if the scope is another Horde application, where the table 
might exist (if the new share driver get's adopted) but it's not a 
great general solution.  Yes, the table can always be created,  but I 
think one of the strong points of Ansel is that you can use the API to 
create an image storage backend for any custom application, not just 
Horde applications - and having to create a new storage table in the 
Horde database for any new "application" is not a very good solution 
IMO.

I intentionally take in observations only the shares part. Not just 
because shares was the main performance problem but also was clean and 
so easier to understand and implement. I thought that going directly 
to a "true" storage driver will be probably to much for the beginning, 
as this driver does not need any Ansel core modifications. And as the 
share driver is now, can be even easily ported to other applications 
using shares and hierarchies.



I like to see a general driver too. In it, we can freely combine the 
list/countGalleries to one query (now we need two runs for each of 
them).

...forgot to mention, with the new share driver code you've included 
here, I don't see why we would even need to keep the datatree driver 
if everything works well with the new share driver you've provided and 
a conversion script can be written.  I just didn't want to merge the 
gallery and share tables.



Thanks again!

Ok. I looked at this a little closer this afternoon. Maybe I 
misunderstood your intentions, but I thought one of the reasons you 
wanted to do this was to enable you to use a custom "performance" 
driver that would combine the share and gallery data into one 
table...with the way your splitting the various classes, your leaving 
a lot of database specific code behind in the original Ansel.php file 
(in Ansel_Gallery and Ansel_Storage, and Ansel_Image - although the 
Ansel_Image code is probably ok as I don't think it touches the 
galleries table).



I think if we're going to go through the process of moving this to a 
"true" driver structure, we need to abstract out *all* the data 
storage for the vaious drivers.  Otherwise any drivers that use a 
different table structure (as you say your will) will need to override 
various Ansel_* methods.



Also, it seems like the Ansel_Share classes in the two different 
Storage "drivers" have different interfaces - some of which, at least, 
is due to the category methods no being present in the datatree 
version of the driver.



I *do* think it's worth moving to a more modular driver structure - 
but I think we need to rethink the design, maybe?  As always, I'm open 
to other's feedback / suggestions / slaps etc...

Thanks for the updated patch, I'm not ignoring this, but I'm waiting 
until I have more solid dedicated time to look at it, as it looks like 
there is still more work to do to abstract out storage...in 
particular, there are still queries being executed directly in 
Ansel_Gallery (and maybe Image, don't remember) and I'd prefer to have 
*all* the storage access abstracted out if we are going to do this...

Sure

I am unable to successfully apply the Ansel.php patch. Could you 
please upload an updated diff?



Thanks.

Fixed criteria. Now seems to work all actions I tried. I am going to 
create a demo install for my coworkers to try Ansel and create a bug 
and usability enhancement list.

Fix share driver. Should work in most cases. I am going to import my 
image files and test in deeply.

So a part of my patch is what you are looking for. You can omit the 
share driver itself as is not finished yet.

Agreed, and abstracting the storage has been on my TODO for some time now...

By combining we can remove 3 columns and have one join less. It will 
make data more compact and much faster and memory friendly which is 
always a strong point for large systems.



But your wish is not a problem for me, beside data being proccessed. 
Moving the storage mechanism on driver bases, will allow me to have my 
own performance driver and combined tables. When someone can still use 
the classic DT implementation or this share driver.



So the next step will be defenitly be to separate the storega classes 
with driver abstraction. Making my work easier possible to have 
different testing drivers without affecting Ansel core.

I haven't looked at your patch yet, but I will say that I'd prefer to 
keep the shares and galleries table separate.  IMO it makes for a 
cleaner implementation and keeps the share implementation/driver 
decoupled from the gallery objects.  But I'm definitely open to 
other's thoughts to convince me otherwise.

Going forward.



Modifying listGalleries use a direct join so we can get galleries in 
one db query we get a little more time 0,095 sec. but a much smaller 
memory footprint, only 1,572,864.



Modifying table stracture to merge shares and galleries table I fall 
down to 0,089 execution time. But uses a little more memory, 
1,835,008. The last is not logic, but I don't have time to dig in now.



I must go back and revision the code to clean up it and test other 
functionalities other then listing ( I can already CRUD 
sub-galleries). But before going foward I will like to get your 
comments/ideas.

Finally I had some time to merge my flat share driver from request # 
with some hierarchies logic from DataTree. The result is a new share 
driver for Ansel galleries speeding listing of my 86k galleries to an 
usable stage.



84k galleries, just counting and selecting galleries on the first page 
(9 galleries as default pref)

current Ansel : 11,17 sec, memory used 13,37 MB.

applied bug #6290: 7,67 sec, memory used 6,55 MB

**using this driver: 0,12 sec, memory used 4,98 MB**



The main performance benefit is of removing nested joins from datatree 
attributes table to a flat table. Where we have every attribute in a 
dedicated indexed column. So every gallery has only one row. I called 
it share driver. I am going to investigate more. As we still select 
galleries with two queries.