Summary | Ansel flat shares with hierarchies |
Queue | Ansel |
Type | Enhancement |
State | Resolved |
Priority | 1. Low |
Owners | mrubinsk (at) horde (dot) org |
Requester | duck (at) obala (dot) net |
Created | 02/20/2008 (6348 days ago) |
Due | |
Updated | 05/09/2008 (6269 days ago) |
Assigned | 04/16/2008 (6292 days ago) |
Resolved | 05/09/2008 (6269 days ago) |
Milestone | |
Patch | No |
State ⇒ Resolved
shares. Hopefully, we can migrate Trean to use the same driver.
Thanks to Duck for all the work on these drivers.
Bye-Bye Datatree :)
require other api changes or would it be another "drop in"
Ansel_Share driver? If the former, unless the performance is earth
shatteringly different, I'd propose to save that one for Horde 4 and
we can finally "lock in" on a driver architecture for Ansel 1.0
Ansel::createGallery() and Ansel_Gallery::save() methods should be
corrected. But can be moved to the storage driver itself.
Structural changes soul be done after DT will be left behind and we
will be able to avoid some ft logic.
New Attachment: ansel.share.sql
Could you possibly either add some documentation to the driver file,
or even better, provide a sql script to create the database table(s)
needed for your driver?
sql driver.
Could you possibly either add some documentation to the driver file,
or even better, provide a sql script to create the database table(s)
needed for your driver?
Ansel_Storage_share into the main-line driver (Since Ansel_Storage
*creates* Ansel_Shares_* it would be impossible to get the sub class
with the current code structure). Then I'll make the few needed
changes in the share driver and drop into lib/Shares.
Regarding your "best bet" sql driver - is this something that will
require other api changes or would it be another "drop in" Ansel_Share
driver? If the former, unless the performance is earth shatteringly
different, I'd propose to save that one for Horde 4 and we can finally
"lock in" on a driver architecture for Ansel 1.0
Any other thoughts on this from anyone?
New Attachment: share[2].php
parents fixes and avoids _fromDriver loop. It should work in all
cases, but I don't use it any more. Is much better than DT but is just
a was between a pure sql driver in order to leave the data structure
as much as possible common to the original structure.
I even left the idea of having a join based driver. Yes it reducses
the number of queries and ram usage on the php side, but the time
spent by the database server to join two large tables is longer than
selecting ids and then the data as in the share driver.
The best option is actually an sql driver I am working on right now.
Combines the two share and gallery tables in one, while still having
separate the shares and gallery objects side.
Ansel_Gallery_Share class definitions in each one of those files. I
excluded the Ansel_Storage subclasses that you had originally included
as I don't see the reason for those changes...
I attempted to make small changes to your original 'share' driver code
to fit into this structure but haven't tested yet.
We can still create a driver from this one to use a single table. Then
the administrator can choose which one to use.
State ⇒ Feedback
Priority ⇒ 1. Low
split out the storage was that with your share driver, it is not
possible to set a scope other than Ansel unless a SQL table already
exists for the scope you want to use. This might not be too big of
an issue if the scope is another Horde application, where the table
might exist (if the new share driver get's adopted) but it's not a
great general solution. Yes, the table can always be created, but I
think one of the strong points of Ansel is that you can use the API to
create an image storage backend for any custom application, not just
Horde applications - and having to create a new storage table in the
Horde database for any new "application" is not a very good solution
IMO.
because shares was the main performance problem but also was clean and
so easier to understand and implement. I thought that going directly
to a "true" storage driver will be probably to much for the beginning,
as this driver does not need any Ansel core modifications. And as the
share driver is now, can be even easily ported to other applications
using shares and hierarchies.
I like to see a general driver too. In it, we can freely combine the
list/countGalleries to one query (now we need two runs for each of
them).
here, I don't see why we would even need to keep the datatree driver
if everything works well with the new share driver you've provided and
a conversion script can be written. I just didn't want to merge the
gallery and share tables.
Thanks again!
misunderstood your intentions, but I thought one of the reasons you
wanted to do this was to enable you to use a custom "performance"
driver that would combine the share and gallery data into one
table...with the way your splitting the various classes, your leaving
a lot of database specific code behind in the original Ansel.php file
(in Ansel_Gallery and Ansel_Storage, and Ansel_Image - although the
Ansel_Image code is probably ok as I don't think it touches the
galleries table).
I think if we're going to go through the process of moving this to a
"true" driver structure, we need to abstract out *all* the data
storage for the vaious drivers. Otherwise any drivers that use a
different table structure (as you say your will) will need to override
various Ansel_* methods.
Also, it seems like the Ansel_Share classes in the two different
Storage "drivers" have different interfaces - some of which, at least,
is due to the category methods no being present in the datatree
version of the driver.
I *do* think it's worth moving to a more modular driver structure -
but I think we need to rethink the design, maybe? As always, I'm open
to other's feedback / suggestions / slaps etc...
Priority ⇒ 2. Medium
until I have more solid dedicated time to look at it, as it looks like
there is still more work to do to abstract out storage...in
particular, there are still queries being executed directly in
Ansel_Gallery (and maybe Image, don't remember) and I'd prefer to have
*all* the storage access abstracted out if we are going to do this...
State ⇒ Assigned
New Attachment: ansel-split.tgz
State ⇒ Feedback
please upload an updated diff?
Thanks.
New Attachment: share[1].php
create a demo install for my coworkers to try Ansel and create a bug
and usability enhancement list.
New Attachment: share.php
image files and test in deeply.
share driver itself as is not finished yet.
keep the shares and galleries table separate. IMO it makes for a
cleaner implementation and keeps the share implementation/driver
decoupled from the gallery objects. But I'm definitely open to
other's thoughts to convince me otherwise.
make data more compact and much faster and memory friendly which is
always a strong point for large systems.
But your wish is not a problem for me, beside data being proccessed.
Moving the storage mechanism on driver bases, will allow me to have my
own performance driver and combined tables. When someone can still use
the classic DT implementation or this share driver.
So the next step will be defenitly be to separate the storega classes
with driver abstraction. Making my work easier possible to have
different testing drivers without affecting Ansel core.
down to 0,089 execution time. But uses a little more memory,
1,835,008. The last is not logic, but I don't have time to dig in now.
keep the shares and galleries table separate. IMO it makes for a
cleaner implementation and keeps the share implementation/driver
decoupled from the gallery objects. But I'm definitely open to
other's thoughts to convince me otherwise.
Modifying listGalleries use a direct join so we can get galleries in
one db query we get a little more time 0,095 sec. but a much smaller
memory footprint, only 1,572,864.
Modifying table stracture to merge shares and galleries table I fall
down to 0,089 execution time. But uses a little more memory,
1,835,008. The last is not logic, but I don't have time to dig in now.
I must go back and revision the code to clean up it and test other
functionalities other then listing ( I can already CRUD
sub-galleries). But before going foward I will like to get your
comments/ideas.
Priority ⇒ 1. Low
State ⇒ New
New Attachment: ansel-flat.tgz
Queue ⇒ Ansel
Summary ⇒ Ansel flat shares with hierarchies
Type ⇒ Enhancement
with some hierarchies logic from DataTree. The result is a new share
driver for Ansel galleries speeding listing of my 86k galleries to an
usable stage.
84k galleries, just counting and selecting galleries on the first page
(9 galleries as default pref)
current Ansel : 11,17 sec, memory used 13,37 MB.
applied
bug #6290: 7,67 sec, memory used 6,55 MB**using this driver: 0,12 sec, memory used 4,98 MB**
The main performance benefit is of removing nested joins from datatree
attributes table to a flat table. Where we have every attribute in a
dedicated indexed column. So every gallery has only one row. I called
it share driver. I am going to investigate more. As we still select
galleries with two queries.