6.0.0-git
2019-03-24

[#6109] optimized sql Share driver
Summary optimized sql Share driver
Queue Horde Framework Packages
Queue Version HEAD
Type Enhancement
State Resolved
Priority 2. Medium
Owners chuck (at) horde (dot) org, mrubinsk (at) horde (dot) org
Requester duck (at) obala (dot) net
Created 2008-01-14 (4087 days ago)
Due
Updated 2008-04-30 (3980 days ago)
Assigned 2008-04-13 (3997 days ago)
Resolved 2008-04-30 (3980 days ago)
Milestone 3.2
Patch Yes

History
2008-04-30 03:34:27 Chuck Hagenbuch Comment #35
State ⇒ Resolved
Reply to this comment
This is done and a pretty darn solid beta if I say so myself. Huge 
thanks to Duck for getting this started and to Michael R. for all the 
help testing and for writing the migration scripts.
2008-04-19 18:39:25 Michael Rubinsky Comment #34 Reply to this comment
Also added upgrade files to Genie as well...
2008-04-17 20:05:29 Chuck Hagenbuch Comment #33
Assigned to Michael Rubinsky
Taken from Duck
Taken from Horde DevelopersHorde Developers
Reply to this comment
No great shakes, but yup, it's done. Now we test :)
2008-04-17 19:36:50 Michael Rubinsky Comment #32 Reply to this comment
Looks like all we have left now is the conf.xml entry. Chuck, you said 
you have it done, right?




2008-04-17 16:03:47 Chuck Hagenbuch Comment #31 Reply to this comment
As I remember, even when shares are disabled, the default user's
share is still created. Like the default user calendar. Performing
the select joins the users/group table. But those two tables will be
always empty. So I add a check to avoid the JOIN.
Shouldn't the check be the other way then? Right now it only joins in 
those tables if no_sharing is enabled? Actually I think it should be 
removed, since even if users aren't allowed to control who sees their 
shares, someone might create system shares that everyone sees.
2008-04-17 15:27:01 Duck Comment #30 Reply to this comment
Duck-
can you explain what the check in Horde_Share_sql::_getShareCriteria
against $conf['share']['no_sharing'] is for? It doesn't appear in the
other backends, and I'm inclined to remove it here also unless
there's a reason I'm missing.
As I remember, even when shares are disabled, the default user's share 
is still created. Like the default user calendar. Performing the 
select joins the users/group table. But those two tables will be 
always empty. So I add a check to avoid the JOIN.
2008-04-17 14:44:32 Chuck Hagenbuch Comment #29
Assigned to Duck
Reply to this comment
Duck-

can you explain what the check in Horde_Share_sql::_getShareCriteria 
against $conf['share']['no_sharing'] is for? It doesn't appear in the 
other backends, and I'm inclined to remove it here also unless there's 
a reason I'm missing.
2008-04-17 02:34:29 Michael Rubinsky Comment #28 Reply to this comment
Ingo migration and UPGRADING done that leaves:
- datatree-to-sql upgrade scripts for Mnemo, Nag
- conf.xml entry (done, just waiting for full functionality)
- docs/UPGRADING entries Mnemo, Nag
2008-04-16 23:07:01 Michael Rubinsky Comment #27 Reply to this comment
Kronolith migration and UPGRADING done
2008-04-16 22:21:40 Michael Rubinsky Comment #26 Reply to this comment
Turba migrations and UPGRADING is done (that should be it for Turba)

So..to update the summary:
Ingo table defs are done. That leaves, to summarize:

- datatree-to-sql upgrade scripts for Ingo, Kronolith, Mnemo, Nag
- conf.xml entry (done, just waiting for full functionality)
- docs/UPGRADING entries Ingo, Kronolith, Mnemo, Nag
I'll start moving the migration script to the other apps and post as I 
complete them.
2008-04-14 20:20:41 Chuck Hagenbuch Comment #25 Reply to this comment
Ingo table defs are done. That leaves, to summarize:



- datatree-to-sql upgrade scripts for Ingo, Kronolith, Mnemo, Nag, Turba

- conf.xml entry (done, just waiting for full functionality)

- docs/UPGRADING entries
2008-04-14 18:11:43 Chuck Hagenbuch Comment #24 Reply to this comment
Table defs and table def upgrades are done for Nag and Mnemo.
2008-04-13 19:01:45 Michael Rubinsky Comment #23 Reply to this comment
Table defs are done for Turba - still need to work on the conversion script


2008-04-13 17:18:58 Michael Rubinsky Comment #22 Reply to this comment
I'll try to work on what's required for Turba...
2008-04-13 05:01:06 Chuck Hagenbuch Comment #21 Reply to this comment
table defs have been added for Kronolith; still need the upgrade 
scripts (both for creating the tables and migration).
2008-04-13 02:59:35 Chuck Hagenbuch Comment #20
Assigned to Chuck Hagenbuch
Assigned to Horde DevelopersHorde Developers
State ⇒ Assigned
Reply to this comment
Code is working for me on SQLite now as well as MySQL. Still to do:

- sql scripts for Ingo, Kronolith (done), Mnemo, Nag, Turba

- datatree-to-sql upgrade scripts for Ingo, Kronolith, Mnemo, Nag, Turba

- conf.xml entry (done, just waiting for full functionality)

- docs/UPGRADING entries



I'm working at this, but help is certainly welcome. If anyone else 
wants to/can jump in here, please just add a note here so we don't 
overlap.
2008-04-07 02:14:42 Chuck Hagenbuch Comment #19 Reply to this comment
Committed along with making the code parse under PHP 4. Still need to 
remove the REPLACE INTO statement so it works in more than MySQL.
2008-04-06 18:05:14 Duck Comment #18
State ⇒ Accepted
New Attachment: sql.diff Download
Reply to this comment
After framework update I noticed that I didn't update the code here.

- Fix unsetting the permissions for guest/default/creator.

- Don't avoid permission check with attributes.
2008-03-08 09:27:39 Duck Comment #17 Reply to this comment
Okay, so we just need a 'sql' option with no extra params in the
$conf['shares'] array?
yes
2008-03-07 20:45:42 Chuck Hagenbuch Comment #16 Reply to this comment
Okay, so we just need a 'sql' option with no extra params in the 
$conf['shares'] array?
2008-03-07 09:50:44 Duck New Attachment: kronoiths.shares.sql Download
 
2008-03-07 09:50:21 Duck Comment #15 Reply to this comment
I've committed the base driver code. Can you post what the config
settings look like for this driver?
No configuration options is needed. It uses the global horde sql 
configuration (with split write support). All you need is to create 
additional tables per very share group. Like is already attached here 
in horde.shares.genie.sql. You must just change the prefix by scope 
name and add attributes columns if needed. I am attaching another 
example for Kronolith. Structures for each application can be bundled 
with application sql structure.
2008-03-07 04:32:05 Chuck Hagenbuch Comment #14
State ⇒ Feedback
Reply to this comment
I've committed the base driver code. Can you post what the config 
settings look like for this driver?
2008-03-06 00:24:01 Jan Schneider Comment #13
Priority ⇒ 2. Medium
Milestone ⇒ 3.2
Patch ⇒ Yes
Reply to this comment
Should go into 3.2 as an experimental driver.
2008-02-19 18:38:31 Chuck Hagenbuch Deleted Original Message
 
2008-02-19 18:34:49 Duck Comment #12
New Attachment: sql[7].php Download
Reply to this comment
I am uploading a new version that fixes a little bug in   
_getShareCriteria I discovered while trying to port it to Ansel.



Horde_Share_Object_sqlmini is us just a hack, and can be completely 
omitted. Is a fact that all applications fetches the list only to get 
the share id, name and owner. All other actions are done on a single 
share object. Only Kronolith uses the results from listShares for 
permission checking. So it would be better to remove 
Horde_Share_Object_sqlmini and add a method to return an indexed array 
for share names/owners as we already discussed some time ago in the 
list.
2008-02-19 16:13:20 Chuck Hagenbuch Deleted Original Message
 
2008-02-19 10:38:23 Duck Comment #11 Reply to this comment
Just to be clear, shareimport-genie.sql is replaced by
horde.shares.genie.sql, right?
yes
2008-02-19 03:16:12 Chuck Hagenbuch State ⇒ Accepted
 
2008-02-19 03:16:04 Chuck Hagenbuch Comment #10 Reply to this comment
Just to be clear, shareimport-genie.sql is replaced by 
horde.shares.genie.sql, right?


2008-02-19 03:11:11 Chuck Hagenbuch Deleted Original Message
 
2008-02-19 03:11:04 Chuck Hagenbuch Deleted Original Message
 
2008-02-19 03:10:54 Chuck Hagenbuch Deleted Original Message
 
2008-02-19 03:10:43 Chuck Hagenbuch Deleted Original Message
 
2008-02-19 03:10:10 Chuck Hagenbuch Deleted Original Message
 
2008-02-19 03:09:37 Chuck Hagenbuch Deleted Original Message
 
2008-01-25 19:37:52 Duck Comment #9
New Attachment: sql[6].php
Reply to this comment
fix harcoded DT permissons and param_extra query
2008-01-19 11:31:41 Duck Comment #8
New Attachment: sql[5].php
Reply to this comment
This is maybe a hack. But introducing an dedicated object with minimal 
data (id,name,owner) to work with share listing in panels etc lower 
the ram usage with 5.200 entries from 18M to 7M.



Tried with indexed arrays and noticed that uses even 1M less.
2008-01-15 18:36:31 Duck Comment #7
New Attachment: sql[4].php
Reply to this comment
Do not sort shares by name inside PHP as SQL can do it so. Now I 
finaly can list all 5.200 wish-list without timeout.  With this I 
think is the last thing I can do without share api changes. Now I am 
just wainting mysql to use indexes for bitwise operators.



Duck
2008-01-14 18:55:22 Duck Comment #6
New Attachment: sql[3].php
Reply to this comment
- Added __sleep and __wakeup methods

- Add perm_extra columns to mark if the share has any additional user 
or group definition. So we avoid queries for shares that don't have 
them set. In a lot of occasions we need only one query to get the 
entire share list.
2008-01-14 17:52:17 Duck Comment #5
New Attachment: sql[2].php
Reply to this comment
Make a local copy of listShares() to do a query less -  do not query 
the db for ids and then select the data, but pick the data directly
2008-01-14 15:09:34 Duck Comment #4
New Attachment: horde.shares.genie.sql Download
Reply to this comment
Update transition script works for kronolith, nag, mnemo and genie. 
Create tables, transfers data, and delete old datatree entries. Just 
replace the string genie with appropriate share name. Unfortunately 
uses subselect so it will probably work with mysql only.
2008-01-14 13:56:37 Duck Comment #3
New Attachment: sql[1].php
Reply to this comment
Fix _getShareCriteria().



Seems to work with kronolith, nag, mnemo and genie.
2008-01-14 11:44:22 Duck Comment #2
New Attachment: shareimport-genie.sql
Reply to this comment
Genie DT to SQL conversion sql subselect script.

With little modification it should work for all shares groups.
2008-01-14 11:42:42 Duck Comment #1
Type ⇒ Enhancement
State ⇒ New
Priority ⇒ 1. Low
Summary ⇒ optimized sql Share driver
Queue ⇒ Horde Framework Packages
New Attachment: sql.php
Reply to this comment
- flat attribute table

- data splited per tables

- split read support

Saved Queries