[#6814] problem of upgrade turba from 2.1.7 to 2.2
Summary problem of upgrade turba from 2.1.7 to 2.2
Queue Turba
Queue Version 2.2
Type Bug
State Resolved
Priority 1. Low
Owners
Requester david (at) tmv (dot) gov (dot) tw
Created 06/02/2008 (80 days ago)
Due
Updated 06/13/2008 (69 days ago)
Assigned 06/04/2008 (78 days ago)
Resolved 06/13/2008 (69 days ago)
Attachments
Milestone
Patch No

History
06/13/2008 Chuck Hagenbuch State ⇒ Resolved
 
06/06/2008 Michael Rubinsky Comment #29
Taken from Michael Rubinsky
Reply to this comment
Alright...I'm giving up on this.  I'm not able to reproduce the 
original issue while running through a full upgrade process.

If this is still occurring you are going to need to do some code 
tracing to figure out where it's breaking, unless any other devs are 
able to reproduce?
06/05/2008 Michael Rubinsky Comment #28 Reply to this comment
This script needs to be run while datatree is still selected as the 
share backend.
06/05/2008 david (at) tmv (dot) gov (dot) tw Comment #27 Reply to this comment
Try removing all your cache files.
Instead of these new errors : Notice: Undefined property: Horde_Share_sql::$_datatree in 
/usr/local/apache/horde-rc/turba/scripts/upgrades/2007-06-17_flatten_shares.php on line 
48

Notice: Trying to get property of non-object in 
/usr/local/apache/horde-rc/turba/scripts/upgrades/2007-06-17_flatten_shares.php on line 
49

Fatal error: Call to a member function getAssoc() on a non-object in 
/usr/local/apache/horde-rc/turba/scripts/upgrades/2007-06-17_flatten_shares.php on line 
53

And also found in horde log:
Jun 05 17:38:10 HORDE [debug] [horde] Connected to the following 
memcache servers:localhost:11211 [pid 3976 on line 123 of 
"/usr/local/apache/horde-rc/lib/Horde/Memcache.php"]
Jun 05 17:38:10 HORDE [debug] [horde] Error retrieving session data 
(id = 54a1e7290889dba256cfcb43ea9a19de) [pid 3976 on line 161 of 
"/usr/local/apache/horde-rc/lib/Horde/SessionHandler/memcache.php"]
Jun 05 17:38:10 HORDE [debug] [turba] Hook _horde_hook_share_init in 
application horde not called. [pid 3976 on line 1629 of 
"/usr/local/apache/horde-rc/lib/Horde.php"]
Jun 05 17:38:11 HORDE [debug] [turba] Hook 
_prefs_change_hook_addressbooks in application horde not called. [pid 
3976 on line 1629 of "/usr/local/apache/horde-rc/lib/Horde.php"]
Jun 05 17:38:11 HORDE [debug] [turba] Hook _prefs_change_hook_columns 
in application horde not called. [pid 3976 on line 1629 of 
"/usr/local/apache/horde-rc/lib/Horde.php"]
Jun 05 17:38:11 HORDE [debug] [turba] Hook 
_prefs_change_hook_search_sources in application horde not called. 
[pid 3976 on line 1629 of "/usr/local/apache/horde-rc/lib/Horde.php"]
Jun 05 17:38:11 HORDE [debug] [turba] Guest user does not have READ 
permission for imp [pid 3976 on line 818 of 
"/usr/local/apache/horde-rc/lib/Horde/Registry.php"]

thank you.
david

06/05/2008 Jan Schneider Comment #26 Reply to this comment
Try removing all your cache files.
06/05/2008 david (at) tmv (dot) gov (dot) tw Comment #25 Reply to this comment
Just for your info:
If i changed these setting in horde-3.2/config/conf.php
$conf['perms']['driver'] = 'sql';
$conf['share']['driver'] = 'sql';
and re-run
# php -f ./2007-06-17_flatten_shares.php , got these error

Fatal error: Perms::getPermissions(): The script tried to execute a 
method or access a property of an incomplete object. Please ensure 
that the class definition "DataTreeObject_Permission" of the object 
you are trying to operate on was loaded _before_ unserialize() gets 
called or provide a __autoload() function to load the class definition 
  in /usr/local/apache/horde-rc/lib/Horde/Perms.php on line 275

06/04/2008 david (at) tmv (dot) gov (dot) tw Comment #24 Reply to this comment

Typo,how stupid i am.
However i didn't use sql as share ($USE_SHARE="N" here) , so that
won't change the result,i thought.
No, but that should probably fix the other issue you mentioned
migrating to the SQL shares.
Should i use sql as share instead datatree ?
Well, that's totally your call. I would recommend SQL, with the
caveat that it's not as well tested, but it will give you large
performance benefits if you have a large user base.

I'll probably give it one more round of debugging/testing, but if I
can't duplicate the problem, your going to have to do some code
tracing to find out where the upgrade is failing.
Hello mike,i am very appreciate for your patience about this issue. And i'd like to know,should i have to change share from datatree to 
sql in horde/conf.php before run my upgrade shell again ? if i decide 
to use sql share for Horde-3.2.
thank you,mike.

david
06/04/2008 Michael Rubinsky Comment #23 Reply to this comment
Typo,how stupid i am.
However i didn't use sql as share ($USE_SHARE="N" here) , so that
won't change the result,i thought.
No, but that should probably fix the other issue you mentioned 
migrating to the SQL shares.
Should i use sql as share instead datatree ?
Well, that's totally your call. I would recommend SQL, with the caveat 
that it's not as well tested, but it will give you large performance 
benefits if you have a large user base.

I'll probably give it one more round of debugging/testing, but if I 
can't duplicate the problem, your going to have to do some code 
tracing to find out where the upgrade is failing.

06/04/2008 david (at) tmv (dot) gov (dot) tw Comment #22 Reply to this comment

php -f ./2.1_to_2.2_sql_schema.php
case $USE_SHARE in
y|Y)
    echo -n " processing ..."
    php -f ./2.1_to_2.2_add_sql_share_tables.sql
Why are you running a sql script through php?
Typo,how stupid i am. However i didn't use sql as share ($USE_SHARE="N" here) , so that 
won't change the result,i thought.
Should i use sql as share instead datatree ?

06/04/2008 Michael Rubinsky Comment #21 Reply to this comment
php -f ./2.1_to_2.2_sql_schema.php
case $USE_SHARE in
y|Y)
    echo -n " processing ..."
    php -f ./2.1_to_2.2_add_sql_share_tables.sql
Why are you running a sql script through php?

06/04/2008 david (at) tmv (dot) gov (dot) tw Comment #20 Reply to this comment
Here are the complete upgrade shell sripts:

#!/bin/bash
Wait () {
   echo -n "$STEP  Done!!"
   echo
}

HORDE="/usr/local/apache/horde-rc"
HD="/root/bin/upgrade-horde3.2"
S_DB="horde317"
D_DB="horde32
# Share not stable for SQL in Horde-3.2
USE_SHARE="N"

cd $HD

STEP="Duplicate Database"
echo "Duplicate Database from $S_DB to $D_DB"
echo "DROP DATABASE $D_DB;" >/tmp/Recreate_db.sql
echo "CREATE DATABASE $D_DB;" >>/tmp/Recreate_db.sql
mysql </tmp/Recreate_db.sql
/usr/local/bin/mysqldump --opt $S_DB >/tmp/$S_DB.sql
mysql $D_DB </tmp/$S_DB.sql
Wait

# HORDE
STEP="Horde upgrade scripts"
cd $HORDE/scripts/upgrades
mysql $D_DB<./3.1_to_3.2.mysql.sql
case $USE_SHARE in
y|Y)
    echo -n " processing ..."
    php -f ./convert_datatree_groups_to_sql.php
    php -f ./convert_datatree_perms_to_sql.php
    ;;
esac
Wait

# NAG
STEP="Nag upgrade scripts"
cd $HORDE/nag/scripts/upgrades
mysql $D_DB<./2.1_to_2.2.sql
php -f ./2006-04-18_add_creator_and_assignee_fields.php
mysql $D_DB<$HD/upg_nag.sql
case $USE_SHARE in
y|Y)
    echo -n " processing ..."
    php -f ./convert_datatree_shares_to_sql.php
    ;;
esac

Wait

# MNEMO
STEP="Mmemo upgrade scripts"
cd $HORDE/mnemo/scripts/upgrades
mysql $D_DB<./2.1_to_2.2.sql
case $USE_SHARE in
y|Y)
    echo -n " processing ..."
    php -f ./convert_datatree_shares_to_sql.php
    ;;
esac

Wait

# TREAN
STEP="Trean upgrade scripts"
cd $HORDE/trean/scripts/sql
mysql $D_DB<trean.sql
cd $HORDE/trean/scripts/upgrades
php -f ./2006-12-29_sql_bookmarks.php
php -f ./2006-12-29_cleanup.php
Wait

# WHUPS
STEP="Whups upgrade scripts"
cd $HORDE/whups/scripts/upgrades
mysql $D_DB <./2006-07-12_add_due_date.sql
mysql $D_DB <./2006-08-04_drop_subjectlist.sql
mysql $D_DB <./2007-01-16_add_queries.sql
mysql $D_DB <./2007-01-16_add_ticket_dates.sql
mysql $D_DB <./2007-01-16_drop_searches.sql
php -f ./2007-01-16_query_share_updates.php
php -f ./2007-01-16_cache_calculated_dates.php
mysql $D_DB <./2007-01-20_add_indexes.sql
mysql $D_DB <./2007-09-07_rename_listeners_table.sql
mysql $D_DB <./2007-12-09_add_slugs.sql
mysql $D_DB <./2007-12-12_additional_indexes.sql
mysql $D_DB <./2008-02-27_add_defaults.sql
case $USE_SHARE in
y|Y)
    echo -n " processing ..."
    mysql $D_DB <./2008-04-29_add_sql_share_tables.sql
    php -f ./convert_datatree_shares_to_sql.php
    ;;
esac
Wait

# KRONOLITH
STEP="Kronolith upgrade scripts"
cd $HORDE/kronolith/scripts/upgrades
mysql $D_DB<./2.1_to_2.2.sql
php -f ./2006-08-20_fix_recur_int_fields.mysql.sql
case $USE_SHARE in
y|Y)
    echo -n " processing ..."
    php -f ./convert_datatree_shares_to_sql.php
    ;;
esac
Wait

# TURBA
STEP="Turba upgrade scripts"
cd $HORDE/turba/scripts/upgrades
php -f ./2007-06-17_flatten_shares.php
php -f ./2.1_to_2.2_sql_schema.php
case $USE_SHARE in
y|Y)
    echo -n " processing ..."
    php -f ./2.1_to_2.2_add_sql_share_tables.sql
    php -f ./convert_datatree_shares_to_sql.php
    ;;
esac
Wait

# ANSEL
STEP="Ansel upgrade scripts"
cd $HORDE/ansel/scripts/upgrades
php -f ./2006-11-27_sql_images.php
php -f ./2006-11-27_datatree_to_sql_renames.php
php -f ./2006-11-27_fix_default_images.php
php -f ./2006-11-27_cleanup.php
mysql $D_DB <./2008_01_07_add_galleries.sql
php -f ./2008_01_07_sql_galleries.php
php -f ./2008_01_28_clean_datatree.php
mysql $D_DB <./2008_02_04_add_images_and_slugs.sql
php -f ./2008_02_04_add_initial_image_counts.php
mysql $D_DB <./2008_02_29_fix_category_defaults.sql
mysql $D_DB <./2008_03_18_add_gallery_age.sql
mysql $D_DB <./2008_03_19_add_gallery_download.sql
case $USE_SHARE in
y|Y)
    echo -n " processing ..."
    mysql $D_DB <./2008-04-28_add_sql_share_tables.sql
    php -f ./2008-04-28_sql_share_migration.php
    ;;
esac
Wait

# INGO
STEP="Ingo"
cd  $HORDE/ingo/scripts/sql
mysql $D_DB<./ingo.sql
cd $HORDE/ingo/scripts/upgrades
php -f ./convert_prefs_to_sql.php </etc/mail/login
cd $HD
mysql $D_DB <./convert_spam.sql
cd $HORDE/ingo/scripts/upgrades
case $USE_SHARE in
y|Y)
    echo -n " processing ..."
    php -f ./convert_datatree_shares_to_sql.php
    ;;
esac

# For ingo 2.x
#php -f ./convert_imp_filters.php </etc/mail/login
Wait

echo "Upgrade completed !"
06/04/2008 david (at) tmv (dot) gov (dot) tw Comment #19 Reply to this comment
What driver are you using for Shares?  If you switched to the SQL
driver, did you run the turba upgrade *after* you ran the SQL share
migration script?
Allways datatree for share. Because there is error while execute 
convert_datatree_shares_to_sql.php ,so i stay using datatree as share 
(see 
http://lists.horde.org/archives/turba/Week-of-Mon-20080602/005876.html 
).
Here are the scripts i run:
1. horde/scripts/upgrades/3.1_to_3.2.mysql.sql
2. turba/scripts/upgrades/2007-06-17_flatten_shares.php
3. turba/scripts/upgrades/2.1_to_2.2_sql_schema.php

I setup a Horde 3.1/Turba 2.1 install and ran a full conversion with
fresh data.  The only way I can reproduce this is by switching to the
SQL share driver and then logging in before I run the sql share
migration script.  In every other case I tried, the maintenance tasks
run perfectly.
06/04/2008 Michael Rubinsky Comment #18
State ⇒ Feedback
Reply to this comment
What driver are you using for Shares?  If you switched to the SQL 
driver, did you run the turba upgrade *after* you ran the SQL share 
migration script?

I setup a Horde 3.1/Turba 2.1 install and ran a full conversion with 
fresh data.  The only way I can reproduce this is by switching to the 
SQL share driver and then logging in before I run the sql share 
migration script.  In every other case I tried, the maintenance tasks 
run perfectly.
06/04/2008 Jan Schneider State ⇒ Assigned
 
06/04/2008 david (at) tmv (dot) gov (dot) tw Comment #17 Reply to this comment
Hello jan,
2.1_to_2.2_sql_schema.php is fixed,however it doesn't matter about 
this problem.
I thought it should be stay open at this moment.
06/04/2008 Jan Schneider State ⇒ Resolved
 
06/03/2008 david (at) tmv (dot) gov (dot) tw Comment #16 Reply to this comment
...this one should at least give you the root cause of your DB errors...
Got it ,and no more error return. thank you.
06/03/2008 Michael Rubinsky Comment #15 Reply to this comment
...this one should at least give you the root cause of your DB errors...
06/03/2008 CVS Commit Comment #14 Reply to this comment
06/02/2008 david (at) tmv (dot) gov (dot) tw Comment #13 Reply to this comment
Another error message return,with the newly 2.1_to_2.2_sql_schema.php. Fatal error: Call to undefined method DB_Error::getAssoc() in 
/usr/local/apache/horde-rc/turba/scripts/upgrades/2.1_to_2.2_sql_schema.php on 
line 140

06/02/2008 david (at) tmv (dot) gov (dot) tw Comment #12 Reply to this comment
I"m sorry, are you saying the problem still persists, or that it is
working for you?
The problem still there.i mean in both turba-addressbooks and  imp-search_fields,preference UI are therefore break.Because of 
"localsql" didn't convert to  user_a.

06/02/2008 Michael Rubinsky Comment #11 Reply to this comment
I"m sorry, are you saying the problem still persists, or that it is 
working for you?
06/02/2008 david (at) tmv (dot) gov (dot) tw Comment #10 Reply to this comment
In the meantime, what happens if you delete a user's
turba_maintenance_tasks pref, make sure the user is logged out, clear
any session caches etc... and then have the user log back in?
Notthing changed for the related preference,and  turba_maintenance_tasks back again.

06/02/2008 david (at) tmv (dot) gov (dot) tw Comment #9 Reply to this comment
The preference has a little change,and much closer to what i wish,if
i *disable* token system in horde-3.2 for this converation .
Are you saying that the prefs are only correctly updated if the user
logs in for the first time when form tokens are disabled? If they are
enabled, the prefs do not update?  That is bizarre.
Sorry,my fault. That doesn't matter what token system changed.

06/02/2008 Michael Rubinsky Comment #8 Reply to this comment
The preference has a little change,and much closer to what i wish,if
i *disable* token system in horde-3.2 for this converation .
Are you saying that the prefs are only correctly updated if the user 
logs in for the first time when form tokens are disabled? If they are 
enabled, the prefs do not update?  That is bizarre.

06/02/2008 david (at) tmv (dot) gov (dot) tw Comment #7 Reply to this comment
The preference has a little change,and much closer to what i wish,if
i *disable* token system in horde-3.2 for this converation .
All the preferences for turba-2.2 were got since user_a login. Another  report for turba_maintenance_tasks deleted will coming soon.
06/02/2008 david (at) tmv (dot) gov (dot) tw Comment #6 Reply to this comment
The preference has a little change,and much closer to what i wish,if i 
*disable* token system in horde-3.2 for this converation .

1. turba-addressbooks
pref_uid         pref_scope        pref_name         pref_value  (2.1.7)
========  ==========  =========    ==========
user_a          turba                 addressbooks     localsql
                                                                       
local_ldap
                                                                       
localsql:dd38b1b284f79796351d0c5fda5fd80f
                                                                       
localsql:55dae36c62c8e10ef9583cf522299ff1
pref_uid         pref_scope        pref_name         pref_value  (2.2)
========  ==========  =========    ==========
user_a          turba                 addressbooks     localsql
                                                                       
local_ldap
                                                                       
dd38b1b284f79796351d0c5fda5fd80f
                                                                       
55dae36c62c8e10ef9583cf522299ff1
2. turba-columns
pref_uid         pref_scope        pref_name         pref_value  (2.1.7)
========  ==========  =========    ==========
user_a          turba                 columns             localsql        email
                                                                       
local_ldap        email
                                                                       
localsql:dd38b1b284f79796351d0c5fda5fd80f        email
                                                                       
localsql:55dae36c62c8e10ef9583cf522299ff1        email
pref_uid         pref_scope        pref_name         pref_value  (2.2)
========  ==========  =========    ==========
user_a          turba                 columns             user_a        email
                                                                       
local_ldap        email
                                                                       
dd38b1b284f79796351d0c5fda5fd80f        email
                                                                       
55dae36c62c8e10ef9583cf522299ff1        email

3. imp-search_fields
pref_uid         pref_scope        pref_name         pref_value  (2.1.7)
========  ==========  =========    ==========
user_a          imp                    search_fields     localsql        name
                                                                       
local_ldap          name
                                                                       
localsql:dd38b1b284f79796351d0c5fda5fd80f name

                                                                       
localsql:55dae36c62c8e10ef9583cf522299ff1 name
pref_uid         pref_scope        pref_name         pref_value  (2.2)
========  ==========  =========    ==========
user_a          imp                    search_fields     localsql        name
                                                                       
local_ldap          name
                                                                       
localsql:dd38b1b284f79796351d0c5fda5fd80f name

                                                                       
localsql:55dae36c62c8e10ef9583cf522299ff1 name
4. imp-search_sources
pref_uid         pref_scope        pref_name         pref_value  (2.1.7)
========  ==========  =========    ==========
user_a          imp                    search_sources  localsql
                                                                       
local_ldap
                                                                       
localsql:dd38b1b284f79796351d0c5fda5fd80f
                                                                     
localsql:55dae36c62c8e10ef9583cf522299ff1
pref_uid         pref_scope        pref_name         pref_value  (2.2)
========  ==========  =========    ==========
user_a          imp                    search_sources  user_a
                                                                       
local_ldap
                                                                       
dd38b1b284f79796351d0c5fda5fd80f
                                                                       
55dae36c62c8e10ef9583cf522299ff1
06/02/2008 Michael Rubinsky Comment #5 Reply to this comment
Fatal error: Cannot redeclare getbareemail() (previously declared in
/usr/local/apache/horde-rc/turba/scripts/upgrades/2.1_to_2.2_sql_schema.php:285) in /usr/local/apache/horde-rc/turba/scripts/upgrades/2.1_to_2.2_sql_schema.php on 
line
285
This error was due to a function inadvertently being declared inside 
another function.  This has been corrected by the patch mentioned 
below.  I'm not sure I see how that would be the cause of the original 
issue in the bug, but it wouldn't hurt to try the upgrade process 
again on the original database and see if that helps?
06/02/2008 CVS Commit Comment #4 Reply to this comment
06/02/2008 david (at) tmv (dot) gov (dot) tw Comment #3 Reply to this comment
Forgot to mention what scripts i run & return code on it.
#cd turba/scripts/upgrades
#php -f ./2007-06-17_flatten_shares.php
Successfully flattened shared address books.
#php -f ./2.1_to_2.2_sql_schema.php
            .
            .
            .
[   OK   ] CREATE INDEX turba_firstname_idx ON turba_objects 
(object_firstname)
[   OK   ] CREATE INDEX turba_lastname_idx ON turba_objects (object_lastname)
[   OK   ] ALTER TABLE turba_objects ADD COLUMN object_photo BLOB
[   OK   ] ALTER TABLE turba_objects ADD COLUMN object_logo BLOB
[   OK   ] Contact name fields parsed.
[   OK   ] Home address fields parsed.

Fatal error: Cannot redeclare getbareemail() (previously declared in 
/usr/local/apache/horde-rc/turba/scripts/upgrades/2.1_to_2.2_sql_schema.php:285) in /usr/local/apache/horde-rc/turba/scripts/upgrades/2.1_to_2.2_sql_schema.php on line 
285
06/02/2008 Michael Rubinsky Comment #2 Reply to this comment
A preliminary test with the only old database data I have around at 
the moment does not reproduce this. I'll test more in-depth once I set 
up a Turba 2.1.7 install to generate fresh data...

In the meantime, what happens if you delete a user's 
turba_maintenance_tasks pref, make sure the user is logged out, clear 
any session caches etc... and then have the user log back in?
06/02/2008 Jan Schneider State ⇒ Assigned
Assigned to Michael Rubinsky
 
06/02/2008 david (at) tmv (dot) gov (dot) tw Comment #1
Patch ⇒
Milestone ⇒
Queue ⇒ Turba
Summary ⇒ problem of upgrade turba from 2.1.7 to 2.2
Type ⇒ Bug
Priority ⇒ 1. Low
State ⇒ Unconfirmed
Reply to this comment
After upgrade localsql backend from 2.1.7 to 2.2,some of prefences did 
not work/map successfully,eg.   
imp[search_source],imp[search_fields],turba[addressbooks].

For example, after run done 2007-06-17_flatten_shares.php ,and then 
login to horde-3.2,the imp[search_source] preference looks like this:
pref_uid         pref_scope        pref_name         pref_value
========  ==========  =========    ==========
user_a          turba                 addressbooks     localsql
                                                                       
local_ldap
                                                                   
localsql:dd38b1b284f79796351d0c5fda5fd80f
                                                                       
localsql:55dae36c62c8e10ef9583cf522299ff1

  However,as i know,if it work ,that should be convert to,
pref_uid         pref_scope        pref_name         pref_value
========  ==========  =========    ==========
user_a          turba                 addressbooks     user_a
                                                                       
local_ldap
                                                                       
dd38b1b284f79796351d0c5fda5fd80f
                                                                       
55dae36c62c8e10ef9583cf522299ff1

but,it doesn't. Therefore some of address book break in turba/UI.
FYI,here are the turba's conf.php in Horde-3.2/turba:
$conf['client']['addressbook'] = 'localsql';
$conf['shares']['source'] = 'localsql';

and prefs.php
$_prefs['addressbooks'] = array(
     'value' => "localsql\nlocal_ldap",
     'locked' => false,
     'shared' => false,
     'type' => 'implicit',
);
$_prefs['columns'] = array(
     'value' => "localsql\temail\nlocal_ldap\temail",
     'locked' => false,
     'shared' => false,
     'type' => 'implicit',
);
$_prefs['default_dir'] = array(
     'value' => 'localsql',
     'locked' => false,
     'shared' => true,
     'type' => 'select',
     'desc' => _("This will be the default address book when adding or 
importing contacts."),
);

P.S There is a entry in prefs after upgrade completed and then login 
to hore-3.2.
user_a          turba                turba_maintenance_tasks 
a:2:{i:0;s:12:"upgradeprefs";i:1;s:12:"upgradelists";}