db_migrate and incorrect charset handling

I been testing data migration from framework3 to h4 using db

Wed, 02 Mar 2011 12:02:40 +0000

I been testing data migration from framework3 to h4 using db_migrate. Either it is mysql which is case insentive or it is the migration script, but it seems as if you cannot add tags to rampage_tags if tags differ only by their case. Eg. these tags are consider the same: TYÖ työ If those tags exists in old data then db_migrate will fail with error: QUERY FAILED: Duplicate entry 'työ' for key 'rampage_tags_tag_name' INSERT INTO `rampage_tags` (tag_name) VALUES ('työ')

Changes have been made in Git for this ticket: Use Horde_St

Fri, 04 Mar 2011 14:37:52 +0000

Changes have been made in Git for this ticket: Use Horde_String::lower Possibly fixes Bug: 9617 1 files changed, 1 insertions(+), 1 deletions(-) http://git.horde.org/horde-git/-/commit/ae5714cbb917c43e184f942db0e1b5f3197f679f

Can you try what I just committed?

Fri, 04 Mar 2011 14:38:29 +0000

Can you try what I just committed?

> Can you try what I just committed? I tried, but script

Fri, 04 Mar 2011 14:57:31 +0000

> Can you try what I just committed? I tried, but script fails with the same error.

Changes have been made in Git for this ticket: Need to conv

Fri, 04 Mar 2011 19:36:48 +0000

Changes have been made in Git for this ticket: Need to convert from database's charset before comparing Bug: 9617 1 files changed, 1 insertions(+), 1 deletions(-) http://git.horde.org/horde-git/-/commit/0f36ff69b64c96e3ab91d6f479fa34cb15a451a9

> Changes have been made in Git for this ticket: > > Need

Fri, 04 Mar 2011 21:15:24 +0000

> Changes have been made in Git for this ticket: > > Need to convert from database's charset before comparing > Bug: 9617 > > 1 files changed, 1 insertions(+), 1 deletions(-) > http://git.horde.org/horde-git/-/commit/0f36ff69b64c96e3ab91d6f479fa34cb15a451a9 Attached patch seemed to fix it for me. Another bug has appeared. Output from debug log: DEBUG: SQL SELECT user_id, user_name FROM `rampage_users` WHERE user_name IN ('ntllt') DEBUG: SQL QUERY FAILED: Duplicate entry 'ntllt' for key rampage_users_user_name' INSERT INTO `rampage_users` (user_name) VALUES ('ntllt')

Changes have been made in Git for this ticket: Need to conv

Fri, 04 Mar 2011 21:49:52 +0000

Changes have been made in Git for this ticket: Need to convert to utf-8 when reading the category before tagging. Bug: 9617 1 files changed, 2 insertions(+), 2 deletions(-) http://git.horde.org/horde-git/-/commit/3baf0b99425470dfdd77e02de1da4f32bf4851ff

> Attached patch seemed to fix it for me. It should be

Fri, 04 Mar 2011 21:52:55 +0000

> Attached patch seemed to fix it for me. It should be sufficient to convert from the db charset to utf-8 here. However, there was the missing conversion from the database to utf-8 in the migration script that probably made the extra conversion in your patch necessary. This has been fixed. > Another bug has appeared. Output from debug log: > DEBUG: SQL SELECT user_id, user_name FROM `rampage_users` WHERE > user_name IN ('ntllt') > DEBUG: SQL QUERY FAILED: Duplicate entry 'ntllt' for key > rampage_users_user_name' INSERT INTO `rampage_users` (user_name) > VALUES ('ntllt') Yeah, looks like there is a bunch of charset conversions missing in content. Working on it, and updated the title of the ticket to reflect the actual problem.

Changes have been made in Git for this ticket: Fix charset

Sat, 05 Mar 2011 07:45:59 +0000

Changes have been made in Git for this ticket: Fix charset handling in tagger Bug: 9617 3 files changed, 26 insertions(+), 8 deletions(-) http://git.horde.org/horde-git/-/commit/5ac680c13a93b32df97fc5bb3c29cb3c4b8e4cbb

Changes have been made in Git for this ticket: Bug #9617: F

Mon, 07 Mar 2011 16:23:21 +0000

Changes have been made in Git for this ticket: Bug #9617: Fix property name. 1 files changed, 2 insertions(+), 2 deletions(-) http://git.horde.org/horde-git/-/commit/7d484c517ddde6a6818845ea1b33be3f20c36c89

> Changes have been made in Git for this ticket: > > Bug #

Mon, 07 Mar 2011 21:02:50 +0000

> Changes have been made in Git for this ticket: > > Bug #9617: Fix property name. > > 1 files changed, 2 insertions(+), 2 deletions(-) > http://git.horde.org/horde-git/-/commit/7d484c517ddde6a6818845ea1b33be3f20c36c89

The original problem still remains. PHP's manual suggest tha

Mon, 07 Mar 2011 21:07:44 +0000

The original problem still remains. PHP's manual suggest that one should not assume that strtolower()/strtoupper() work correctly with multibyte charset like utf-8. Should the code use mb_strtoupper()/mb_strtolower() or Horde::String instead of strtolower()/strtoupper()?

> PHP's manual suggest that one should not assume that strto

Mon, 07 Mar 2011 21:19:30 +0000

> PHP's manual suggest that one should not assume that strtolower()/strtoupper() work correctly with > multibyte charset like utf-8. Where does it say that? I don't see any such suggestions in the man pages.

>> PHP's manual suggest that one should not assume that >>

Mon, 04 Apr 2011 14:24:39 +0000

>> PHP's manual suggest that one should not assume that >> strtolower()/strtoupper() work correctly with >> multibyte charset like utf-8. > > Where does it say that? I don't see any such suggestions in the man pages. It does not it say it in so many words or at least says it ambiguously: "Note that 'alphabetic' is determined by the current locale" But if we look at php's source code for strtoupper() it works by bytes, therefore it will not work correctly with UTF-8 encoded strings that contain non ascii characters. Excerpt from ext/standard/string.c: char *php_strtoupper(char *s, size_t len) { unsigned char *c, *e; c = (unsigned char *)s; e = (unsigned char *)c+len; while (c < e) { *c = toupper(*c); c++; } return s; } The non ascii characters in UTF-8 are multi byte. Therefore using php's strtoupper()/strtolower() will not work correctly with UTF-8 encoded strings with non ascii characters.

>>> PHP's manual suggest that one should not assume that >>

Mon, 04 Apr 2011 14:33:13 +0000

>>> PHP's manual suggest that one should not assume that >>> strtolower()/strtoupper() work correctly with >>> multibyte charset like utf-8. >> >> Where does it say that? I don't see any such suggestions in the man pages. > > It does not it say it in so many words or at least says it > ambiguously: "Note that 'alphabetic' is determined by the current > locale" Which is exactly what we want. > But if we look at php's source code for strtoupper() it works by > bytes, therefore it will not work correctly with UTF-8 encoded > strings that contain non ascii characters. So the manual is plain wrong. > Excerpt from ext/standard/string.c: > char *php_strtoupper(char *s, size_t len) > { > unsigned char *c, *e; > > c = (unsigned char *)s; > e = (unsigned char *)c+len; > > while (c < e) { > *c = toupper(*c); > c++; > } > return s; > } > > The non ascii characters in UTF-8 are multi byte. Therefore using > php's strtoupper()/strtolower() will not work correctly with UTF-8 > encoded strings with non ascii characters. Thanks for tracking this down so deep.

The test suite runs fine though. Can you provide a patch to

Mon, 04 Apr 2011 15:15:37 +0000

The test suite runs fine though. Can you provide a patch to TaggerTest.php that demonstrates the broken behavior?

Nevermind, I found one. Just to get this straight, the idea

Mon, 04 Apr 2011 15:30:26 +0000

Nevermind, I found one. Just to get this straight, the idea is that tags TYÖ and työ are considered equal, right?

> Nevermind, I found one. Just to get this straight, the ide

Mon, 04 Apr 2011 15:37:38 +0000

> Nevermind, I found one. Just to get this straight, the idea is that > tags TYÖ and työ are considered equal, right? That is right.

Okay, but this opens a complete new can of worms. The "SELEC

Mon, 04 Apr 2011 16:46:58 +0000

Okay, but this opens a complete new can of worms. The "SELECT ... WHERE tag_name IN (...)" won't work in this case, because it is case sensitive. The correct solution would be to delegate the lowercasing to the database, but at least for SQLite this doesn't seem to work. "SELECT LOWER('TYÖ')" returns "tyÖ" there. It works fine in MySQL though.

http://www.sqlite.org/faq.html#q18 This makes unit testing

Mon, 04 Apr 2011 16:55:18 +0000

http://www.sqlite.org/faq.html#q18 This makes unit testing this stuff a PITA.

Changes have been made in Git for this ticket: Add failing

Mon, 04 Apr 2011 17:07:41 +0000

Changes have been made in Git for this ticket: Add failing test for bug #9617. 1 files changed, 5 insertions(+), 4 deletions(-) http://git.horde.org/horde-git/-/commit/55802691eafbb6931b93e8c96ee8d2d4fd5b441b

Changes have been made in Git for this ticket: Fix case-ins

Mon, 04 Apr 2011 17:07:45 +0000

Changes have been made in Git for this ticket: Fix case-insensitive filtering of duplicate tags (Bug #9617). This simplifies the _checkTags() method a lot too. Unfortunately it doesn't work at all with SQLite, so unit tests are rather useless. 3 files changed, 19 insertions(+), 26 deletions(-) http://git.horde.org/horde-git/-/commit/a90c671771adbbb1aa08576a2b9d13e011ca6790

I don't see any short-term solution for that. We MUST expect

Mon, 04 Apr 2011 17:09:53 +0000

I don't see any short-term solution for that. We MUST expect the database to do case-insensitive searches, it's completely insane that it doesn't work by default with SQLite. I'm surprised it didn't break anything else yet. This makes SQLite pretty useless for any real-world usage of Horde.

Well, sqlite isn't going to scale for a full Horde installat

Tue, 05 Apr 2011 14:45:48 +0000

Well, sqlite isn't going to scale for a full Horde installation in the real world anyway, so while it's a pain, I'm not sure how much of a problem it is...

Original problem is fixed, resolving. Testing problems like

Wed, 06 Apr 2011 18:38:52 +0000

Original problem is fixed, resolving. Testing problems like this is impossible due to SQLite issues, but nothing we can do about that.