Committed and merged.



Thanks again.

Changes have been made in CVS for this ticket:

Stricter determination of when to use string literals.
Bug: 8535
Submitted by: noah (at) lsit (dot) ucsb (dot) edu
http://cvs.horde.org/diff.php/framework/Net_IMSP/IMSP.php?rt=horde&r1=1.44&r2=1.45&ty=u

forgot the patch, here it is. Net_IMSP/IMSP.php

Some users on our system have very large and complicated sets of 
address books stored on a cyrus IMSP server. Performance of the IMSP 
backend is a serious issue for us. One of the things I've noticed in 
comparing IMSP conversations between horde's IMSP client and say, 
Mulberry's, is that horde is sending many more strings as literals 
instead of double quoted strings. From RFC, "In the case of literals 
transmitted from client to server, the client must wait to receive a 
command continuation request (described later in this document) before 
sending the octet data (and the remainder of the command)." Every time 
a string is sent as a literal there's an extra To/From with the 
server, so per user, per request, it can add up.



I didn't do a lot of time benchmarking to back this up, but this seems 
like one of the easiest and best ways to improve performance. In a 
particular address book on our system, the number of separate IMSP 
sends To: the server on each request for Turba's browse interface 
would normally be around 1900. More conservative use of literals drops 
that to around 550 for the same request. We have a custom mass 
batching fetchaddress function that also helps, so mileage will vary.



Literal usage in the IMSP backend is governed by this definition:

define('IMSP_MUST_USE_LITERAL', "/[\W]/i");



The main reason /[\W]/i is overbroad is that it catches a vast 
majority of contact names (generally "First Last" etc). Technically, 
transferring all strings as literals would be valid behavior, but 
there is a cost associated with waiting for all those command 
continuation responses.  It seems like there is some performance to be 
gained by tightening the use of literals to (at least) not include 
space characters (but still include CR & LF) and maybe commas.



From reading the IMSP RFC, looking at other client conversations, and 
testing with imspd, it seems like a better value for 
IMSP_MUST_USE_LITERAL might be:

define('IMSP_MUST_USE_LITERAL', "/[\x80-\xFF\\r\\n\"\\\\]/");



Any 8-bit char, CR, LF, double-quote <">, or backslash \. The Regex is 
based on what a double-quoted string is NOT, and what a literal IS 
(RFC bit follows below). It works well with cyrus imspd.



The new regex also handles the cases that were being fixed last time 
IMSP_MUST_USE_QUOTE was changed. Fields with multi-line input continue 
to work fine, thanks to CR & LF in the regex. It's also been tested 
with UTF-8, latin1, and ascii chars.



In the provided patch, I've retained the /[\W]/i as 
IMSP_MUST_USE_QUOTE, but I've only used it once in the 
quoteSpacedString function, so it might not be necessary. I had been 
thinking there might be additional places in the IMSP code where 
quoteSpacedString would have to be called, but if there are, I have 
not found them.



From the RFC, regarding quoted strings and literals:

"A quoted string is a sequence of zero or more 7-bit characters, 
excluding CR and LF, with double quote (<">) characters at each end."

   TEXT_CHAR       ::= <any CHAR except CR and LF>

   QUOTED_CHAR     ::= <any TEXT_CHAR except quoted_specials> /

                        "\" quoted_specials



   quoted_specials ::= <"> / "\"



"A literal is a sequence of zero or more octets (including CR and LF)"

   literal         ::= "{" number "}" CRLF *CHAR8

                        ;; The number represents the number of CHAR8 octets



   CHAR            ::= <any 7-bit US-ASCII character except NUL, 0x01 - 0x7f>



   CHAR8           ::= <any 8-bit octet except NUL, 0x01 - 0xff>