Fri, 10 Apr 2026 10:04:42 +0000 https://bugs.horde.org/ticket/3359 Make use of streams for speed and memory efficiency http://blog.iworks.at/?/archives/37-imap_savebody.html Sat, 28 Jan 2006 22:10:38 +0000 https://bugs.horde.org/ticket/3359#t15990 For instance, imap_savebody can take a stream, and if we need to do decoding, the most common cases can be handled by the base64 and quoted-printable stream filters: http://www.php.net/manual/en/filters.convert.php Some compression can be done on streams as well: http://us2.php.net/manual/en/wrappers.compression.php (there's a reader stream for zip, not sure about write support) All of this will take some significant re-jiggering of the MIME libs, but I think it's worth it to focus on streams and other newer PHP features that'll help us save memory and speed in the MIME rewrite. I'm moving this to a Horde 4 milestone and to the framework queue because of that. Mon, 05 Feb 2007 01:04:45 +0000 https://bugs.horde.org/ticket/3359#t29157 Un-stalling since Michael S. is working on this area now. Sun, 09 Nov 2008 01:24:37 +0000 https://bugs.horde.org/ticket/3359#t50484 Some notes on some testing: Will want to use this format: php://filter/read=convert.base64-encode/resource=file:///tmp/foo See http://us.php.net/manual/en/wrappers.php.php Don't use file_get_contents()/stream_get_contents(). On my test file (11 MB of text data), file_get_contents() required 23 MB. This is much more efficient: while (!feof($a)) { $b .= fread($a, 8192); } This used only 300-400KB over the file size. Same with writing - don't use file_write_contents() for large data. fwrite in chunks (4096 is probably good). And use temp stream. temp will copy to memory for first X MB (configurable) and then will write to disk. Reading the 11 MB file via the method above, and then writing to temp stream using fwrite(), and then rewinding temp stream pointer and reading entire file using fread() took only 14 MB total. Thu, 26 Mar 2009 05:43:28 +0000 https://bugs.horde.org/ticket/3359#t53316 Added to Horde_Mime_Part. Implemented socket return for body-ish return types in Horde_Imap_Client. Added support for downloads and retrieving body parts in imp. Closing ticket - there can probably further optimizations, but this is the low-hanging fruit. Tue, 30 Jun 2009 05:00:54 +0000 https://bugs.horde.org/ticket/3359#t54745 Nice! Do you happen to have any numbers about how this works out, either speed or memory_limit needs or ... ? Wed, 01 Jul 2009 02:34:49 +0000 https://bugs.horde.org/ticket/3359#t54756 > Nice! Do you happen to have any numbers about how this works out, > either speed or memory_limit needs or ... ? Not really. In general terms - using streams in the Imap library (when grabbing body parts/text) eliminates the need to read (some of) this data into memory. As it stands, data under 2 MB is read into memory. Larger files will be spooled to disk above that point. So for smaller data, there will probably be no gain (and my guess is that performance might slightly be reduced due to the stream overhead code) but for larger files we cap memory usage at 2 MB. The bigger gains come in Horde_Mime_Part. For large body parts, we can pass the stream object we grab from Imap_Client directly to Horde_Part. If possible, we can reuse this stream without any further action (if the data is already in 7bit, 8bit or binary format). If not, we have to copy this stream to a new stream while running through a decompression filter. Again, we are using temp streams so the maximum memory usage during this activity should be approx. 4 MB (2 MB for each of the streams), and we can easily free up the 2 MB for the one stream after we are finished. Certain memory bottlenecks unfortunately still require strings and there is not much we can do about this (DB/VFS still require strings, so this doesn't help memory usage for large attachments; Mail doesn't allow us to pass a stream so we have to create the entire text of the message before we can send). Right now, my focus is more on making sure I didn't break anything too badly. Wed, 01 Jul 2009 07:02:46 +0000 https://bugs.horde.org/ticket/3359#t54757 >> Nice! Do you happen to have any numbers about how this works out, >> either speed or memory_limit needs or ... ? FWIW, I opened up a 5 MB text message (approx. 3.5 MB un-base64 encoded) using IMP 4 and IMP 5. This is a message that exceeds the maximum size filter, so there is only an option to download the message. According to the memory stats from the debug log, IMP 4 required approx 26 MB to download and IMP 5 required approx 16 MB. However, I have no idea how much of this is attributable to streams vs. previous improvements in the code (IMAP, MIME, IMP, etc.) Wed, 01 Jul 2009 07:13:14 +0000 https://bugs.horde.org/ticket/3359#t54758 Changes have been made in Git for this ticket: Request #3359: Horde_Mime_Part & streams Use PHP temporary streams when working with message body data to reduce memory usage. http://git.horde.org/diff.php/imp/docs/CHANGES?rt=horde-git&r1=f1264bd13cd8eb1255f8078b2691c719c8598a2f&r2=8a436842e46e1ad943914031e0b54acadaac26b9 http://git.horde.org/diff.php/imp/lib/Contents.php?rt=horde-git&r1=58e871fd57002fc788317dfb765c320d9c9ad5a3&r2=8a436842e46e1ad943914031e0b54acadaac26b9 http://git.horde.org/diff.php/imp/lib/Mime/Viewer/Partial.php?rt=horde-git&r1=7dc3f0b944c0bebc4d64f09f749ebf6929f998fc&r2=8a436842e46e1ad943914031e0b54acadaac26b9 http://git.horde.org/diff.php/imp/view.php?rt=horde-git&r1=78fd3f9eaf63e8d7c79a458a73e2a14a3ae7ca97&r2=8a436842e46e1ad943914031e0b54acadaac26b9 Wed, 13 Jan 2010 00:05:39 +0000 https://bugs.horde.org/ticket/3359#t57583