Changes have been made in Git for this ticket:

Request #3359: Horde_Mime_Part & streams
Use PHP temporary streams when working with message body data to reduce
memory usage.

http://git.horde.org/diff.php/imp/docs/CHANGES?rt=horde-git&r1=f1264bd13cd8eb1255f8078b2691c719c8598a2f&r2=8a436842e46e1ad943914031e0b54acadaac26b9
http://git.horde.org/diff.php/imp/lib/Contents.php?rt=horde-git&r1=58e871fd57002fc788317dfb765c320d9c9ad5a3&r2=8a436842e46e1ad943914031e0b54acadaac26b9
http://git.horde.org/diff.php/imp/lib/Mime/Viewer/Partial.php?rt=horde-git&r1=7dc3f0b944c0bebc4d64f09f749ebf6929f998fc&r2=8a436842e46e1ad943914031e0b54acadaac26b9
http://git.horde.org/diff.php/imp/view.php?rt=horde-git&r1=78fd3f9eaf63e8d7c79a458a73e2a14a3ae7ca97&r2=8a436842e46e1ad943914031e0b54acadaac26b9

FWIW, I opened up a 5 MB text message (approx. 3.5 MB un-base64 
encoded) using IMP 4 and IMP 5.  This is a message that exceeds the 
maximum size filter, so there is only an option to download the 
message.  According to the memory stats from the debug log, IMP 4 
required approx 26 MB to download and IMP 5 required approx 16 MB.   
However, I have no idea how much of this is attributable to streams 
vs. previous improvements in the code (IMAP, MIME, IMP, etc.)

Not really.  In general terms - using streams in the Imap library 
(when grabbing body parts/text) eliminates the need to read (some of) 
this data into memory.  As it stands, data under 2 MB is read into 
memory.  Larger files will be spooled to disk above that point.  So 
for smaller data, there will probably be no gain (and my guess is that 
performance might slightly be reduced due to the stream overhead code) 
but for larger files we cap memory usage at 2 MB.



The bigger gains come in Horde_Mime_Part.  For large body parts, we 
can pass the stream object we grab from Imap_Client directly to 
Horde_Part.  If possible, we can reuse this stream without any further 
action (if the data is already in 7bit, 8bit or binary format).  If 
not, we have to copy this stream to a new stream while running through 
a decompression filter.  Again, we are using temp streams so the 
maximum memory usage during this activity should be approx. 4 MB (2 MB 
for each of the streams), and we can easily free up the 2 MB for the 
one stream after we are finished.



Certain memory bottlenecks unfortunately still require strings and 
there is not much we can do about this (DB/VFS still require strings, 
so this doesn't help memory usage for large attachments; Mail doesn't 
allow us to pass a stream so we have to create the entire text of the 
message before we can send).



Right now, my focus is more on making sure I didn't break anything too badly.

Nice! Do you happen to have any numbers about how this works out, 
either speed or memory_limit needs or ... ?

Added to Horde_Mime_Part.  Implemented socket return for body-ish 
return types in Horde_Imap_Client.  Added support for downloads and 
retrieving body parts in imp.  Closing ticket - there can probably 
further optimizations, but this is the low-hanging fruit.

Some notes on some testing:



Will want to use this format:

php://filter/read=convert.base64-encode/resource=file:///tmp/foo



See http://us.php.net/manual/en/wrappers.php.php



Don't use file_get_contents()/stream_get_contents().  On my test file 
(11 MB of text data), file_get_contents() required 23 MB.  This is 
much more efficient:



while (!feof($a)) {

     $b .= fread($a, 8192);

}



This used only 300-400KB over the file size.



Same with writing - don't use file_write_contents() for large data.   
fwrite in chunks (4096 is probably good).



And use temp stream.  temp will copy to memory for first X MB 
(configurable) and then will write to disk.  Reading the 11 MB file 
via the method above, and then writing to temp stream using fwrite(), 
and then rewinding temp stream pointer and reading entire file using 
fread() took only 14 MB total.

Un-stalling since Michael S. is working on this area now.

For instance, imap_savebody can take a stream, and if we need to do 
decoding, the most common cases can be handled by the base64 and 
quoted-printable stream filters:

http://www.php.net/manual/en/filters.convert.php



Some compression can be done on streams as well:

http://us2.php.net/manual/en/wrappers.compression.php



(there's a reader stream for zip, not sure about write support)



All of this will take some significant re-jiggering of the MIME libs, 
but I think it's worth it to focus on streams and other newer PHP 
features that'll help us save memory and speed in the MIME rewrite. 
I'm moving this to a Horde 4 milestone and to the framework queue 
because of that.

http://blog.iworks.at/?/archives/37-imap_savebody.html