Summary | Make use of streams for speed and memory efficiency |
Queue | Horde Framework Packages |
Queue Version | HEAD |
Type | Enhancement |
State | Resolved |
Priority | 2. Medium |
Owners | slusarz (at) horde (dot) org |
Requester | chuck (at) horde (dot) org |
Created | 01/28/2006 (7074 days ago) |
Due | |
Updated | 01/13/2010 (5628 days ago) |
Assigned | 11/09/2008 (6058 days ago) |
Resolved | 06/30/2009 (5825 days ago) |
Milestone | Horde 4.0 |
Patch | No |
Request #3359: Horde_Mime_Part & streamsUse PHP temporary streams when working with message body data to reduce
memory usage.
http://git.horde.org/diff.php/imp/docs/CHANGES?rt=horde-git&r1=f1264bd13cd8eb1255f8078b2691c719c8598a2f&r2=8a436842e46e1ad943914031e0b54acadaac26b9
http://git.horde.org/diff.php/imp/lib/Contents.php?rt=horde-git&r1=58e871fd57002fc788317dfb765c320d9c9ad5a3&r2=8a436842e46e1ad943914031e0b54acadaac26b9
http://git.horde.org/diff.php/imp/lib/Mime/Viewer/Partial.php?rt=horde-git&r1=7dc3f0b944c0bebc4d64f09f749ebf6929f998fc&r2=8a436842e46e1ad943914031e0b54acadaac26b9
http://git.horde.org/diff.php/imp/view.php?rt=horde-git&r1=78fd3f9eaf63e8d7c79a458a73e2a14a3ae7ca97&r2=8a436842e46e1ad943914031e0b54acadaac26b9
either speed or memory_limit needs or ... ?
encoded) using IMP 4 and IMP 5. This is a message that exceeds the
maximum size filter, so there is only an option to download the
message. According to the memory stats from the debug log, IMP 4
required approx 26 MB to download and IMP 5 required approx 16 MB.
However, I have no idea how much of this is attributable to streams
vs. previous improvements in the code (IMAP, MIME, IMP, etc.)
either speed or memory_limit needs or ... ?
(when grabbing body parts/text) eliminates the need to read (some of)
this data into memory. As it stands, data under 2 MB is read into
memory. Larger files will be spooled to disk above that point. So
for smaller data, there will probably be no gain (and my guess is that
performance might slightly be reduced due to the stream overhead code)
but for larger files we cap memory usage at 2 MB.
The bigger gains come in Horde_Mime_Part. For large body parts, we
can pass the stream object we grab from Imap_Client directly to
Horde_Part. If possible, we can reuse this stream without any further
action (if the data is already in 7bit, 8bit or binary format). If
not, we have to copy this stream to a new stream while running through
a decompression filter. Again, we are using temp streams so the
maximum memory usage during this activity should be approx. 4 MB (2 MB
for each of the streams), and we can easily free up the 2 MB for the
one stream after we are finished.
Certain memory bottlenecks unfortunately still require strings and
there is not much we can do about this (DB/VFS still require strings,
so this doesn't help memory usage for large attachments; Mail doesn't
allow us to pass a stream so we have to create the entire text of the
message before we can send).
Right now, my focus is more on making sure I didn't break anything too badly.
either speed or memory_limit needs or ... ?
State ⇒ Resolved
return types in Horde_Imap_Client. Added support for downloads and
retrieving body parts in imp. Closing ticket - there can probably
further optimizations, but this is the low-hanging fruit.
Will want to use this format:
php://filter/read=convert.base64-encode/resource=file:///tmp/foo
See http://us.php.net/manual/en/wrappers.php.php
Don't use file_get_contents()/stream_get_contents(). On my test file
(11 MB of text data), file_get_contents() required 23 MB. This is
much more efficient:
while (!feof($a)) {
$b .= fread($a, 8192);
}
This used only 300-400KB over the file size.
Same with writing - don't use file_write_contents() for large data.
fwrite in chunks (4096 is probably good).
And use temp stream. temp will copy to memory for first X MB
(configurable) and then will write to disk. Reading the 11 MB file
via the method above, and then writing to temp stream using fwrite(),
and then rewinding temp stream pointer and reading entire file using
fread() took only 14 MB total.
State ⇒ Assigned
Assigned to Michael Slusarz
Queue ⇒ Horde Framework Packages
Summary ⇒ Make use of streams for speed and memory efficiency
State ⇒ Stalled
decoding, the most common cases can be handled by the base64 and
quoted-printable stream filters:
http://www.php.net/manual/en/filters.convert.php
Some compression can be done on streams as well:
http://us2.php.net/manual/en/wrappers.compression.php
(there's a reader stream for zip, not sure about write support)
All of this will take some significant re-jiggering of the MIME libs,
but I think it's worth it to focus on streams and other newer PHP
features that'll help us save memory and speed in the MIME rewrite.
I'm moving this to a Horde 4 milestone and to the framework queue
because of that.
Priority ⇒ 2. Medium
State ⇒ Accepted
Queue ⇒ IMP
Summary ⇒ Look in to using imap_savebody when it's available
Type ⇒ Enhancement