(as the author of the above) Thanks for the quick fix.
BTW, I'm running the latest Dovecot version, and it's not a software 
bug. My maildir is broken.

Closing.

From a duplicate ticket, these were the kind of errors that were 
causing problems:

* 12 FETCH (UID 20 FLAGS (\Seen Junk))
* BYE Internal error occurred. Refer to server log for more 
information. [2013-05-29 17:30:25]

This should be a wakeup call for those of you that ran into this issue 
that there are some serious issues with your IMAP server and you may 
want/need to do some maintenance on that end.

The issue seems fixed for me. No more error messages in the logs after 
24 hours. Before the patch, it was showing a lot (my log files were 
3GB long) and whitin less than 24 hours. I´m using dovecot 2.0.19 and 
imapproxy 1.2.7.

Correct.

The 4 most important IMAP extensions are UIDPLUS, SORT, THREAD, and 
CONDSTORE.  Without those 4, your disconnected IMAP client (i.e. 
webmail) is going to have poor performance.

Yes, I was just about to update my previous post that I didnt notice 
that badly outdated dovecot's wiki page.
I'm running an old box with Dovecot 1.0, thats why I was hitting that 
bug that has just been fixed.

Thanks,
Rui

This is outdated information. Dovecot has supported UIDPLUS for years 
already. If you want to check, connect to your IMAP server and run   
the CAPABILITY command.

Thanx for the fix.
But it seems to me that even current versions of Dovecot lacks the 
UIDPLUS feature: http://wiki.dovecot.org/FeatUIDPLUS

Rui

The point being is that the current issue exposed longstanding (known) 
issues with IMAP servers.  The regression was the handling of these 
errors.

In other words: the recent changes to the Imap Client library would 
NOT have caused any of these issues (with one exception; see below).   
These errors have been occurring in your IMAP server for years.  The 
issue was that, in an effort to fix another issue, these errors were 
not being handled properly.

The only other change that has happened in the IMAP library has been 
the implementation of pipelined commands - the sending of several 
commands in a row before reading any output from the server.  But 
pipelining is a basic principle outlined in the base RFC (3501), so 
any server that doesn't support what we are doing - which is fairly 
basic pipelining - is broken in every sense of the word.

Normally, Horde/IMP has issues because our IMAP library is on the 
cutting edge when it comes to advanced extension support.  I can vouch 
that IMP usage and error reporting has fixed multiple bugs in advanced 
extensions in both Dovecot and Cyrus.

It is off-topic, but it is a totally valid bug report.  You may be the 
only one seeing this because it relies on the absence of UIDPLUS.   
Almost every IMAP server supports UIDPLUS so you are using one of the 
rare servers that doesn't support it (either that or you have disabled 
the extension in your IMP configuration).

This issue has been fixed in Horde_imap_Client 2.11.2 and, since it is 
fairly critical for these non-UIDPLUS servers, a new release has been 
created.

No more problems so far after the update of imap client through pear...
I'll post in this thread if it will happen again, though I think it 
will not happen again after the fix to prevent the infinite loop that 
was causing it, but I agree that indeed it would be better to know the 
exact cause of the problem, if a real bug in the imap server or just a 
problem in its config (maybe not enough max processes?). Maybe I'll 
dig it when I have time, what I know so far is that only yesterday it 
started to happen at my server, although I've updated the imap client 
over 6 days ago. So it seems it is triggered only on specific 
situations, maybe when more than a specified number of users are 
connected, then maybe just a case of an insufficient number of max 
imap server processes.

What I still dont know and bugs me completely is why I am the only one 
here that had again after this update have to manually edit 
"Imap/Client/Socket.php" at line 3300 and replace "_queryCapability()" 
with "queryCapability()" to remove the underscore, whithout which I'd 
just see a "500 server error" browser page:
I had to do this tweak already before, after the previous update to 
imap client some days ago and again with this last update. That line 
is the only place where "queryCapability()" is referenced with an 
underscore prefix.
Sorry for the off-topic.

Rui

If it loses connectivity to the actual IMAP backend, yes.

I am using imap proxy in front of dovecot, can that be the problem?

I have.  Many times in the past.  We've uncovered literally dozens of 
IMAP server bugs the last few years.  You can search the list archives 
and/or the source code commits for further details on these various 
bugs.

For the record, this bug can only be triggered *if*:

1) The PHP server loses network connection to the IMAP server.  This 
could be either a network issue or a crash on the IMAP server side, 
which happens often - especially with certain older versions of 
Courier - that are triggered by a specific IMAP command.

2) The IMAP server responds to a command with a unsolicited BYE.   
Again, this should NEVER happen during a normal conversation (although 
this is something that is allowed, or at least contemplated, by the RFC)

Unfortunately, in moving things around to fix an error that I COULD 
reliably reproduce locally, these two cases were ignored.  And both 
cases are extremely difficult to write test cases for (I would 
essentially need to build a "dummy" PHP server for the test unit, 
which I don't have the time or funding for right now).
No, that doesn't mean anything.  Can't help but notice that nobody 
running Dovecot 2.1 or 2.2 are seeing anything, for example.

And FWIW, the other day I found a bug in Dovecot 2.2.2 (the latest 
version) that is not yet fixed.  Although granted, that issue would 
not trigger this bug - it's a problem with parsing CATENATE parameters 
with literal8 data.

I'm also seeing this issue with Dovecot 1.0 (yes, I know...)

Will try this fix and report back later.

Rui

Dovecot 2.0.19.

I´ve applied the patches from github. I´ll observe to see if the error 
has gone way and report back.

Would be useful to actually explain the problem so it might get fixed 
also for the IMAP servers affected. The fact that different versions 
of Dovecot and also Courier are affected point to a not obvious but 
common problem, no?

The fact that nobody who has applied the patch is seeing the issues 
makes me think that, in fact, this is fixed.

I've released Horde_Imap_Client 2.11.0.  I'll leave this open for a 
few days, but I don't anticipate the need for more fixes.

What's astonishing is how many people have broken IMAP servers.  I 
never realized there were that many out there.

Changes have been made in Git (master):

commit c46e5625c1a7855c49a55dc2be3845f7f61ab7b7
Author: Michael M Slusarz <slusarz@horde.org>
Date:   Mon May 27 20:25:12 2013 -0600

     [mms] Workaround broken IMAP servers and prevent infinite loops 
(Bug #12265).

  framework/Imap_Client/package.xml |    2 ++
  1 files changed, 2 insertions(+), 0 deletions(-)

http://git.horde.org/horde-git/-/commit/c46e5625c1a7855c49a55dc2be3845f7f61ab7b7

Changes have been made in Git (master):

commit 4bd2f71d239a821de118b78ab5c50be7636f989c
Author: Michael M Slusarz <slusarz@horde.org>
Date:   Sun May 26 21:18:36 2013 -0600

     Bug #12265: Another tweak

  .../Imap_Client/lib/Horde/Imap/Client/Socket.php   |    4 +++-
  1 files changed, 3 insertions(+), 1 deletions(-)

http://git.horde.org/horde-git/-/commit/4bd2f71d239a821de118b78ab5c50be7636f989c

I'am also affected by this bug.
http://article.gmane.org/gmane.comp.horde.user/31359

debian wheezy -> dovecot-imapd 2.1.7-7
debian squeeze -> dovecot-imapd 1.2.15-7

I use courier 4.8.0-3 as well (debian squeeze)

Since adding the debug code I didn't observe the problem again, but as 
it seems this is not necessary anymore.

I just wanted to add, for the sake of completeness, that I am using 
Courier 4.8.0-3.

Changes have been made in Git (master):

commit cb95e9f2c198917cab55b8dfd805930956be572c
Author: Michael M Slusarz <slusarz@horde.org>
Date:   Sun May 26 13:06:17 2013 -0600

     Further fixes to #12265

  .../Imap_Client/lib/Horde/Imap/Client/Socket.php   |    8 +++-----
  1 files changed, 3 insertions(+), 5 deletions(-)

http://git.horde.org/horde-git/-/commit/cb95e9f2c198917cab55b8dfd805930956be572c

I have added the debugging code and reduced the number of imap 
processws to triggee the error again - I will report it as soon as it 
happens.

Thanks, Torben

Thanks, I'll keep the debug code in and see if this happens again 
during the week.

FWIW, this is against Dovecot 2.0.19.

Maybe this fixes?  (Although this can only happen if your IMAP server 
is broken-ish, so that could be why I am not seeing this).

Changes have been made in Git (master):

commit e68f6547c5cdccaa5a36887761778e83f978b20f
Author: Michael M Slusarz <slusarz@horde.org>
Date:   Sun May 26 12:43:05 2013 -0600

     Possible fix for Bug #12265

  .../Imap_Client/lib/Horde/Imap/Client/Socket.php   |    5 +++++
  1 files changed, 5 insertions(+), 0 deletions(-)

http://git.horde.org/horde-git/-/commit/e68f6547c5cdccaa5a36887761778e83f978b20f

Happened again overnight:

2013-05-26T12:00:32-04:00 DEBUG: Backtrace:
  1. Horde_Rpc_ActiveSync->getResponse() 
/usr/local/horde/horde/horde/rpc.php:156
  2. Horde_ActiveSync->handleRequest() 
/usr/local/horde/horde/framework/Rpc/lib/Horde/Rpc/ActiveSync.php:141
  3. Horde_Core_ActiveSync_Driver->clearAuthentication() 
/usr/local/horde/horde/framework/ActiveSync/lib/Horde/ActiveSync.php:860
  4. Horde_Core_ActiveSync_Connector->clearAuth() 
/usr/local/horde/horde/framework/Core/lib/Horde/Core/ActiveSync/Driver.php:212
  5. Horde_Registry->clearAuth() 
/usr/local/horde/horde/framework/Core/lib/Horde/Core/ActiveSync/Connector.php:737
  6. Horde_Registry->callAppMethod() 
/usr/local/horde/horde/framework/Core/lib/Horde/Registry.php:2059
  7. Horde_Registry->pushApp() 
/usr/local/horde/horde/framework/Core/lib/Horde/Registry.php:1153
  8. Horde_Registry->callAppMethod() 
/usr/local/horde/horde/framework/Core/lib/Horde/Registry.php:1545
  9. call_user_func_array() 
/usr/local/horde/horde/framework/Core/lib/Horde/Registry.php:1156
10. Horde_Registry_Application->authenticated()
11. IMP_Application->_authenticated() 
/usr/local/horde/horde/framework/Core/lib/Horde/Registry/Application.php:96
12. IMP_Auth::authenticateCallback() 
/usr/local/horde/horde/imp/lib/Application.php:121
13. IMP_Imap->doPostLoginTasks() /usr/local/horde/horde/imp/lib/Auth.php:299
14. IMP_Imap->updateFetchIgnore() /usr/local/horde/horde/imp/lib/Imap.php:282
15. IMP_Mailbox::getSpecialMailboxes() 
/usr/local/horde/horde/imp/lib/Imap.php:298
16. IMP_Mailbox::getPref() /usr/local/horde/horde/imp/lib/Mailbox.php:1390
17. IMP_Mailbox::prefFrom() /usr/local/horde/horde/imp/lib/Mailbox.php:262
18. IMP_Imap->defaultNamespace() 
/usr/local/horde/horde/imp/lib/Mailbox.php:1446
19. IMP_Imap->getNamespaceList() /usr/local/horde/horde/imp/lib/Imap.php:440
20. IMP_Imap->getNamespaces() /usr/local/horde/horde/imp/lib/Imap.php:385
21. IMP_Imap->__call() /usr/local/horde/horde/imp/lib/Imap.php:385
22. call_user_func_array() /usr/local/horde/horde/imp/lib/Imap.php:569
23. Horde_Imap_Client_Base->getNamespaces()
24. Horde_Imap_Client_Socket->_getNamespaces() 
/usr/local/horde/horde/framework/Imap_Client/lib/Horde/Imap/Client/Base.php:686
25. Horde_Imap_Client_Socket->_sendCmd() 
/usr/local/horde/horde/framework/Imap_Client/lib/Horde/Imap/Client/Socket.php:260
26. Horde_Imap_Client_Socket->_sendCmdChunk() 
/usr/local/horde/horde/framework/Imap_Client/lib/Horde/Imap/Client/Socket.php:3796
27. Horde_Imap_Client_Socket->_getLine() 
/usr/local/horde/horde/framework/Imap_Client/lib/Horde/Imap/Client/Socket.php:3865
28. Horde_Imap_Client_Socket->_readStream() 
/usr/local/horde/horde/framework/Imap_Client/lib/Horde/Imap/Client/Socket.php:4104
29. Horde::debug() 
/usr/local/horde/horde/framework/Imap_Client/lib/Horde/Imap/Client/Socket.php:4192

I will do exactly that.

The issue occured again this afternoon. Unfortunately I hadn't added 
the debugging code by then. I will report back, as soon as the error 
leaves some traces I can post.

Yes.  See the comments on Horde::debug() earlier in this ticket.

I need a backtrace from somebody who is seeing this problem.  I have 
five different test VMs with five different IMAP servers and I can't 
reproduce so nothing is going to happen unless/until somebody gives me 
more feedback.

Just to confirm the issue I want to report my server running into this 
problem, stalling the whole thing after filling up the log partition 
completly within hours.
Probably this may be related to an pear upgrade of several Horde 
libraries earlier this day. After restarting the webserver, its gone 
for the moment.
Anything I can contribute?

Bumping, this took down two of my servers and a development laptop a 
handful of times in the last two days.

You are confusing the scope of the function.

In a public API, in which I (as a developer) do not have control over 
the input, then yes - it makes sense (when possible) to check for bad 
input.

However, this is a private function.  All input is controlled by the 
developer.  These functions SHOULD be very strict about data inputs 
since, if an input is incorrect, that is a giant red flag that 
something is probably wrong in the calling code.

The _readLine() function is only called from functions that MUST have 
an active IMAP connection.  If that is not happening, then this error 
must be fixed at the source.  Fixing it there will actually fix the 
problem rather than suppressing the error messages.
Shouldn't be that difficult.  You just need to post the backtrace part 
of the debug output.  SInce $_stream should never be null in that 
method, any trigger of the debug code has found an issue.  (I'm not 
asking for a patch of the broken code - I'm just asking for help in 
finding where the broken code is).

Thanks for the debug lines, this seems to be a more efficient 
debugging than my debug code was.

However, I think we have a problem at two ends.
IMO, yes, we need to figure out the reason. But additionally I'm still 
convinced that a good function should filter out garbage input  to 
prevent crashes or loops as this one.

I'll need some days to dive into it, I'll call back once I know the 
reason forthe null resources.

regards,
Torben

No.  A PHP resource is a specific data type.  It can never be changed 
by PHP itself to a different data type.

This is not a correct fix.  This simply masks the issue.  It doesn't 
fix anything.

The correct fix is to figure out WHY _stream is null before entering 
that method (i.e. the debug backtrace).  You can use Horde::debug() to 
help you with this, e.g. in _readStream():

if (is_null($this->_stream)) {
     Horde::debug();
     exit;
}

Thanks for clarification Jan, your description is better. :)
Torben

This even occurs some time after a reboot, so it is certtain nothing 
in the context of a wrong update procedure.
It seems to happen if the IMAP server has lots of open connections.

could it happen that the IMAP server invalidates the handle and thus 
it is now null?

The error is located in Imap sockets protected function _readStream()
The reason for the flooded error logs is, that the do-while loop is 
infinite if the stream is NULL, since the "break;" is only called with 
a vaild stream.
So even it seems to be a rare occassion whrer the strem handle is 
NULL, the code should handle such a sittuation correctly.

  have a code proposal ( see attached patch) which handles the NULL 
handle correctly.

Therfore I recommend to check in the loop at first for a null null 
handle and exit the loop with break, please see the fix

4190,4193d4189
<           if( is_null($this->_stream)) {
<               $this->_debug->info("ERROR: Stream is NULL!");
<               break;
<           }

Nothing.  Logging in would have failed if the connection to the server 
could not be established.  Even if the connection is broken after 
logging in, _stream will still be a stream resource.

Unless you did something like update Horde_Imap_Client and had active 
sessions which you should never do.

Finally, maybe its a bug in Courier that has not being tickled until 
now.  But at a minimum would need an IMAP log (and I can verify that I 
can access a Courier server without any issues).

well, that error persists, so I have to solve it. I looked into the 
horde confuig, but ther is nothing to configure about the the IMAP 
client.

Looking into the file mentioned in the error message, I found that in 
both cases the parameter $this->_stream is used, but it is NULL 
instead of a reasource. What can cause this problem?

What is that stream? Which component fills it with data so it is not 
null usually?

P.S.: I'm using Courier IMAP.


thanks,
Torben

Can't reproduce.

May 20 00:20:35 jonathan HORDE: [imp] PHP ERROR: feof() expects 
parameter 1 to be resource, null given [pid 15042 on line 4190 of 
"/usr/share/php/Horde/Imap/Client/Socket.php"]
May 20 00:20:35 jonathan HORDE: [imp] PHP ERROR: fgets() expects 
parameter 1 to be resource, null given [pid 15042 on line 4203 of 
"/usr/share/php/Horde/Imap/Client/Socket.php"]