Unhandled socket.send() raised exception.

[dmacnet]

I've written a server using Python 3.5.1 and asynqp (master branch as of March 1). It listens to rabbitmq 3.2.4 for commands from clients to execute and sends back the results via rabbitmq. asynqp is the only asyncio code running in the server. It's running on Ubuntu 14.04.

Once or twice a day, the server becomes unresponsive and the log file stops with:
socket.send() raised exception.

That message is printed by asyncio in selector_events.py in write().
As far as I can tell, for some reason the AMQP connection has closed and asynqp has not detected it, so the writes just keep failing until we restart our server.

I suspect that asynqp needs to check whether transport._conn_lost > 0 after doing a write() and do something like raise a ConnectionLostError if so. I'm not sure if doing it in protocol.py send_frame() and send_protocol_header() would be sufficient or if there are more places that need to check for this condition. There are a couple dozen write() calls in asynqp and I'm not sure what the 'stream' class they are a method of is exactly.

The asyncio streams.py _drain_helper() in Python 3.5.1 handles this situation for StreamWriter, but asynqp doesn't use that API. More discussion about the issue is at
python/asyncio#301

[tvoinarovskyi]

Basically it relates to both #62 and #56:

• We don't check, that the connection is alive at publish calls
• We don't wait for socket flush after publish calls