Occasionally missing work

[GoogleCodeExporter]

Amqplib 1.0.2 with timeout patch. Python 3.2. RabbitMQ 2.7.1.

I have two workers on the same server processing incoming messages and replying 
RPC-style (i.e. preserving correlation_id). Also, 
chan.basic_qos(prefetch_size=0,prefetch_count=1,a_global=False) and 
chan.basic_consume(queue=q, no_ack=False, callback=data_received)

Each work item takes 200-2000ms depending on content. There is plenty of time 
between work items, they arrive 1-20 per second. The two workers are mainly for 
redundancy (being able to upgrade and restart without downtime).

I have logging, with timestamps, in my own code both on submitting work to 
RabbitMQ and on processing it.

Occasionally, I will see work packets being delayed maybe 25 seconds from being 
given to RabbitMQ before arriving at the client. This happens randomly and 
about once every 5K-10K work items. After 20 seconds the client times out so 
while the work is eventually done, no one is interested in the result.

RabbitMQ and the workers are separated by one network switch, the servers are 
just next to eachother. The Rabbit queue setup is very simple with just a 
default/direct exchange.

I am unsure where the problem lies, but my own logging indicates it is either 
RabbitMQ or amqplib. I would be happy to log more, share code, share data or 
otherwise do work to follow up on people's hunches.

If anyone is interested, the site in question is boardword.com

Original issue reported on code.google.com by [email protected] on 5 Mar 2012 at 10:09