Consumer Connection to Broker Lost: redis.exceptions.ConnectionError

[kellanburket]

I'm running celery-redis-sentinel in a Kubernetes cluster, which means that if my Redis container goes down it often comes back up with a new IP. When this happens, celery seems to connect successfully then throws an error pointing to the original IP:

[2017-08-01 19:00:34,182: INFO/MainProcess] Connected to redis-sentinel://<new redis host>:6379/<db>
[2017-08-01 19:00:34,681: INFO/MainProcess] mingle: searching for neighbors
[2017-08-01 19:00:39,780: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/redis/connection.py", line 439, in connect
    sock = self._connect()
  File "/usr/local/lib/python3.5/dist-packages/redis/connection.py", line 494, in _connect
    raise err
  File "/usr/local/lib/python3.5/dist-packages/redis/connection.py", line 482, in _connect
    sock.connect(socket_address)
OSError: [Errno 113] No route to host

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/redis/client.py", line 572, in execute_command
    connection.send_command(*args)
  File "/usr/local/lib/python3.5/dist-packages/redis/connection.py", line 563, in send_command
    self.send_packed_command(self.pack_command(*args))
  File "/usr/local/lib/python3.5/dist-packages/redis/connection.py", line 538, in send_packed_command
    self.connect()
  File "/usr/local/lib/python3.5/dist-packages/redis/sentinel.py", line 44, in connect
    self.connect_to(self.connection_pool.get_master_address())
  File "/usr/local/lib/python3.5/dist-packages/redis/sentinel.py", line 34, in connect_to
    super(SentinelManagedConnection, self).connect()
  File "/usr/local/lib/python3.5/dist-packages/redis/connection.py", line 442, in connect
    raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error 113 connecting to <old redis host>:6379. No route to host.

Not sure why this behavior is occurring if Celery is aware of the new IP. Note that this same string of messages is repeated every few seconds.

[kellanburket]

I now see that this is expected behavior in the CelerySentinelConnectionPool--but if I understand this right Celery will continue to check every bad connection each time and continue to throw errors when it can't connect. My events continue to be scheduled but none of my workers are picking them up. This can't be right.