Optimize ML inference connection retry logic (#1059)

Optimize ML inference connection retry logic (#1059) #1807