Optimize ML inference connection retry logic

Optimize ML inference connection retry logic #1797