[Backport 2.x] Optimize ML inference connection retry logic

[Backport 2.x] Optimize ML inference connection retry logic #610