Remote Python Support for Benchmark Runner

[SujeethJinesh]

Description

Adds a recipe for testing and eventually benchmarking the Pathways Remote Python feature.

This is an initial PR that will set the precedent for future recipes. It is designed to be shareable and have common functionality (like flag parsing) that can be shared across recipes, but flexible and standard so that multiple team members can simply run the file with minimal configuration changes. It's designed to avoid superfluous flag adding and continue pythonically using configs.

This recipe in particular is for testing functionality for remote python.

Tests

Tested this change on a test remote python sidecar server on a v6e and a v5e shared test cluster.

Sibling XPK change: AI-Hypercomputer/xpk#326

Checklist

Before submitting this PR, please make sure (put X in square brackets):

• I have performed a self-review of my code.
• I have necessary comments in my code, particularly in hard-to-understand areas.
• I have run end-to-end tests tests and provided workload links above if applicable.
• I have made or will make corresponding changes to the doc if needed.

[shauryagup]

I am unclear on why we added this new recipe file. Why is updating the pathways yaml if sidecar is provided sufficient? This should be agnostic to the model config used to run with remote python.

[SujeethJinesh]

I'm not sure I fully understand the question. There's an equivalent XPK change (AI-Hypercomputer/xpk#326) needed for this to work end to end, and this benchmark runner uses that feature on XPK.

And yes the model config used is agnostic, since anyone can technically run remote python with the current setup. Since we're still developing, we'll likely need to make some edits to this recipe once RP is feature complete -- specifically to feature gate (e.g. using a USE_REMOTE_PYTHON option or such, similar to ENABLE_PERSISTENCE in pathwaysutils).