[KFP] Obtain the Ray cluster run ID from the user for KFP v2. #956
+435
−354
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Revital Sur <[email protected]>
Signed-off-by: Revital Sur <[email protected]>
Signed-off-by: Revital Sur <[email protected]>
Signed-off-by: Revital Sur <[email protected]>
Signed-off-by: Revital Sur <[email protected]>
Signed-off-by: Revital Sur <[email protected]>
Signed-off-by: Revital Sur <[email protected]>
Signed-off-by: Revital Sur <[email protected]>
|
Please update the PR explanation: |
roytman
reviewed
Jan 22, 2025
kfp/doc/simple_transform_pipeline.md
Outdated
| # In KFPv2 dsl.RUN_ID_PLACEHOLDER is deprecated and cannot be used since SDK 2.5.0. On another hand we cannot create | ||
| # a unique string in a component (at runtime) and pass it to the `clean_up_task` of `ExitHandler`, due to | ||
| # https://github.com/kubeflow/pipelines/issues/10187. Therefore, meantime the user is requested to insert | ||
| # a unique string created at compilation time. |
roytman
reviewed
Jan 22, 2025
| "same version of the same pipeline !!!") | ||
| run_id = ray_id_KFPv2 | ||
| else: | ||
| run_id = dsl.RUN_ID_PLACEHOLDER |
There was a problem hiding this comment.
the next line uses dsl.RUN_ID_PLACEHOLDER ?
roytman
reviewed
Jan 23, 2025
| func=compute_exec_params_func, base_image=base_kfp_image | ||
| ) | ||
| else: | ||
| compute_exec_params_op = comp.create_component_from_func(func=compute_exec_params_func, base_image=base_kfp_image) | ||
| # create Ray cluster | ||
| create_ray_op = comp.load_component_from_file("../../../kfp_ray_components/createRayComponent.yaml") |
There was a problem hiding this comment.
should it be component_spec_path + "createRayClusterComponent.yaml"
and below
| @@ -57,11 +57,16 @@ Ray cluster. For each step we have to define a component that will execute them: | |||
| ```python | |||
| # components | |||
| base_kfp_image = "quay.io/dataprep1/data-prep-kit/kfp-data-processing:0.0.2" | |||
| @@ -107,6 +112,7 @@ The input parameters section defines all the parameters required for the pipelin | |||
| The parameters used here are as follows: | |||
|
|
|||
| * ray_name: name of the Ray cluster | |||
| * ray_run_id_KFPv2: Ray cluster unique ID used only in KFP v2 | |||
There was a problem hiding this comment.
it is not in the pipeline parameters; it should be around line 91
| @@ -111,9 +99,11 @@ def {{ pipeline_name }}( | |||
| ray_name: str = "{{ pipeline_name }}-kfp-ray", # name of Ray cluster | |||
| # Add image_pull_secret and image_pull_policy to ray workers if needed | |||
| {%- if image_pull_secret != "" %} | |||
| ray_run_id_KFPv2: str = "", | |||
There was a problem hiding this comment.
why do we need to define it twice? Here and at L 106?
| @@ -139,6 +128,7 @@ def code2parquet( | |||
| """ | |||
| Pipeline to execute NOOP transform | |||
| :param ray_name: name of the Ray cluster | |||
| :param ray_run_id_KFPv2: string holding the id used for the Ray cluster used only in KFP v2 | |||
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
In KFPv2 dsl.RUN_ID_PLACEHOLDER is deprecated and cannot be used since SDK 2.5.0. On another hand we cannot create
a unique string in a component (at runtime) and pass it to the
clean_up_taskofExitHandler, due tokubeflow/pipelines#10187.
Therefore, meantime the user is requested to insert a unique Ray cluster name at Run creation time.