[VirtualCluster] Dashboard supports CRUD of virtual clusters [2/N] #430
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Chong-Li
commented
Dec 23, 2024
| allocation_mode = "mixed" | ||
| wait_for_condition(_create_or_update_virtual_cluster, timeout=10) | ||
|
|
||
| # TODO: uncomment this when `MixedCluster::IsIdleNodeInstance` is fixed. |
There was a problem hiding this comment.
We have to fix MixedCluster::IsIdleNodeInstance, making it work with ClusterResourceManager
Chong-Li
commented
Dec 23, 2024
| revision=0, | ||
| ) | ||
| assert result["result"] is False | ||
| assert str(result["msg"]).endswith("No enough node instances to assign.") |
There was a problem hiding this comment.
We may have to tell users which node type and how many nodes we lack, so that users know how to scale up the primary cluster or adapt the virtual cluster requirements.
Chong-Li
commented
Dec 23, 2024
| # The virtual cluster can not be removed because some nodes | ||
| # are still in use. | ||
| assert result["result"] is False | ||
| assert str(result["msg"]).endswith("still in use.") |
There was a problem hiding this comment.
We may have to tell users which nodes are still in use, so that they know which nodes to drain before removing. We should tell similar things when users are trying to scale down a virtual cluster.
wumuzi520
reviewed
Dec 25, 2024
| @@ -328,15 +348,40 @@ void ExclusiveCluster::ForeachJobCluster( | |||
|
|
|||
| ///////////////////////// MixedCluster ///////////////////////// | |||
| bool MixedCluster::IsIdleNodeInstance(const std::string &job_cluster_id, | |||
| const std::string &node_instance_id, | |||
There was a problem hiding this comment.
Put node_instance_id into struct NodeInstance
wumuzi520
reviewed
Dec 25, 2024
| } | ||
|
|
||
| bool MixedCluster::InUse() const { | ||
| // TODO(Shanly): Check if the virtual cluster still running jobs or placement groups. | ||
| bool MixedCluster::InUse(ReplicaInstances *in_use_instances) const { |
There was a problem hiding this comment.
Suggested change
| bool MixedCluster::InUse(ReplicaInstances *in_use_instances) const { | |
| bool MixedCluster::InUse() const { |
wumuzi520
reviewed
Dec 25, 2024
| auto message = ostr.str(); | ||
| RAY_LOG(ERROR) << message; | ||
| // TODO(Shanly): build a new status. | ||
|
|
There was a problem hiding this comment.
return Status::UnsafeToRemove();
wumuzi520
reviewed
Dec 25, 2024
| @@ -261,10 +276,11 @@ class MixedCluster : public VirtualCluster { | |||
| /// Check if the virtual cluster is in use. | |||
| /// | |||
| /// \return True if the virtual cluster is in use, false otherwise. | |||
| bool InUse() const override; | |||
| bool InUse(ReplicaInstances *in_use_instances = nullptr) const override; | |||
There was a problem hiding this comment.
Suggested change
| bool InUse(ReplicaInstances *in_use_instances = nullptr) const override; | |
| bool InUse() const override; |
wumuzi520
reviewed
Dec 25, 2024
| @@ -238,10 +250,11 @@ class ExclusiveCluster : public VirtualCluster { | |||
| /// Check if the virtual cluster is in use. | |||
| /// | |||
| /// \return True if the virtual cluster is in use, false otherwise. | |||
| bool InUse() const override; | |||
| bool InUse(ReplicaInstances *in_use_instances = nullptr) const override; | |||
There was a problem hiding this comment.
Suggested change
| bool InUse(ReplicaInstances *in_use_instances = nullptr) const override; | |
| bool InUse() const override; |
wumuzi520
reviewed
Dec 25, 2024
| /// \return True if the virtual cluster is in use, false otherwise. | ||
| virtual bool InUse() const = 0; | ||
| virtual bool InUse(ReplicaInstances *in_use_instances = nullptr) const = 0; |
There was a problem hiding this comment.
Suggested change
| virtual bool InUse(ReplicaInstances *in_use_instances = nullptr) const = 0; | |
| virtual bool InUse() const = 0; |
wumuzi520
reviewed
Dec 25, 2024
| /// | ||
| /// \param replica_sets The demand final replica sets. | ||
| /// \param replica_instances The node instances lookuped best effort from the visible | ||
| /// node instances. | ||
| /// \return OK if the lookup is successful, otherwise return an error. | ||
| Status LookupIdleNodeInstances(const ReplicaSets &replica_sets, | ||
| Status LookupIdleNodeInstances(ReplicaSets &replica_sets, |
There was a problem hiding this comment.
Suggested change
| Status LookupIdleNodeInstances(ReplicaSets &replica_sets, | |
| Status LookupIdleNodeInstances(const ReplicaSets &replica_sets, |
wumuzi520
reviewed
Dec 25, 2024
| // TODO(Shanly): Give a more detailed error message about the demand replica set and | ||
| // the idle replica instances. | ||
| return Status::OutOfResource("No enough node instances to assign."); | ||
| return Status::OutOfResource(""); |
There was a problem hiding this comment.
just return success directly.
wumuzi520
reviewed
Dec 25, 2024
| # Create a new virtual cluster with exclusive allocation mode. | ||
| virtual_cluster_id = "virtual_cluster_1" | ||
| allocation_mode = "exclusive" | ||
| replica_sets = {"4c8g": 1, "8c16g": 1} |
There was a problem hiding this comment.
add an argument to _create_or_update_virtual_cluster and fill it with replica_sets
wumuzi520
reviewed
Dec 27, 2024
| @@ -460,6 +471,32 @@ Status PrimaryCluster::CreateOrUpdateVirtualCluster( | |||
| replica_instances_to_add_to_logical_cluster, | |||
| replica_instances_to_remove_from_logical_cluster); | |||
| if (!status.ok()) { | |||
| ReplicaInstances *replica_instances = nullptr; | |||
| if (status.IsOutOfResource()) { | |||
| replica_instances = &replica_instances_to_add_to_logical_cluster; | |||
There was a problem hiding this comment.
if (replica_sets) {
*replica_sets = toReplicaSets(replica_instances_to_add_to_logical_cluster);
}
wumuzi520
reviewed
Dec 27, 2024
| @@ -530,6 +572,7 @@ void PrimaryCluster::OnNodeAdd(const rpc::GcsNodeInfo &node) { | |||
| const auto &template_id = node.node_type_name(); | |||
| auto node_instance_id = NodeID::FromBinary(node.node_id()).Hex(); | |||
| auto node_instance = std::make_shared<gcs::NodeInstance>(); | |||
| node_instance->set_node_instance_id(node_instance_id); | |||
There was a problem hiding this comment.
auto node_instance = std::make_sharedgcs::NodeInstance(node_instance_id);
wumuzi520
reviewed
Dec 27, 2024
| RAY_LOG(WARNING) << "Failed to create or update virtual cluster " | ||
| << virtual_cluster_id << ", status = " << status.ToString(); | ||
| if (data != nullptr && (status.IsOutOfResource() || status.IsUnsafeToRemove())) { |
There was a problem hiding this comment.
remove there logic from callback
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
This PR adds some e2e tests for the CRUD of virtual clusters.
Related issue number
#417 #418
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.