[DNM]: Cluster-specific Optimize pipelines MVP
#25165
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
aalexandrov
force-pushed
the
cluster_specific_optimization
branch
5 times, most recently
from
8143009 to
8476351
Compare
aalexandrov
changed the title
aalexandrov
force-pushed
the
cluster_specific_optimization
branch
from
8476351 to
db0af85
Compare
nrainer-materialize
changed the title
aalexandrov
force-pushed
the
cluster_specific_optimization
branch
22 times, most recently
from
d987f6b to
5ba3d42
Compare
aalexandrov
force-pushed
the
cluster_specific_optimization
branch
14 times, most recently
from
73c3a44 to
bebe890
Compare
Forcing `DataflowBuilder::reoptimize_imported_views` to reoptimize all catalog-backed views imported in the `DataflowDescription` in every possible `EXPLAIN` run is a bit of an overkill. Instead, we should enable work in this method call only when one of the overrides of `OptimizerConfig` (`CREATE CLUSTER ... FEATURES(...)` or `EXPLAIN PLAN WITH(...)`) explicitly sets the value of a feature flag that influences an optimizer stage that runs before we produce the locally optimized `MirRelationExpr`. To do this, we introduce a dedicated optimizer feature flag for that. Engineers should enable this flag in conjunction with any other flags that modify the view optimization pipeline in order to see differences in catalog items that depend on the view.
Design doc for the #25273 epic.
Add parser and planner support for the following `EXPLAIN ... VIEW` statements: - `EXPLAIN CREATE VIEW ...` - `EXPLAIN REPLAN VIEW ...` - `EXPLAIN VIEW ...` To do this, introduce new enum variants `ExplainStage::LocalPlan` and `NamedPlan::Local` that correspond to the `optimize/local` stage and teach the parser to default to this stage for the above variants if an explicitly defined stage is not present.
Add coordinator support for sequencing the following `EXPLAIN ... VIEW` statements: - `EXPLAIN CREATE VIEW ...` - `EXPLAIN REPLAN VIEW ...` - `EXPLAIN VIEW ...`
Move `explain_materialized_view` to `inner/create_materialized_view.rs`.
Move `explain_index` to `inner/create_index.rs`.
Add support for handling the `RAW` and `LOCALLY OPTIMIZED` stages using the plans stored in the catalog entry (as in `EXPLAIN VIEW`).
aalexandrov
force-pushed
the
cluster_specific_optimization
branch
from
bebe890 to
71fa86d
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
WIP branch for the current prototype of the design proposed in #25010.
Motivation
This is a draft PR meant to facilitate discussion of implementation details related to the design proposed in #25010.
Tips for reviewer
Do not review this, unless I specifically ask you to look at certain parts for high-level feedback.
Checklist
$T ⇔ Proto$Tmapping (possibly in a backwards-incompatible way), then it is tagged with aT-protolabel.