Parallel manual tools #29
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This is also the beginning of extending the integration tests towards usage of various manual tools, starting with this newly accessible one.
These include separate class and isolation tests for a test object and also for a test node.
Drop excessive complexity from extra should_scan attribute in order to determine if the same previous static should_run switch was customized or not. Furthermore, scanning now returns an explicit value instead of using a side-effect over a widely modified attribute. However, the main step of improvement here is the logic collected back in the clean decision policy which can now be used to finally handle parallely reversible nodes. Specifying the complete run/clean decision policy in a specialized function will also allow us to add test status rerun decisions too later on but at present is mostly needed in order to be able to run/clean custom nodes from intersected graphs using the restored graph intersection and node children flagging functionality.
Add support for multi-worker cache updates (previous update/full manual steps) with node reversal instead of tests. These will only run on one graph instead of separate graphs for reversal and traversal objectives where the single graph is intersected with multiple other graphs and flagged for specific run/cleanup policy at each stage. Handling the install nodes also means also supporting all capabilities of the previous install manual step and integrating it into the same tool as well.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
The test objects we iterate over could be used to parse separate graphs terminating at object roots only that could then be connected as vm-specific cache updating branches. Let's also use the shared root refactoring opportunity to test it in more isolation and simplify root skipping during graph flagging. Note that flagging graph intersection works with disconnected graphs while flagging children works only with connected graphs.
These tools have to be performed once per worker environment without any regard on other workers. We can thus utilize a simple always-true run decision policy. A current limitation is that the workers would wait for each other to traverse the setup node (pseudotest) but this limitation will be lifted once we allow node reentrancy based on the same run decision policy (e.g. for parallel test retries). Also adapt the sample test development tool since it is based on the same implementation as all of the above.
We now capture the initial iterated objects in a graph where we also add each separately parsed node before starting the traversal with simple enough run decision where each worker should execute each test.
A fallout in the parent parsing and retrieval for permanent test objects was hidden by the fact the permanent vm creation tool was disabled for a while and not used recently enough after all the distributed testing branches.
More specifically, drop the (setup=)internal/deploy as we should now have any nodes (internal or leaf ones) already accessible via any test restriction and redeploying is both (1) emulated scripting that could include multiple tool calls like get->upload->control->set keeping the plugin commands atomic in nature and (2) gradually phased out at least in traditional form as newly developed tests are now recommended to deploy on-demand during the test pre-conditions.
This is based on tool types instead of all tools since the isolation test can better focus on and reduce the tool variantization instead.
This is a more useful way to error out in the context of the update-cache tool and its node children flagging which could happen for a singular reason.
The only exception to this is the `run_suite()` function where the job is first passed as an argument. The only affected module by this change is the runner module. The previous use of the job object was inconsistent. Some future todos will include simpler node environment setting possibly dropping the entire `set_environment` method.
User code that could import the available tools should now be able to use just the fewer published and most useful tools.
pevogam
force-pushed
the
parallel-manual-tools
branch
from
176fb0d to
f1d879d
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This branch adds support for parallelism to all manual tools reducing
the selection of such to the minimal set of truly reusable steps and
removing redundancy where possible, all while improving isolation
and integration testing for the more easily maintainable final set.