Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add mode where each featurizer is applied individually #127

Merged
merged 2 commits into from
Feb 2, 2023
Merged

Add mode where each featurizer is applied individually #127

merged 2 commits into from
Feb 2, 2023

Conversation

ml-evs
Copy link
Collaborator

@ml-evs ml-evs commented Jan 10, 2023

This PR adds an option to the base featurizer to apply each sub-featurizer individually. This can be useful for timing featurisations and will open up the possibility of more flexible parallelisation (over features rather than structures).

@ml-evs ml-evs force-pushed the ml-evs/add_single_featurizer_mode branch from 9ebf9c8 to 06aee93 Compare January 10, 2023 15:59
@ml-evs ml-evs marked this pull request as ready for review January 23, 2023 14:54
@ml-evs ml-evs force-pushed the ml-evs/add_single_featurizer_mode branch from 06aee93 to 002f623 Compare January 23, 2023 14:54
@ml-evs ml-evs requested a review from ppdebreuck January 23, 2023 14:54
@ml-evs
Copy link
Collaborator Author

ml-evs commented Jan 23, 2023

@ppdebreuck this was mostly to deal with the issues raised in #126, let me know what you think! Can eventually build it out to enable distributed featurization where each process/node takes a different featurizer.

@ppdebreuck
Copy link
Owner

featurize_site should also benefit from the modes. I see we already featurized sequentially (i.e. single mode) up to now. From what I remember, I did that for renaming - with is already handled separately in the meanwhile.

ppdebreuck
ppdebreuck approved these changes Feb 2, 2023
View reviewed changes
Copy link
Owner

@ppdebreuck ppdebreuck left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the nice addition. I am strongly in favour of having a concurrent non-blocking compound per thread execution in the future.

@ppdebreuck ppdebreuck force-pushed the ml-evs/add_single_featurizer_mode branch from 65d30ec to 7c32124 Compare February 2, 2023 13:58
@ml-evs ml-evs merged commit aad123b into master Feb 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ppdebreuck ppdebreuck ppdebreuck approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@ml-evs @ppdebreuck