Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Individuals are copied during distributed evaluation #280

Closed
lukepmccombs opened this issue Jun 13, 2023 · 2 comments
Closed

Individuals are copied during distributed evaluation #280

lukepmccombs opened this issue Jun 13, 2023 · 2 comments

Comments

@lukepmccombs
Copy link
Collaborator

This is half an issue and half a question. Currently, when using distributed evaluation an Individual is passed to the executor to be evaluated. Due to pickling / unpickling, that leads to everything about the individual being deepcopyed which makes shared objects a no-go at the Individual level. Should this be fixed? The potential for adaptive fitness I mentioned in #279 is hampered by this, but that isn't a current feature of LEAP.

@SigmaX
Copy link
Collaborator

SigmaX commented Jun 27, 2023

Discussion with @lukepmccombs, @markcoletti:

The basic problem is that ops such as synchronous.eval_population() call client.map() on the population of Individual objects, mapping distrib.evaluate.evaluate() over them. This causes all Individuals to be sent out to the workers.

distrib.evaluate.evaluate() calls Individual.evaluate() and assigns some book-keeping attributes to each Individual.

To avoid memory hoggage it would be better if we could somehow pass phenotype, Problem tuples or such into the distributed function instead of whole Individual objects.

We aren't sure how to do that elegantly and flexibly, however... so we are going to procrastinate on this issue until it is actually a problem for an application.

@SigmaX SigmaX closed this as not planned Won't fix, can't repro, duplicate, stale Jun 27, 2023
@lukepmccombs
Copy link
Collaborator Author

Just wanted to point out that this also impacts Decoders, which may be likelier to have shared state.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants