Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature Request] Kill the entire pipeline if input iterator raises an exception #107

Closed
renxinhe opened this issue Aug 4, 2023 · 0 comments
Labels
enhancement New feature or request

Comments

@renxinhe
Copy link

renxinhe commented Aug 4, 2023

Is your feature request related to a problem? Please describe.
In all the examples, the input is a simple, hardcoded list (e.g. [1, 2, 3]). In my case, my input function is an iterator function that may raise an exception during an arbitrary iteration. In that case, I'd like pypeln to invalidate the entire pipeline's work, and propagate the exception and optionally offer a way to do some clean up.

Describe the solution you'd like

import pypeln

def input_func():
    for i in range(10):
        if i < 7:
            yield i
        else:
            raise Exception()

def compute_func():
    # some arbitrary operation
    pass

try:
     encryption_stage = pl.process.each(
            compute_func,
            input_func,
            run=True,
    )
except Exception e:
     # allows me to handle input_func error here

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered, any example in any other framework

Additional context
#63 describes a way to handle error if compute_func throws an exception. Currently I don't see a way for pypeln to raise the exception raised by input_func. Can anyone see a good workaround for this issue as well? Thanks!

@renxinhe renxinhe added the enhancement New feature or request label Aug 4, 2023
@renxinhe renxinhe closed this as not planned Won't fix, can't repro, duplicate, stale Aug 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant