Block & fail early if produce requests don't work #4
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The kafka-python docs[1] note that
producer.flush()
only blocksuntil all messages are put on the network, and it doesn't guarantee
delivery or success. Even if the provided credentials correspond to a
principal that doesn't have permission to produce to the topic,
everything might look like it has succeeded to the user:
The change proposed here (also based on the docs[1]) will cause each
iteration of the loop to block until the produce request has either
succeeded or failed:
NOTE: I haven't actually tried this notebook out, as I don't know how
to run notebooks yet and haven't set up an environment for that -- I
was just trying to understand why a buddy was not seeing a failure
when they should have, when running the code from this notebook. I
just copied the code from here into my local python interpreter. I
don't know if there's any special way that exceptions should be
handled in notebooks or anything like that.
[1] https://pypi.org/project/kafka-python/