Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handling tablet split should modify the task configuration #140

Open
vaibhav-yb opened this issue Jan 4, 2023 · 0 comments
Open

Handling tablet split should modify the task configuration #140

vaibhav-yb opened this issue Jan 4, 2023 · 0 comments
Assignees
Labels
branch/main Tickets related to the branch master enhancement New feature or request

Comments

@vaibhav-yb
Copy link
Collaborator

Currently we handle tablet split by modifying the polling list in place where we remove the parent tablet from the list and add the child tablets.

The above logic causes an issue in case one of the task is restarted. Let's say when we start the connector, we divide the tablets into task configs. Now this tablet list gets interpreted at the task level and is polled onto. Now let's say a tablet A gets split into B and C, we will modify the polling list but the task config remains the same - in this case, if a task is restarted, we will again try to poll on the parent tablet A (which may be deleted by this time now) and thus the task will start throwing errors.

The proposed solution over here is that we modify the task configuration itself and let the Kafka Connect runtime restart the connector so that we start the whole process from the top again.

@vaibhav-yb vaibhav-yb added enhancement New feature or request branch/main Tickets related to the branch master labels Jan 4, 2023
@vaibhav-yb vaibhav-yb self-assigned this Jan 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
branch/main Tickets related to the branch master enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant