Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: increase the receive_timeout to 24 hours for delete job #28231

Merged
merged 3 commits into from
Feb 3, 2025

Conversation

fuziontech
Copy link
Member

Problem

We timeout waiting for lightweight deletes to finish on CH deleting events

Changes

This will basically just let the job wait for the delete event to finish (should always finish in 24 hours) - if the workflow is killed it will pick up from the waiting on mutation node anyways so just lean into that

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

Does this work well for both Cloud and self-hosted?

How did you test this code?

Copy link
Contributor

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR Summary

This PR increases the ClickHouse delete job timeout from 10 minutes to 24 hours to prevent timeouts during event deletion operations.

  • Increased receive_timeout to 24 hours in dags/deletes.py to allow delete jobs to complete without timing out
  • Removed lightweight_deletes_sync config option since it's now handled at cluster level
  • Set mutations_sync and lightweight_deletes_sync to 0 for async operation mode
  • Updated documentation for max_execution_time parameter to be more general

1 file(s) reviewed, no comment(s)
Edit PR Review Bot Settings | Greptile

@fuziontech fuziontech requested a review from a team February 3, 2025 20:34
@orian orian changed the title fix: increase the recieve_timeout to 24 hours for delete job fix: increase the receive_timeout to 24 hours for delete job Feb 3, 2025
@fuziontech fuziontech merged commit fd8baf5 into master Feb 3, 2025
94 checks passed
@fuziontech fuziontech deleted the increase_timeout branch February 3, 2025 22:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants