Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(anomaly-detection): do not store data if operation takes too long #1701

Merged
merged 2 commits into from
Jan 9, 2025

Conversation

ram-senth
Copy link
Member

@ram-senth ram-senth commented Jan 8, 2025

This change aborts saving data if the batch detection takes too long. This will help reduce the instances when Seer data goes out of sync with alerting system.

With this implementation, both store_data and detect_anomalies APIs return 500 - server error if the detection takes longer than 4.5 mins.

@ram-senth ram-senth marked this pull request as ready for review January 8, 2025 04:54
@ram-senth ram-senth requested a review from a team as a code owner January 8, 2025 04:54
@ram-senth ram-senth force-pushed the anomaly_detection/abandon_long_running branch 5 times, most recently from 27f8e11 to 723bee1 Compare January 8, 2025 21:03
@ram-senth ram-senth force-pushed the anomaly_detection/abandon_long_running branch from 723bee1 to a666cae Compare January 8, 2025 23:31
@ram-senth ram-senth merged commit ccebdd9 into main Jan 9, 2025
11 checks passed
@ram-senth ram-senth deleted the anomaly_detection/abandon_long_running branch January 9, 2025 00:11
Copy link

sentry-io bot commented Jan 16, 2025

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

  • ‼️ ServerError: Batch detection took too long app.store_data_endpoint View Issue

Did you find this useful? React with a 👍 or 👎

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants