-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Investigate bulk_save_objects for notification inserts #1533
Conversation
…tch batch insert timing
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Awesome, thanks @terrazoon!
There were some general questions I had, and this is just for the saves/inserts - not any updates, right?
Thanks for writing/updating all of the tests, too, those look good!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Awesome, everything looks good to me now @terrazoon, thank you!
Description
The next roadblock on scaling up to millions of notifications a day is the fact that we current insert notifications to the db one at a time. Try batch inserts using redis, a scheduled task, and bulk_save_objects().
So instead of writing one at a time to the db, and immediately sending to AWS afterwards, we:
Security Considerations
With this new approach, notifications would held in redis for 10-50 seconds, meaning potentially there could be a few hundred notifications in redis at any particular time when big sends are happening.