Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🐛 Use pg cascade delete on delete study resource #346

Open
wants to merge 7 commits into
base: master
Choose a base branch
from

Conversation

znatty22
Copy link
Member

@znatty22 znatty22 commented May 21, 2018

Partially addresses #329, read below

Problem

Deleting a single study takes so long that it times out before the delete is complete and the API returns a 503. This is because the request timeout on the webserver (gunicorn) is set to 30 sec. The deletion of a study takes longer than 30 sec (Chung takes ~2.5 minutes) because of two problems 1) a study has large tree of related entities and the dataservice is currently using the ORM to do a cascading delete of this large tree. 2) Genomic files and study files must also be deleted in the indexd service. This also takes minutes complete. Chung takes ~1.5 minutes just to delete genomic files from indexd.

Solution

This PR addresses bottleneck 1 above by deleting a study more efficiently using Postgres's cascading delete rather than the ORM's cascading delete. We may need to address bottleneck 2 by making delete study a background/long running task. We can do this either in this PR or another PR.

After implementing postgres cascading deletes, Chung takes ~1 seconds to be deleted. This does NOT include deleting indexd records from the indexd service.

  • Add ondelete=CASCADE to foreign keys
  • Update DELETE /studies/<kf_id> logic
    -- Delete study's genomic files from indexd
    -- Use db's cascading delete instead of the ORMs cascading delete

@znatty22 znatty22 added the bug Something isn't working label May 21, 2018
@znatty22 znatty22 self-assigned this May 21, 2018
@znatty22 znatty22 force-pushed the pg-cascade-delete branch from 7d8cf46 to b77cc5c Compare May 21, 2018 22:06
@znatty22 znatty22 force-pushed the pg-cascade-delete branch 5 times, most recently from 5af5ac0 to 62ddc99 Compare May 22, 2018 13:00
@znatty22 znatty22 changed the title 🐛 Fix delete study 🐛 Use pg cascade delete on delete study resource May 22, 2018
@znatty22 znatty22 force-pushed the pg-cascade-delete branch from 62ddc99 to 43ccc99 Compare May 22, 2018 15:28
@znatty22 znatty22 force-pushed the pg-cascade-delete branch from 43ccc99 to 8c6062b Compare May 22, 2018 15:32
@fiendish
Copy link
Contributor

fiendish commented Jun 15, 2021

What ever happened to this? Especially in relation to the many thousands of requests needed for https://github.com/kids-first/kf-utils-python/blob/master/kf_utils/dataservice/delete.py

@znatty22
Copy link
Member Author

@fiendish I think the reason we didn't merge is bc we ultimately decided we don't want to make deleting studies that easy?

@fiendish
Copy link
Contributor

lololol

@znatty22 znatty22 mentioned this pull request May 8, 2023
10 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants