bulk delete using STS #799
Conversation
|
pre-commit unrelated |
| client = storage_transfer_v1.StorageTransferServiceClient() | ||
|
|
||
| # Define the transfer job | ||
| transfer_job = storage_transfer_v1.types.TransferJob( |
There was a problem hiding this comment.
Confused - is this a transfer job or a deletion job?
There was a problem hiding this comment.
I think it's a transfer job to an empty bucket
There was a problem hiding this comment.
Actually, it's a transfer from the Empty Bucker to the destination bucket, making sure delete_objects_unique_in_sink is true, which means "Whether objects that exist only in the sink should be deleted. "
There was a problem hiding this comment.
yes, it's a dumb trick for STS. I'll add a comment
| # List transfer operations for the job | ||
| transfer_operations = operations_client.list_operations("transferOperations", filter_string) | ||
|
|
||
| # Get the latest operation |
There was a problem hiding this comment.
latest by date - is this list guaranteed to be sorted?
| if __name__ == "__main__": | ||
| # Check for correct usage | ||
| if len(sys.argv) != 2: | ||
| print("Usage: python gcs_bulk_delete.py gs://bucket_name/path/to/delete") |
There was a problem hiding this comment.
| print("Usage: python gcs_bulk_delete.py gs://bucket_name/path/to/delete") | |
| print(f"Usage: python {sys.argv[0]} gs://bucket_name/path/to/delete") |
|
|
||
| parser = argparse.ArgumentParser(description="List directories that can be deleted.") | ||
| parser.add_argument("base_dir", help="The base directory to clean up.", type=str, nargs="+") | ||
| parser.add_argument("--age", help="The age in days of the checkpoints to delete.", type=int, default=30) |
There was a problem hiding this comment.
NI (maybe):
The age of the checkpoints to delete (in days)
No description provided.