Restore from cold storage errors with: The operation is not valid for the object's storage class

[philipmw]

I set up rustic with a pair of S3 buckets, one hot one cold. I created a backup, and now I am trying to restore it.

My rustic.toml:

[[backup.snapshots]]
sources = ["/"]

[repository]
repository = "opendal:s3"
repo-hot = "opendal:s3"

[repository.options]
append_only = "true"
allow_anonymous = "false"
region = "us-west-2"

[repository.options-cold]
bucket = "pmw-backups-laptop-rustic-cold"
default_storage_class = "DEEP_ARCHIVE"

[repository.options-hot]
bucket = "pmw-backups-laptop-rustic-hot"

Here's my attempt to restore the data, using rustic 0.9.5:

% ./rustic restore latest ~/rustic-latest-snapshot -p repo-password.dat
[INFO] using config ./rustic.toml
[INFO] repository opendal:pmw-backups-laptop-rustic-cold#opendal:pmw-backups-laptop-rustic-hot: password is correct.
[INFO] using cache at /Users/pmw/Library/Caches/rustic/7e2aeced83b5ea2923e9cd1f9391d762b015abedd588d6394bb24568c228fdea
[00:00:00] reading index...               ████████████████████████████████████████          1/1                                                                                                                                                      [00:00:00] getting latest snapshot...     ████████████████████████████████████████          1/1                                                                                                                                                      [00:00:00] collecting file information...                                                                                                                                                                                                            [INFO] total restore size: 1.4 GiB
Files:  38904 to restore, 564 unchanged, 0 verified, 0 to modify, 0 additional
Dirs:   0 to restore, 6761 to modify, 0 additional
[00:00:00] restoring file contents...     ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░        0 B/1.36 GiB   0 B/s        (ETA -)                                                                                                                        
[WARN] service=s3 name=pmw-backups-laptop-rustic-cold path=data/3e/3ec9d189e90d64c9da21f1bd9e5e86c006ee2a409267a1f23e395ed71240b816: read failed PermissionDenied (persistent) at read, context: { uri: https://s3.us-west-2.amazonaws.com/pmw-backups-laptop-rustic-cold/data/3e/3ec9d189e90d64c9da21f1bd9e5e86c006ee2a409267a1f23e395ed71240b816, response: Parts { status: 403, version: HTTP/1.1, headers: {"x-amz-request-id": "XGRZTREGWXR7D3XE", "x-amz-id-2": "RjEefVOLLLuTTHAkk0OfqredLCx44SgElyKMq5iO45VnUIAXY8HADygm5GVMyatrBUA8zBn5i+A=", "content-type": "application/xml", "transfer-encoding": "chunked", "date": "Mon, 09 Dec 2024 20:08:13 GMT", "server": "AmazonS3"} }, service: s3, path: data/3e/3ec9d189e90d64c9da21f1bd9e5e86c006ee2a409267a1f23e395ed71240b816, range: 1671612-1671839 } => S3Error { code: "InvalidObjectState", message: "The operation is not valid for the object's storage class", resource: "", request_id: "XGRZTREGWXR7D3XE" }
[WARN] service=s3 name=pmw-backups-laptop-rustic-cold path=data/04/0436190337a3424624733ab63715384d80e71401362e54589738fc7e8dbcf5fc: read failed PermissionDenied (persistent) at read, context: { uri: https://s3.us-west-2.amazonaws.com/pmw-backups-laptop-rustic-cold/data/04/0436190337a3424624733ab63715384d80e71401362e54589738fc7e8dbcf5fc, response: Parts { status: 403, version: HTTP/1.1, headers: {"x-amz-request-id": "XGRW997R9H8MDZV2", "x-amz-id-2": "PGXL3gBYx80NWQDnCjUgcDr3vkQ6dBBAetooiKJvD0bxieqO/56Bz8NNoV6j2tT/hpjZwlVt+z0=", "content-type": "application/xml", "transfer-encoding": "chunked", "date": "Mon, 09 Dec 2024 20:08:13 GMT", "server": "AmazonS3"} }, service: s3, path: data/04/0436190337a3424624733ab63715384d80e71401362e54589738fc7e8dbcf5fc, range: 0-37157614 } => S3Error { code: "InvalidObjectState", message: "The operation is not valid for the object's storage class", resource: "", request_id: "XGRW997R9H8MDZV2" }
Well, this is embarrassing.

rustic-rs had a problem and crashed. To help us diagnose the problem you can send us a crash report.

We have generated a report file at "/var/folders/45/kzx7hb411nb90kv5_f_phw6c0000gq/T/report-963d21b1-125f-48e2-a7c0-463adc141ff2.toml". Submit an issue or email with the subject of "rustic-rs Crash Report" and include the report as an attachment.

The generated report:

name = "rustic-rs"
operating_system = "Mac OS 15.1.1 [64-bit]"
crate_version = "0.9.5"
explanation = """
Panic occurred in file '/Users/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rustic_core-0.7.2/src/commands/restore.rs' at line 532
"""
cause = '''
called `Result::unwrap()` on an `Err` value: RusticError { kind: Backend, guidance: "Partially reading file `{path}` failed in the backend. Please check if the given path is correct.", docs_url: None, error_code: None, ask_report: false, existing_issue_urls: [], new_issue_url: None, context: [("path", "data/3e/3ec9d189e90d64c9da21f1bd9e5e86c006ee2a409267a1f23e395ed71240b816"), ("type", "Pack"), ("id", "3ec9d189"), ("offset", "1671612"), ("length", "228")], source: Some(PermissionDenied (persistent) at read => S3Error { code: "InvalidObjectState", message: "The operation is not valid for the object's storage class", resource: "", request_id: "XGRZTREGWXR7D3XE" }

Context:
   uri: https://s3.us-west-2.amazonaws.com/pmw-backups-laptop-rustic-cold/data/3e/3ec9d189e90d64c9da21f1bd9e5e86c006ee2a409267a1f23e395ed71240b816
   response: Parts { status: 403, version: HTTP/1.1, headers: {"x-amz-request-id": "XGRZTREGWXR7D3XE", "x-amz-id-2": "RjEefVOLLLuTTHAkk0OfqredLCx44SgElyKMq5iO45VnUIAXY8HADygm5GVMyatrBUA8zBn5i+A=", "content-type": "application/xml", "transfer-encoding": "chunked", "date": "Mon, 09 Dec 2024 20:08:13 GMT", "server": "AmazonS3"} }
   service: s3
   path: data/3e/3ec9d189e90d64c9da21f1bd9e5e86c006ee2a409267a1f23e395ed71240b816
   range: 1671612-1671839
), severity: None, status: None, backtrace: Some(<disabled>) }'''
method = "Panic"
backtrace = """
   0:        0x102b56180 - __mh_execute_header
   1:        0x102b55ec8 - __mh_execute_header
   2:        0x102c73954 - __mh_execute_header
   3:        0x1036d35ec - __mh_execute_header
   4:        0x1037c8abc - __mh_execute_header
   5:        0x1037c87a4 - __mh_execute_header
   6:        0x1037c8714 - __mh_execute_header
   7:        0x1037c8708 - __mh_execute_header
   8:        0x103900bdc - __mh_execute_header
   9:        0x103900ec4 - __mh_execute_header
  10:        0x103624704 - __mh_execute_header
  11:        0x103915c04 - __mh_execute_header
  12:        0x102db0800 - __mh_execute_header
  13:        0x102db0468 - __mh_execute_header
  14:        0x1037cca1c - __mh_execute_header
  15:        0x191ef72e4 - __pthread_deallocate
"""

I also tried passing the --warm-up argument, but rustic dies with the same error.

The S3 files data/04/0436190337a3424624733ab63715384d80e71401362e54589738fc7e8dbcf5fc and data/3e/3ec9d189e90d64c9da21f1bd9e5e86c006ee2a409267a1f23e395ed71240b816 do exist in the cold bucket. Their storage class is "Glacier Deep Archive".