RemoteCompaction Corruption: Bad table magic number

[xiebinlin]

I am currently using the remote compaction feature in RocksDB. On the client side, I implement Schedule and Wait, and it acts as the primary DB. On the server side, I use OpenAndCompact, and it serves as the secondary DB.
When server side sent the SST to client side, the client side load SST fail like following:

2024/12/05-16:20:06.082841 35345 [db/compaction/compaction_service_job.cc:182] src_file: ./rocks_db/./rocks_db_secondary/00aab702-652c-4ab4-8a9d-363ec796709f/002190.sst, tgt_file: ./rocks_db/002199.sst
2024/12/05-16:20:06.082866 35345 [db/compaction/compaction_service_job.cc:182] src_file: ./rocks_db/./rocks_db_secondary/00aab702-652c-4ab4-8a9d-363ec796709f/002191.sst, tgt_file: ./rocks_db/002200.sst
2024/12/05-16:20:06.082888 35345 [db/compaction/compaction_service_job.cc:182] src_file: ./rocks_db/./rocks_db_secondary/00aab702-652c-4ab4-8a9d-363ec796709f/002192.sst, tgt_file: ./rocks_db/002201.sst
2024/12/05-16:20:06.082908 35345 [db/compaction/compaction_service_job.cc:182] src_file: ./rocks_db/./rocks_db_secondary/00aab702-652c-4ab4-8a9d-363ec796709f/002193.sst, tgt_file: ./rocks_db/002202.sst
2024/12/05-16:20:06.082931 35345 [db/compaction/compaction_service_job.cc:182] src_file: ./rocks_db/./rocks_db_secondary/00aab702-652c-4ab4-8a9d-363ec796709f/002194.sst, tgt_file: ./rocks_db/002203.sst
2024/12/05-16:20:06.098799 35345 [WARN] [table/block_based/block_based_table_reader.cc:907] [./rocks_db/002198.sst] Tail prefetch size 1311 is calculated based on TailPrefetchStats.
2024/12/05-16:20:06.099022 35345 [WARN] [table/block_based/block_based_table_reader.cc:907] [./rocks_db/002199.sst] Tail prefetch size 1311 is calculated based on TailPrefetchStats.
2024/12/05-16:20:06.099163 35345 [WARN] [table/block_based/block_based_table_reader.cc:907] [./rocks_db/002200.sst] Tail prefetch size 1311 is calculated based on TailPrefetchStats.
2024/12/05-16:20:06.099276 35345 [WARN] [table/block_based/block_based_table_reader.cc:907] [./rocks_db/002201.sst] Tail prefetch size 1311 is calculated based on TailPrefetchStats.
2024/12/05-16:20:06.099386 35345 [WARN] [table/block_based/block_based_table_reader.cc:907] [./rocks_db/002202.sst] Tail prefetch size 1311 is calculated based on TailPrefetchStats.
2024/12/05-16:20:06.099565 35345 [WARN] [db/db_impl/db_impl_compaction_flush.cc:4012] Compaction error: Corruption: Bad table magic number: expected 9863518390377041911, found 8135895605147551567 in ./rocks_db/002202.sst
2024/12/05-16:20:06.099577 35345 [WARN] [db/error_handler.cc:392] Background IO error Corruption: Bad table magic number: expected 9863518390377041911, found 8135895605147551567 in ./rocks_db/002202.sst, reason 1
2024/12/05-16:20:06.115049 35345 [db/error_handler.cc:278] ErrorHandler: Set regular background error
2024/12/05-16:20:06.115256 34209 (Original Log Time 2024/12/05-16:19:20.638586) [db/db_impl/db_impl_compaction_flush.cc:2138] [data] Manual compaction starting
2024/12/05-16:20:06.115263 35345 (Original Log Time 2024/12/05-16:20:06.099459) [db/compaction/compaction_job.cc:920] [data] compacted to: base level 6 level multiplier 10.00 max bytes base 1073741824 files[4 0 0 0 0 0 1031] max score 0.60, estimated pending compaction bytes 0, MB/sec: 7.1 rd, 16.8 wr, level 6, files in(1, 4) out(6 +0 blob) MB in(81.8, 39.9 +0.0 blob) out(287.3 +0.0 blob), read-write-amplify(5.0) write-amplify(3.5) Corruption: Bad table magic number: expected 9863518390377041911, found 8135895605147551567 in ./rocks_db/002202.sst, records in: 6319145,
2024/12/05-16:20:06.115274 35345 (Original Log Time 2024/12/05-16:20:06.099536) EVENT_LOG_v1 {"time_micros": 1733386806099483, "job": 19, "event": "compaction_finished", "compaction_time_micros": 17887940, "compaction_time_cpu_micros": 17886574, "output_level": 6, "num_output_files": 6, "total_output_size": 301279103, "num_input_records": 6319145, "num_output_records": 6319145, "num_subcompactions": 1, "output_compression": "ZSTD", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [4, 0, 0, 0, 0, 0, 1031]}
2024/12/05-16:20:06.115293 35345 [ERROR] [db/db_impl/db_impl_compaction_flush.cc:3453] Waiting after background compaction error: Corruption: Bad table magic number: expected 9863518390377041911, found 8135895605147551567 in ./rocks_db/002202.sst, Accumulated background error counts: 1
2024/12/05-16:20:07.120653 34173 [db/db_impl/db_impl_write.cc:2272] [data] New memtable created with log file: #2204. Immutable memtables: 1.
2024/12/05-16:20:07.132206 35345 [db/wal_manager.cc:292] Move log file ./rocks_db/001630.log to ./rocks_db/archive/001630.log -- OK
2024/12/05-16:20:07.132249 35345 [db/wal_manager.cc:292] Move log file ./rocks_db/001645.log to ./rocks_db/archive/001645.log -- OK
2024/12/05-16:20:07.132275 35345 [db/wal_manager.cc:292] Move log file ./rocks_db/001655.log to ./rocks_db/archive/001655.log -- OK
2024/12/05-16:20:07.132498 35345 [file/delete_scheduler.cc:72] Deleted file ./rocks_db/002198.sst immediately, rate_bytes_per_sec 0, total_trash_size 0, total_size 97938034, max_trash_db_ratio 0.250000
2024/12/05-16:20:07.132549 35345 EVENT_LOG_v1 {"time_micros": 1733386807132539, "job": 19, "event": "table_file_deletion", "file_number": 2198}
2024/12/05-16:20:07.132590 35345 [file/delete_scheduler.cc:72] Deleted file ./rocks_db/002199.sst immediately, rate_bytes_per_sec 0, total_trash_size 0, total_size 97938034, max_trash_db_ratio 0.250000
2024/12/05-16:20:07.132604 35345 EVENT_LOG_v1 {"time_micros": 1733386807132601, "job": 19, "event": "table_file_deletion", "file_number": 2199}
2024/12/05-16:20:07.132638 35345 [file/delete_scheduler.cc:72] Deleted file ./rocks_db/002200.sst immediately, rate_bytes_per_sec 0, total_trash_size 0, total_size 97938034, max_trash_db_ratio 0.250000
2024/12/05-16:20:07.132659 35345 EVENT_LOG_v1 {"time_micros": 1733386807132656, "job": 19, "event": "table_file_deletion", "file_number": 2200}