[RFC] Hard fork data migration

[psteckler]

RFC on getting data from the mainnet archive db to a db using the berkeley schema.

The branch feature/berkeley-db-migrator is used in #12906.

The branch feature/add-berkeley-account-tables is used in #14339.

Part of #12676.

[psteckler]

!ci-build-me

[ghost-not-in-the-shell]

You may want to delete the p here

[psteckler]

whoops

[psteckler]

!ci-build-me

[ghost-not-in-the-shell]

Typo here.

[psteckler]

Whoops, again.

[kaozenn]

• After the HF (Hard Fork), will the "bucket for storing migrated database dumps" become the new reference bucket for retrieving Archive DB Dumps?

• Post HF, could we designate an Archive Dump as a reference for data retrieval? Applications such as Rosetta would not only need to be refactored to read from the new schema but also to pull data from the new S3 Bucket.

[psteckler]

After the HF (Hard Fork), will the "bucket for storing migrated database dumps" become the new reference bucket for retrieving Archive DB Dumps?

I would expect there will be a archive dump cron job for the new mainnet, which will write to the existing mina-archive-dumps bucket (or maybe some new one). It won't be the bucket described here.

Post HF, could we designate an Archive Dump as a reference for data retrieval? Applications such as Rosetta would not only need to be refactored to read from the new schema but also to pull data from the new S3 Bucket.

The current mainnet archive dumps are named mainnet-archive-dump-<DATE>_nnnn.sql. That naming convention could continue, or a new name could be chosen. As mentioned, the same bucket could continue to be used, or a new one created, if desired.

The current bucket is in Google Cloud Storage, not S3, which is an Amazon product.

[kantp]

Looks good to me.

[kantp]

I would suggest ending migration at slot_tx_end in RFC 51, #14138. We'll only have empty blocks for fork resolution after that anyway.

[psteckler]

OK, in the first-phase migration app, in #12906, I've add a --end-global-slot command-line arg, and tested that. The second-phase app doesn't need to worry about the end slot, because it can only process what the first-phase app has produced.

If you omit that arg, the app will migrate only canonical blocks. My understanding is that there may be some pending blocks to be migration, so if you do provide that arg, the app will migrate both pending and canonical (but not orphaned) blocks.

[psteckler]

I removed the unresolved question, because I think this command-line arg solves the problem.

[deepthiskumar]

!ci-build-me

[deepthiskumar]

!approved-for-mainnet