From 4a1db2b3934c2c254992d85394899ab7a15659ad Mon Sep 17 00:00:00 2001 From: "It's me, CI" Date: Fri, 6 Oct 2023 12:56:48 -0700 Subject: [PATCH 1/5] [RFC] Hard fork data migration --- rfcs/0052-hard-fork-data-migration.md | 141 ++++++++++++++++++++++++++ 1 file changed, 141 insertions(+) create mode 100644 rfcs/0052-hard-fork-data-migration.md diff --git a/rfcs/0052-hard-fork-data-migration.md b/rfcs/0052-hard-fork-data-migration.md new file mode 100644 index 00000000000..78bcc5821bd --- /dev/null +++ b/rfcs/0052-hard-fork-data-migration.md @@ -0,0 +1,141 @@ +p## Summary +[summary]: #summary + +This document describes a strategy for migrating mainnet archive data +to a new archive data for use at the hard fork. + +## Motivation +[motivation]: #motivation + +We wish to have archive data available from mainnet, so that the +archive database at the hard fork contains a complete history of +blocks and transactions. + +## Detailed design +[detailed-design]: #detailed-design + +There are significant differences between the mainnet and proposed +hard fork database schemas. Most notably, the `balances` table in the +mainnet schema no longer exists, and is replaced in the new schema +with the table `accounts_accessed`. The data in `accounts_accessed` +cannot be determined statically from the mainnet data. There is also +a new table `accounts_created`, which might be determinable statically +The new schema also has the columns `min_window_density` and +`sub_window_densities` in the `blocks` table; those columns do not +exist in the `blocks` table for mainnet. + +To populate the new database, there can be two applications: + +- The first application migrates as much data as possible from the +mainnet database, and downloads precomputed blocks to get the window +density data. The `accounts_accessed` and `accounts_created` tables +are not populated in this step. This application runs against +the mainnet and the new database. + +- The second application, based on the replayer app, replays the +transactions in the partially-migrated database, and populates the +`accounts_accessed` and `accounts_created` tables. This application +also performs the checks performed by the standard replayer, except +that ledger hashes are not checked, because the hard fork ledger has +greater depth, which results in different hashes. This application +runs only against the new database. + +These applications can be run in sequence to get a fully-migrated +database. They should be able to work incrementally, so that part of +the mainnet database can be migrated and, as new blocks are added on +mainnet, the new data in the databannnnnse can be migrated. + +To obtain that incrementality, the first application can look at the +migrated database, and determine the most recent migrated block. It +can continue migrating starting at the next block in the mainnet +data. The second application can use the checkpointing mechanism +already in place for the replayer. A checkpoint file indicates the +global slot since genesis for starting the replay, and the ledger to +use for that replay. The application writes new checkpoint files as it +proceeds. + +To take advantage of such incrementality, there can be a cron job +that migrates a day's worth of data at a time (or some other interval). +With the cron job in place, at the time of the actual hard fork, only a small +amount of data will need to be migrated. + +The cron job will need Google Cloud buckets (or other storage): + + - a bucket to store migrated-so-far database dumps + - a bucket to store checkpoint files + +To prime the cron job, upload an initial database dump, and an +initial checkpoint file. Those can be created via these steps, +run locally: + - download a mainnet archive dump, and loading it into PostgreSQL + - create a new, empty database using the new archive schema + - run the first migration app against the mainnet and new databases + - run the second migration app with the `--checkpoint-interval` set + to some suitable value (perhaps 100), and starting with the + original mainnet ledger in the input file + - use `pg_dump` to dump the migrated database, upload it + - upload the most recent checkpoint file + +The cron job will perform these same steps in an automated fashion: + - pull latest mainnet archive dump, load into PostgresQL + - pull latest migrated database, load into PostgreSQL + - pull latest checkpoint file + - run first migration app against the two databases + - run second migration app, using the downloaded checkpoint file; checkpoint interval + should be smaller (perhaps 50), because there are typically only 200 or so blocks in a day + - upload migrated database + - upload most recent checkpoint file + +There should be monitoring of the cron job, in case there are errors. + +Just before the hard fork, the last few blocks can be migrated by running locally: +- download the mainnet archive data directly from the k8s PostgreSQL node, not from + the archive dump, load it into PostgreSQL +- download the most recent migrated database, load it into PostgresQL +- download the most recent checkpoint file +- run the first migration application against the two database +- run the second migration application using the most recent checkpoint file + +It is worthwhile to perform these last steps as a `dry run` to make sure all goes +well. Those steps can be run as many times as needed. + +## Drawbacks +[drawbacks]: #drawbacks + +If we want mainnet data to be available after the hard fork, there +needs to be migration of that data. + +## Rationale and alternatives +[rationale-and-alternatives]: #rationale-and-alternatives + +It may be possible to add or delete columns in the original schema to +perform some of the migration without transferring data between +databases. It would still be necessary to add the windowing data from +precomputed blocks, and to have a separate pass to populate the +`accounts...` tables. + +## Prior art +[prior-art]: #prior-art + +There are preliminary implementations of the two applications: + +- The first application is in branch `feature/berkeley-db-migrator`. +Downloading precomputed blocks appears to be the main bottleneck there, +so those blocks are downloaded in batches, which helps considerably. + +- The second application is in branch `feature/add-berkeley-accounts-tables`. + +There has been some local testing of these applications. + +## Unresolved questions +[unresolved-questions]: #unresolved-questions + +How do we limit the migration to the final block of mainnet? There could be +flags to the migration apps to stop at a given state hash or height. + +The second application populates the `accounts_created` table, but the +first application could do so, by examining the +`...account_creation_fee_paid` columns of the `blocks_user_commands` +table in the mainnet schema. The current implementation relies on +dynamic behavior, rather than static data, which overcomes potential +errors in that data. From 4edc6c406a93f9cb61ecdb6f162b3f3f725ebfe7 Mon Sep 17 00:00:00 2001 From: "It's me, CI" Date: Fri, 6 Oct 2023 13:01:32 -0700 Subject: [PATCH 2/5] remove stray character --- rfcs/0052-hard-fork-data-migration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0052-hard-fork-data-migration.md b/rfcs/0052-hard-fork-data-migration.md index 78bcc5821bd..41295bb8b9d 100644 --- a/rfcs/0052-hard-fork-data-migration.md +++ b/rfcs/0052-hard-fork-data-migration.md @@ -1,4 +1,4 @@ -p## Summary +## Summary [summary]: #summary This document describes a strategy for migrating mainnet archive data From 89f88e9900fc6d88fe046170287caa831874cca4 Mon Sep 17 00:00:00 2001 From: "It's me, CI" Date: Fri, 6 Oct 2023 13:07:57 -0700 Subject: [PATCH 3/5] remove more stray characters --- rfcs/0052-hard-fork-data-migration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0052-hard-fork-data-migration.md b/rfcs/0052-hard-fork-data-migration.md index 41295bb8b9d..06df7d89141 100644 --- a/rfcs/0052-hard-fork-data-migration.md +++ b/rfcs/0052-hard-fork-data-migration.md @@ -43,7 +43,7 @@ runs only against the new database. These applications can be run in sequence to get a fully-migrated database. They should be able to work incrementally, so that part of the mainnet database can be migrated and, as new blocks are added on -mainnet, the new data in the databannnnnse can be migrated. +mainnet, the new data in the database can be migrated. To obtain that incrementality, the first application can look at the migrated database, and determine the most recent migrated block. It From df636c5cbf27a2f85c53ea6957224889b8415a16 Mon Sep 17 00:00:00 2001 From: "It's me, CI" Date: Mon, 9 Oct 2023 18:41:55 -0700 Subject: [PATCH 4/5] mention existing cron jobs --- rfcs/0052-hard-fork-data-migration.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/rfcs/0052-hard-fork-data-migration.md b/rfcs/0052-hard-fork-data-migration.md index 06df7d89141..1de862ed36c 100644 --- a/rfcs/0052-hard-fork-data-migration.md +++ b/rfcs/0052-hard-fork-data-migration.md @@ -127,6 +127,10 @@ so those blocks are downloaded in batches, which helps considerably. There has been some local testing of these applications. +The replayer cron jobs for mainnet, devnet, and berkeley can serve as +a starting point for the implementation of the cron job described +here. + ## Unresolved questions [unresolved-questions]: #unresolved-questions From e9b717da4f87e7c6a2d9b1904f47e5bab70683fd Mon Sep 17 00:00:00 2001 From: "It's me, CI" Date: Fri, 20 Oct 2023 14:43:12 -0700 Subject: [PATCH 5/5] Remove unresolved question about ending migration --- rfcs/0052-hard-fork-data-migration.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/rfcs/0052-hard-fork-data-migration.md b/rfcs/0052-hard-fork-data-migration.md index 1de862ed36c..d4e21cf229f 100644 --- a/rfcs/0052-hard-fork-data-migration.md +++ b/rfcs/0052-hard-fork-data-migration.md @@ -134,9 +134,6 @@ here. ## Unresolved questions [unresolved-questions]: #unresolved-questions -How do we limit the migration to the final block of mainnet? There could be -flags to the migration apps to stop at a given state hash or height. - The second application populates the `accounts_created` table, but the first application could do so, by examining the `...account_creation_fee_paid` columns of the `blocks_user_commands`