Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

11 to 12: Improvements to migration #145

Merged
merged 3 commits into from
Dec 13, 2021
Merged

11 to 12: Improvements to migration #145

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
9f59b73
Increase workers to 4 for reverts
hsanjuan Dec 10, 2021
d45a5c8
11-to-12: Use a patched version of go-ds-badger with 2m GC interval
hsanjuan Dec 10, 2021
844ea5a
go mod tidy after rebase
hsanjuan Dec 10, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions fs-repo-11-to-12/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ require (
github.com/ipfs/fs-repo-migrations/tools v0.0.0-20211209222258-754a2dcb82ea
github.com/ipfs/go-cid v0.0.7
github.com/ipfs/go-datastore v0.4.5
github.com/ipfs/go-ds-badger v0.2.7-0.20211210151007-a2805355dcf5 // indirect
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reliance on a non-tagged commit shouldn't hurt us here since we're using vendoring, right?

github.com/ipfs/go-filestore v1.0.0
github.com/ipfs/go-ipfs v0.8.0
github.com/ipfs/go-ipfs-ds-help v1.0.0
Expand Down
3 changes: 2 additions & 1 deletion fs-repo-11-to-12/go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -294,8 +294,9 @@ github.com/ipfs/go-ds-badger v0.0.5/go.mod h1:g5AuuCGmr7efyzQhLL8MzwqcauPojGPUaH
github.com/ipfs/go-ds-badger v0.0.7/go.mod h1:qt0/fWzZDoPW6jpQeqUjR5kBfhDNB65jd9YlmAvpQBk=
github.com/ipfs/go-ds-badger v0.2.1/go.mod h1:Tx7l3aTph3FMFrRS838dcSJh+jjA7cX9DrGVwx/NOwE=
github.com/ipfs/go-ds-badger v0.2.3/go.mod h1:pEYw0rgg3FIrywKKnL+Snr+w/LjJZVMTBRn4FS6UHUk=
github.com/ipfs/go-ds-badger v0.2.6 h1:Hy8jw4rifxtRDrqpvC1yh36oIyE37KDzsUzlHUPOFiU=
github.com/ipfs/go-ds-badger v0.2.6/go.mod h1:02rnztVKA4aZwDuaRPTf8mpqcKmXP7mLl6JPxd14JHA=
github.com/ipfs/go-ds-badger v0.2.7-0.20211210151007-a2805355dcf5 h1:ovdpQk2ZVK6eQLzMCZy1z2tJae7yvE8xaPUw4Pr1RqI=
github.com/ipfs/go-ds-badger v0.2.7-0.20211210151007-a2805355dcf5/go.mod h1:02rnztVKA4aZwDuaRPTf8mpqcKmXP7mLl6JPxd14JHA=
github.com/ipfs/go-ds-flatfs v0.4.5 h1:4QceuKEbH+HVZ2ZommstJMi3o3II+dWS3IhLaD7IGHs=
github.com/ipfs/go-ds-flatfs v0.4.5/go.mod h1:e4TesLyZoA8k1gV/yCuBTnt2PJtypn4XUlB5n8KQMZY=
github.com/ipfs/go-ds-leveldb v0.0.1/go.mod h1:feO8V3kubwsEF22n0YRQCffeb79OOYIykR4L04tMOYc=
Expand Down
7 changes: 1 addition & 6 deletions fs-repo-11-to-12/migration/swapper.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,12 +67,7 @@ func (cswap *CidSwapper) Revert(unswapCh <-chan Swap) (uint64, error) {
swapWorkerFunc := func() (uint64, uint64) {
return cswap.unswapWorker(unswapCh)
}
// We only run 1 worker for revert. Migrations
// many-cid-to-one-multihash mappings, but reverts can have the
// opposite. The unswapWorker keeps a cache to handle that, but
// this only works with a single worker. Otherwise we'd need
// complex syncing, or delayed removal (increased datastore size).
return cswap.runWorkers(1, swapWorkerFunc)
return cswap.runWorkers(NWorkers, swapWorkerFunc)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From @hsanjuan: The logic for why we only needed 1 worker is not true anymore. Before we kept track of whether a reverted block had a v0 reference so we did not delete the v0... but now we don't delete anything on revert.

}

// Run workers launches several workers to run the given function which returns
Expand Down
2 changes: 1 addition & 1 deletion fs-repo-11-to-12/vendor/github.com/ipfs/go-ds-badger/datastore.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion fs-repo-11-to-12/vendor/modules.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,8 @@ github.com/ipfs/go-datastore/mount
github.com/ipfs/go-datastore/namespace
github.com/ipfs/go-datastore/query
github.com/ipfs/go-datastore/sync
# github.com/ipfs/go-ds-badger v0.2.6
# github.com/ipfs/go-ds-badger v0.2.7-0.20211210151007-a2805355dcf5
## explicit
github.com/ipfs/go-ds-badger
# github.com/ipfs/go-ds-flatfs v0.4.5
github.com/ipfs/go-ds-flatfs
Expand Down