-
Notifications
You must be signed in to change notification settings - Fork 167
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
docs: add decisions/20240313-try-w3up.md (#2518)
Motivation: * write decision to implement first part of storacha/project-tracking#7 Status * [x] initial plan drafted * [x] update based on feedback --------- Co-authored-by: Vasco Santos <[email protected]>
- Loading branch information
1 parent
aeb0d36
commit 06712ad
Showing
1 changed file
with
169 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,169 @@ | ||
| # nft.storage should try w3up for web3 storage of new uploads | ||
|
|
||
| Authors | ||
|
|
||
| - [email protected] | ||
|
|
||
| ## Problem Statement | ||
|
|
||
| nft.storage needs to offer web3 storage and currently does so using a homegrown solution that may be expensive to operate, maintain, and scale. It's Filecoin aggregationa and deal-making functionality relies on [nftstorage/dagcargo](https://github.com/nftstorage/dagcargo) running off-site, which may be decommissioned as soon as 2024-03-26. | ||
|
|
||
| Meanwhile the https://web3.storage team has productized a scalable web3 storage solution (named [w3up](https://github.com/web3-storage/w3up/)), and we think we may be able to deliver the same functionality that nft.storage needs for cheaper than it costs nft.storage to operate and improve the homegrown solution they use today that relies on [dagcargo](https://github.com/nftstorage/dagcargo). | ||
|
|
||
| But it's not clear how much it would cost for nft.storage to adopt web3.storage and interoperate with w3up. | ||
|
|
||
| If we lower the cost of adoption enough, perhaps nft.storage would be happy to ship the integration and use w3up/web3.storage as their storage/ipfs-serving/filecoin-dealmaking partner for the foreseeable future. | ||
|
|
||
| If we scout out the integration a bit, in the best case we'll enable short-term adoption of the product and, no matter what, we'll learn more about how usable our tools are in this real-world integration style, and we can use that to inform product improvements that hopefully help adoption of web3.storage in medium term. | ||
|
|
||
| ## Goals | ||
|
|
||
| - enable nft.storage to store new uploads using web3.storage | ||
| - enable nft.storage to migrate to web3.storage, have an account there we have in our CRM, access console.web3.storage, etc | ||
|
|
||
| ## Non-Goals | ||
|
|
||
| - migrate old nft.storage uploads | ||
| - fully remove reliance on dagcargo | ||
| - however, this work may serve as useful input to that. If nft.storage can upload using w3up, filecoin aggregation will happen via [w3-filecoin](https://github.com/web3-storage/specs/blob/main/w3-filecoin.md). And if we have that, it reduces reliance on dagcargo. | ||
| - anything about nft.storage read and filecoin info APIs. We only need to decide that if/when we have nft.storage uploads going through web3.storage and into filecoin deals. This decision is just to ensure we can get to that point. | ||
| - changing how nft.storage UI does uploads. e.g. it's a non-goal to have nft.storage UI upload things in a different way than it does now, which appears to be handling a POST /uploads/ http request with the file in the request body. (iiuc it looks like request body is a CAR) | ||
|
|
||
| ## Proposed Solution | ||
|
|
||
| ### convert nft.storage infra to use w3up-client as a normal customer of w3up | ||
|
|
||
| > convert NFT storage infra to use w3up client to act as a normal customer of w3up and go through normal flow to add the data into w3up. | ||
| https://github.com/w3s-project/project-tracking/issues/7 | ||
|
|
||
| Rationale | ||
|
|
||
| - "Then all existing filecoin pipeline stuff just happens by default" | ||
| - enables proving that web3.storage's w3-filecoin service can be a long-term replacement of dagcargo for nft.storage filecoin dealmaking | ||
| - lowers cost of adoption of web3.storage by nft.storage by proving out the integration incrementally | ||
|
|
||
| #### Architectural Changes | ||
|
|
||
| - when nft.storage api handles request POST /upload/ (e.g. from the nft.storage ui), the uploaded file should be stored using up.web3.storage in a space associated with an web3.storage account controlled by an administrator of nft.storage | ||
| - initially, this will only happen when a feature switch is enabled | ||
|
|
||
| #### Data Model Changes | ||
|
|
||
| Minimal. Only new configuration. | ||
|
|
||
| We'll need to decide which storage space that uploads will be added to. | ||
| To start, we'll just making everything store in a configured storage space, and the app will read that from configuration passed in environment variables, similar to env vars in [add-to-web3](https://github.com/web3-storage/add-to-web3?tab=readme-ov-file#generating-a-secret_key-and-proof). | ||
|
|
||
| ##### `W3_NFTSTORAGE_SPACE` Environment Variable | ||
|
|
||
| configures the web3.storage space that nfts will be stored in. | ||
|
|
||
| Example: `did:key:z6MkfbsERaJ7rtJQWMWtYMxNED56bhQMgrNu8CRdUjB5LfRp` | ||
|
|
||
| ##### `W3_NFTSTORAGE_PRINCIPAL` Environment Variable | ||
|
|
||
| configures how nft.storage will authenticate to web3.storage when sending invocations to store things. e.g. it may contain an ed25519 keypair as a CID with raw multihash. | ||
|
|
||
| ##### `W3_NFTSTORAGE_PROOF` Environment Variable | ||
|
|
||
| configures the capabilities that nft.storage has access to when interacting with web3.storage to store nfts. These capabilities will usually be UCAN delegations whose audience is the identifier of `W3_NFTSTORAGE_PRINCIPAL`. | ||
| W3_NFTSTORAGE_PROOF needs to have proof rooted in W3_NFTSTORAGE_SPACE that authorize W3_NFTSTORAGE_PRINCIPAL to store in W3_NFTSTORAGE_SPACE. | ||
|
|
||
| ##### `NFTSTORAGE_W3S_ALLOW_ACCOUNTS` Environment Variable | ||
|
|
||
| configures feature switch for which nftstorage accounts will have new uploads stored in web3.storage. | ||
|
|
||
| Note: this environment variable may not be a permanent addition to the codebase. It's only meant to be used as a feature switch that decouples enabling the new functionality from deploying the new code. After testing, we may remove or change this feature switch when it is no longer useful. | ||
|
|
||
| Format: JSON Array of Account Identifiers | ||
|
|
||
| #### UI Changes | ||
|
|
||
| None. But the existing UI workflow of uploading via https://nft.storage/files/ and form should behave just like they do now. But after this change, there should be a new side effect, which is that the upload should appear in the listing of uploads for the configured `W3_NFTSTORAGE_SPACE` (e.g. via w3cli `w3 ls` or in console.web3.storage). | ||
|
|
||
| ## Risks | ||
|
|
||
| ### degrading nft.storage's existing functionality | ||
|
|
||
| We wouldn't want these changes to have any negative impact on users that rely on nft.storage's current setup. We only want to enable evaluation of new ways of delivering on that functionality that are seamless for end-users. | ||
|
|
||
| #### Mitigation: feature switch for initial rollout | ||
|
|
||
| We'll deploy this functionality to use web3.storage initially behind an opt-in feature switch. Once deployed, most uploads will not use the web3.storage functionality. Only accounts we add to a configured allow-list will get persisted to w3up. This removes risk of everyday users being impacted by bugs in the new code, while still enabling employees we allowlist to test the code in production. After testing in production, we can add more to the allowlist (e.g. perhaps nft.storage employees), or we can remove the feature switch to roll out the functionality for all users of nft.storage. | ||
|
|
||
| ### Are there any backwards-incompatible changes? | ||
|
|
||
| No | ||
|
|
||
| ### Does this project have special implications for security and data privacy? | ||
|
|
||
| Some minor implications. | ||
| After these changes, uploads will be shared with web3.storage in a new way. However, my understanding is that nft.storage is for public data, and this data is already shared with web3.storage through other means for the purpose of storage/serving on ipfs/filecoin. So there are no new data sharing concerns. | ||
|
|
||
| ### Could this change significantly increase load on any of our backend systems? | ||
|
|
||
| Unlikely. As I understand it, the status quo for file uploads on nft.storage is that it is sent the full file in an http request, and then must store it. There may be a more work to do in some cases to store it (encoding/signing/invoking web3.storage), but this shouldn't be a **significant** increase in load. In the worst case, it may be 100% more cpu used for each file uploads, but we can mitigate the rollout of that via feature switch, and based on testing we can find ways of optimizing it. | ||
|
|
||
| ### Does this project have any dependencies? | ||
|
|
||
| #### dependency on collaboration with nft.storage | ||
|
|
||
| This project can't succeed without a sponsor from nft.storage and willingness to try out the proposed changes to the nft.storage codebase in a production-like environment. If we have this, we can help prove out nft.storage storing data via web3.storage and relying on web3.storage productized services for serving on ipfs and filecoin dealmaking. | ||
|
|
||
| ## Alternative Solutions | ||
|
|
||
| I don't know of any alternate solutions I can link to that provide a path to servicing nft.storage's web3 storage needs without relying on dagcargo. If you know of some, please add them here. | ||
|
|
||
| ### Use w3-filecoin Directly from nft.storage (without w3up) | ||
|
|
||
| The subsystem of w3up that does filecoin dealmaking is called [w3-filecoin](https://github.com/web3-storage/specs/blob/main/w3-filecoin.md) and has an API that nft.storage could call directly (just like other parts of w3up call it directly). | ||
|
|
||
| @vasco-santos proposed this in [api.nft.storage onboarding to w3filecoin](https://hackmd.io/0K9xlcQRTp6pLps1bffxHQ). | ||
|
|
||
| @vasco-santos has weighed in on the tradeoffs of that proposal vs the decision documented here (emphasis added): | ||
|
|
||
| > w3filecoin was designed for multiple storefronts, so w3up/others, and my idea was to not go through w3up to avoid mixing the data at that level. And this together with the separate aggregation process for both. The general sense though is that **using w3up makes more sense on a perspective on having nft.storage as a user of w3up in the long run**. But if the expectation we have is it to be 3 months, making it a client of w3filecoin, while continuing to write to old bucket and so on, would make more sense to me. | ||
| > | ||
| > But after talking with Alan, Hannah, Reid, we opted on the w3up lane, and that is why I say that it seems out of the table. **I generally agree with using w3up because it is simpler if we won't care about splitting aggregation + have nft.storage in the long run as a client of w3up** | ||
| If we determine it is not desirable for nft.storage to use the entirety of w3up's normal functionality, we may still pursue this alternative solution. | ||
|
|
||
| ## Implementation and Rollout Plan | ||
|
|
||
| Prototype Happy path | ||
|
|
||
| - [x] @gobengo set up nft.storage localdev to be able to hack on packages/api | ||
| - [ ] nft.storage/api: can be configured with `W3_NFTSTORAGE_PRINCIPAL` and `W3_NFTSTORAGE_PROOF` and use them to get a w3up-client at runtime | ||
| - [ ] nft.storage/api: modify POST /upload/ handler to store using w3up-client@12 | ||
| - [ ] nft.storage/api: add feature switch requiring allowlist for new uploads to store in web3.storage | ||
|
|
||
| Rollout | ||
|
|
||
| - [ ] release modified nft.storage to staging (production ok too) but with web3.storage disabled-by-default via feature switch | ||
| - [ ] configure web3.storage feature switch to be enabled on staging for a testing account | ||
| - [ ] test on nft.storage staging. make sure uploading a file via staging.nft.storage ui results in the file being stored in a web3.storage space | ||
|
|
||
| Repeat | ||
|
|
||
| - Observe | ||
| - do we have engagement from nft.storage on this being viable? e.g. do we have them set up with a web3.storage account, payment method, etc? Does our hypothesis still make sense that by developing this, we're increasing chance of them becoming a paying customer of web3.storage? | ||
| - what comes up from demoing this to rest of team? | ||
| - Orient | ||
| - What is the web3.storage account and space that nft.storage wants to use in production? | ||
| - Decide | ||
|
|
||
| - should we roll back the change based on new information? | ||
| - do we have what we need to configure nft.storage production to write into a web3.storage space payed for by nft.storage designee? | ||
| - should we enable feature switch in prod allowlisting a known testing account? | ||
| - if we've done that and things work well, should we remove the feature switch and enable this by default for all uploads? | ||
| - If everything is looking promising about uploads going through web3.storage. Should we figure out: | ||
|
|
||
| > NFT storage read and filecoin info API's need to work against both w3up and legacy systems (and know which to use in each case) | ||
| from https://github.com/w3s-project/project-tracking/issues/7 | ||
|
|
||
| - Act | ||
| - P1: configure nft.storage production with a web3.storage space paid for by nft.storage | ||
| - If blocked: | ||
| - design/implement integration with nft.storage filecoin info apis |