NFT.Storage gateway cache MVP

[dchoi27]

We want to provide users with an HTTP endpoint that can fetch any data off the public IPFS network, but be super performant with NFT.Storage data in particular.

The current work plan consists on:

• gateway.nft.storage naive implementation gateway.nft.storage - CF Worker #867
• setup Pinata gateway Setup Pinata Gateway #869
• gateway race with ipfs.io + CF + Pinata gateway.nft.storage - race with multiple gateways #868
• setup logtail Setup logtail for gateway #1036
• setup first Cache layer - CF caching on http request - validation - feat: gateway add cloudflare cache layer #1180 + Limit gateway cache size and keep track of file size distribution #1181
• track and prevent rate limiting on IPFS public gateways Track and Prevent IPFS public gateways rate limite #1165
• load test with gateway logs NFT.Storage Gateway - load test with gateway logs #870
• /ipfs/ route to gateway nft.storage/ipfs/<cid> (just a redirect) Add /ipfs route to API and proxy it to gateway #1049
• Setup rate limit on CF Worker Gateway - Setup Worker Rate limits #1196
• Get 2-5 users to sign up for testing
• Public Launch 🎉

Further plans on the pipeline:

• Website and client with Gateway domain URL Update website and client toGatewayURL util function to new gateway #1345
• Track wether requested content is stored in DB Gateway tracking whether requested content is in Database nftstorage.link#14
  • Have cache in API check feat: add cache to check endpoint #1391
• Augment gateway metrics Augment gateway metrics collected nftstorage.link#18
• Add all nice metrics to Grafana
• Improve rate limit on the integrated public gateways Gateway - Rate Limiting improvements with public IPFS gateways #1195
• (should have rate limit improvements in place) warm cache + async gateway read test async gateway read tests on every write #755
• (should have rate limit improvements in place) async gateway read test alerts on performance degradation NFT.Storage Gateway - monitoring and alerts on top of read test #871

Other ideas:

• Setup second Cache layer
  • Consider S3 with "complete" metadata flag
• add S3 as gateway alternative
• Analyze logs and optimize caching based on user patterns
• CID blocklist (+ potentially integrate with abuse (at) ipfs.io)
• Investigate sharding for partners
• Expand async gateway read test from multiple locations
• Advanced cache logic layer extensions
  • image resizing/compression
  • CBOR support for IPNFT
  • ...
• Add more gateways (Infura, Cloudflare, etc.)
• No-code API to resolve centralized URLs to CIDs Gateway works with any URL #1435

More notes: https://hackmd.io/@DzQs_KK1RBK4_lT36FeM0g/H11PUd7YY

[dchoi27]

@vasco-santos do we need to ticket making sure we're collecting the right logs (with request timestamp, CID, when we receive from each source, etc.) and hooking this up to Logtail? or is that already captured somewhere

[vasco-santos]

@dchoi27 , good point! I will add a task do hooking this with Logtail. Hopefully it is fast and easy, but given I am lacking context I will add it.