Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add public utilities page #1814

Merged
merged 8 commits into from
Feb 23, 2024
Merged

feat: add public utilities page #1814

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
efb3b24
feat: add public utilities page
2color Feb 16, 2024
b5ce4c4
fix: clarify that the gateways are provided by the foundation
2color Feb 20, 2024
07e37df
Update definition & TOC
mishmosh Feb 21, 2024
e2dd031
Update public-utilities.md
mishmosh Feb 21, 2024
986785f
chore: update pln-ignore.txt
lidel Feb 21, 2024
22cac66
Apply suggestions from code review
2color Feb 22, 2024
3841b2f
feat: move faq to utilities page and add bootstrappers
2color Feb 22, 2024
a79e1ac
chore: update ignored words
2color Feb 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/.vuepress/config.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -187,6 +187,7 @@ module.exports = {
},
'/concepts/cod',
'/concepts/comparisons',
'/concepts/public-utilities',
'/concepts/usage-ideas-examples',
['/concepts/measuring', 'Measuring the network'],
'/concepts/faq',
Expand Down
33 changes: 16 additions & 17 deletions docs/concepts/ipfs-gateway.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,16 @@ related:

# IPFS Gateway

An _IPFS gateway_ provides an HTTP-based service that allows IPFS-incompatible browsers, tools and software to access IPFS content. For example, some browsers or tools like [Curl](https://curl.haxx.se/) or [Wget](https://www.gnu.org/software/wget/) don't support IPFS natively and cannot access to IPFS content using canonical addressing like `ipfs://{CID}/{optional path to resource}`. While tools like [IPFS Companion](https://github.com/ipfs-shipyard/ipfs-companion) add browser support for native IPFS URLs, this is not always an option. As such, there are multiple gateway types and <VueCustomTooltip label="A way to address data by its hash rather than its location (IPs)." underlined multiline>gateway providers</VueCustomTooltip> available so that applications of all kinds can interface with IPFS using HTTP.
An _IPFS gateway_ is a web-based service that gets content from the IPFS network and makes it available via HTTP, allowing IPFS-incompatible browsers, tools and software to benefit from the IPFS public network. For example, some browsers or tools like [Curl](https://curl.haxx.se/) or [Wget](https://www.gnu.org/software/wget/) don't support IPFS natively and cannot access to IPFS content using canonical addressing like `ipfs://{CID}/{optional path to resource}`. While tools like [IPFS Companion](https://github.com/ipfs-shipyard/ipfs-companion) add browser support for native IPFS URLs, this is not always an option. As such, there are multiple gateway types and <VueCustomTooltip label="A way to address data by its hash rather than its location (IPs)." underlined multiline>gateway providers</VueCustomTooltip> available so that applications of all kinds can interface with IPFS using HTTP.
2color marked this conversation as resolved.
Show resolved Hide resolved

This page discusses:

- The IPFS gateway request lifecycle
- The several types of gateways.
- Gateway role in the use of IPFS.
- [Gateway request lifecycle](#gateway-request-lifecycle)
- [Gateway providers](#gateway-providers)
- [Gateway types](#gateway-types)
- [Working with gateways](#working-with-gateways)
- [Implementing gateways](#implementing-gateways)
- [FAQs](#frequently-asked-questions-faqs)

## Gateway request lifecycle

Expand All @@ -33,7 +36,7 @@ When a client request for a CID reaches an IPFS gateway, the gateway first check

The CID retrieval process is composed of two parts, content discovery / routing and content retrieval:

1. In the **content discovery / routing** step, the gateway will determine <VueCustomTooltip label="An IPFS network peer that can provide data specified by a particular CID upon request." underlined multiline>provider</VueCustomTooltip> location; that is, _where_ the data specified by the CID can be found:
1. In the **content discovery / routing** step, the gateway will determine <VueCustomTooltip label="An IPFS network peer that can provide data specified by a particular CID upon request." underlined multiline>provider</VueCustomTooltip> location; that is, _where_ the data specified by the CID can be found:

- Asking peers that it is directly connected to if they have the data specified by the CID.
- Query the DHT for the IDs and network addresses of peers that have the data specified by the CID.
Expand All @@ -59,20 +62,16 @@ Your machine may host a gateway as a local service; e.g., at `localhost:8080`. Y

### Private gateways

Running [IPFS Desktop](https://github.com/ipfs-shipyard/ipfs-desktop#ipfs-desktop) or another form of IPFS node triggers connection attempts to other IPFS peers. Private network administrators may treat such connection attempts as potential security vulnerabilities. Private IPFS gateway servers located inside the private network and running a trusted code base provide an alternative architecture for read/write access to externally-hosted IPFS content.
_Private gateways_ are configured to limit access to requests from specific domains or parts of the public internet.

A gateway behind a firewall represents just one potential location for a private gateway. More generally, one may consider any gateway as a _private gateway_ when configured to limit access to requests from specific domains or parts of the public internet. This [tutorial configuring an IPFS gateway on a Google Cloud platform](https://blog.stacktical.com/ipfs/gateway/dapp/2019/09/21/ipfs-server-google-cloud-platform.html) includes a description of constraining access.
They are frequently, but not exclusively, used behind firewalls. Running [IPFS Desktop](https://github.com/ipfs-shipyard/ipfs-desktop#ipfs-desktop) or another form of IPFS node triggers connection attempts to other IPFS peers. Private network administrators may treat such connection attempts as potential security vulnerabilities. Private IPFS gateway servers located inside the private network and running a trusted code base provide an alternative architecture for read/write access to externally-hosted IPFS content.

### Public gateways

Public gateway operators include:
This [tutorial configuring an IPFS gateway on a Google Cloud platform](https://blog.stacktical.com/ipfs/gateway/dapp/2019/09/21/ipfs-server-google-cloud-platform.html) includes information on constraining access for a private gateway setup.

- Protocol Labs, which deploys the public gateway `https://ipfs.io`.
- Third-party public gateways, such as `https://cf-ipfs.com`.
### Public gateways

Protocol Labs maintains a [list of public gateways](https://ipfs.github.io/public-gateway-checker/) and their status.
For more information about public gateways, see the [](./public-utilities.md#public-ipfs-gateways)
2color marked this conversation as resolved.
Show resolved Hide resolved

[Web3IMG](https://web3img.com) also maintains an unofficial [list of public gateways](https://bestipfs.net). Please do your research before using any third-party gateways or websites.

## Gateway types

Expand All @@ -89,7 +88,7 @@ There are multiple gateway types, each with specific use case, security, perform

### Read-only gateways

_Read-only gateways_ are the simplest kind of gateway. This gateway type provides a way to fetch IPFS content using the HTTP GET method.
_Read-only gateways_ are the simplest kind of gateway. This gateway type provides a way to fetch IPFS content using the HTTP GET method.

### Authenticated gateways

Expand Down Expand Up @@ -163,7 +162,7 @@ Currently HTTP gateways may access both IPFS and IPNS services:

For more information on working with gateways, see [best practices](../how-to/gateway-best-practices.md) and [troubleshooting](../how-to/gateway-troubleshooting.md).

## Implementing using the spec
## Implementing gateways

If you would like to read the technical specifications for the various gateway types, and learn more about how to implement a gateway, see the [IPFS HTTP Gateways specification](https://specs.ipfs.tech/http-gateways/) page for more information.

Expand Down Expand Up @@ -213,4 +212,4 @@ No. The ipfs.io gateway is one of many portals used to view content stored by th

- [A Practical Explainer for IPFS Gateways – Part 1](https://blog.ipfs.tech/2022-06-09-practical-explainer-ipfs-gateways-1/), [Part 2](https://blog.ipfs.tech/2022-06-30-practical-explainer-ipfs-gateways-2/)
- [Kubo: Gateway configuration options](https://github.com/ipfs/kubo/blob/master/docs/config.md#gateway)
- [IPFS HTTP Gateways specification](https://specs.ipfs.tech/http-gateways/)
- [IPFS HTTP Gateways specification](https://specs.ipfs.tech/http-gateways/)
55 changes: 55 additions & 0 deletions docs/concepts/public-utilities.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
---
title: Public IPFS Utilities
sidebarDepth: 1
description: Information about hosted public utilities such as public gateways and hosted delegated routing endpoints
---

# Public IPFS Utilities

The IPFS Foundation (and Protocol Labs in the past) and several other organizations provide hosted public utilities to the community on a best-effort basis. As such, it is not intended to be part of your critical path or production infrastructure.

These include [IPFS Gateways](./ipfs-gateway.md) and hosted [Delegated Routing V1 endpoints](./nodes#delegated-routing).

These utilities make it easier to retrieve data from the IPFS network in resource-constrained environments such as browsers and low-powered devices.

## Public IPFS Gateways

The IPFS Foundation provides the following public gateways:

- **`https://ipfs.io`**: Path resolution gateway
2color marked this conversation as resolved.
Show resolved Hide resolved
- **`https://dweb.link`**: Subdomain resolution gateway
2color marked this conversation as resolved.
Show resolved Hide resolved
- **`https://trustless-gateway.link`**: [trustless only gateway](https://specs.ipfs.tech/http-gateways/trustless-gateway/)
2color marked this conversation as resolved.
Show resolved Hide resolved

These gateways block abusive content using the [Bad Bits Denylist](https://badbits.dwebops.pub/).

Check failure on line 23 in docs/concepts/public-utilities.md

View workflow job for this annotation

GitHub Actions / pr-content-check 

[vale] reported by reviewdog 🐶
[ipfs-docs-style.PLNSpelling] Did you really mean 'Denylist'?

Raw Output:
{"message": "[ipfs-docs-style.PLNSpelling] Did you really mean 'Denylist'?", "location": {"path": "docs/concepts/public-utilities.md", "range": {"start": {"line": 23, "column": 58}}}, "severity": "ERROR"}

To report abusive content and content breaches of the [IPFS Community Code of Conduct](https://ipfs.fyi/coc), please send an email with the CIDs to [email protected].

Technical operations are run by the Waterworks Community on behalf of the IPFS Foundation. To report any technical problems with these gateways, you can open an issue in the [Waterworks Community repository](https://github.com/ipshipyard/waterworks-community).

### Abuse Policy

When a takedown request for the ipfs.io and/or `dweb.link` gateway(s) is first received at [email protected], the sender will receive an automated system receipt. Next, someone from the takedown team will vet the request to validate that it is legitimate and should be accepted. Once the validity of the request has been established, it will be submitted for further processing.

Check failure on line 31 in docs/concepts/public-utilities.md

View workflow job for this annotation

GitHub Actions / pr-content-check 

[vale] reported by reviewdog 🐶
[ipfs-docs-style.PLNSpelling] Did you really mean 'takedown'?

Raw Output:
{"message": "[ipfs-docs-style.PLNSpelling] Did you really mean 'takedown'?", "location": {"path": "docs/concepts/public-utilities.md", "range": {"start": {"line": 31, "column": 8}}}, "severity": "ERROR"}

Check failure on line 31 in docs/concepts/public-utilities.md

View workflow job for this annotation

GitHub Actions / pr-content-check 

[vale] reported by reviewdog 🐶
[ipfs-docs-style.PLNSpelling] Did you really mean 'takedown'?

Raw Output:
{"message": "[ipfs-docs-style.PLNSpelling] Did you really mean 'takedown'?", "location": {"path": "docs/concepts/public-utilities.md", "range": {"start": {"line": 31, "column": 183}}}, "severity": "ERROR"}
2color marked this conversation as resolved.
Show resolved Hide resolved

From there, an internal system extracts the relevant information from the request and generates an entry for it in our takedown lists. The list is then published to our gateway nodes, and the content in question will return a 410 status when visited.

Check failure on line 33 in docs/concepts/public-utilities.md

View workflow job for this annotation

GitHub Actions / pr-content-check 

[vale] reported by reviewdog 🐶
[ipfs-docs-style.PLNSpelling] Did you really mean 'takedown'?

Raw Output:
{"message": "[ipfs-docs-style.PLNSpelling] Did you really mean 'takedown'?", "location": {"path": "docs/concepts/public-utilities.md", "range": {"start": {"line": 33, "column": 120}}}, "severity": "ERROR"}

Your browser may have a local cache of the content in question and might not reflect that something has been blocked on the gateways. To avoid browser caching, attempt to view the content using your browser's incognito or private mode. You can also prevent caching issues by using a command-line tool such as Curl or Wget.

## Other Public Gateways

Additionally, there's a community-maintained [tool for finding and testing public gateways](https://ipfs.github.io/public-gateway-checker/) such as the one operated by Cloudflare: `https://cf-ipfs.com`.

## Delegated Routing

While IPFS Gateways are immensely helpful in doing all the heavy lifting of finding providers for CIDs and retrieving them, they can be a choke point for retrieval and a point of centralization.

[_Delegated Routing_](./nodes.md#delegated-routing) endpoints are a key step towards eliminating the emergent centralization of public gateways thereby increasing the health of the network.

Browsers and low-powered devices can make a single HTTP call to a _Delegated Routing_ endpoint with the CID they are looking to retrieve. The endpoint returns the [multiaddresses](./glossary.md#multiaddr) of the providers for the CID, from which the browser can download directly.

Check failure on line 47 in docs/concepts/public-utilities.md

View workflow job for this annotation

GitHub Actions / pr-content-check 

[vale] reported by reviewdog 🐶
[ipfs-docs-style.PLNSpelling] Did you really mean 'multiaddresses'?

Raw Output:
{"message": "[ipfs-docs-style.PLNSpelling] Did you really mean 'multiaddresses'?", "location": {"path": "docs/concepts/public-utilities.md", "range": {"start": {"line": 47, "column": 165}}}, "severity": "ERROR"}

### Delegated Routing Endpoint

The IPFS Foundation provides a public delegated routing endpoint backed by [**someguy**](https://github.com/ipfs/someguy) with the URL **`https://delegated-ipfs.dev/routing/v1`**

Check failure on line 51 in docs/concepts/public-utilities.md

View workflow job for this annotation

GitHub Actions / pr-content-check 

[vale] reported by reviewdog 🐶
[ipfs-docs-style.PLNSpelling] Did you really mean 'someguy'?

Raw Output:
{"message": "[ipfs-docs-style.PLNSpelling] Did you really mean 'someguy'?", "location": {"path": "docs/concepts/public-utilities.md", "range": {"start": {"line": 51, "column": 79}}}, "severity": "ERROR"}

Under the hood, someguy handles requests by looking up the DHT and Network Indexer.

Check failure on line 53 in docs/concepts/public-utilities.md

View workflow job for this annotation

GitHub Actions / pr-content-check 

[vale] reported by reviewdog 🐶
[ipfs-docs-style.PLNSpelling] Did you really mean 'someguy'?

Raw Output:
{"message": "[ipfs-docs-style.PLNSpelling] Did you really mean 'someguy'?", "location": {"path": "docs/concepts/public-utilities.md", "range": {"start": {"line": 53, "column": 17}}}, "severity": "ERROR"}

This endpoint can also be useful for debugging discoverability of CIDs, for example: [https://delegated-ipfs.dev/routing/v1/providers/bafybeicklkqcnlvtiscr2hzkubjwnwjinvskffn4xorqeduft3wq7vm5u4](https://delegated-ipfs.dev/routing/v1/providers/bafybeicklkqcnlvtiscr2hzkubjwnwjinvskffn4xorqeduft3wq7vm5u4) will show the providers for the CID `bafybeicklkqcnlvtiscr2hzkubjwnwjinvskffn4xorqeduft3wq7vm5u4`.

Check failure on line 55 in docs/concepts/public-utilities.md

View workflow job for this annotation

GitHub Actions / pr-content-check 

[vale] reported by reviewdog 🐶
[ipfs-docs-style.PLNSpelling] Did you really mean 'discoverability'?

Raw Output:
{"message": "[ipfs-docs-style.PLNSpelling] Did you really mean 'discoverability'?", "location": {"path": "docs/concepts/public-utilities.md", "range": {"start": {"line": 55, "column": 48}}}, "severity": "ERROR"}
Loading