Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: setup retrieval server and client #681

Merged
merged 5 commits into from
Jan 17, 2025

Conversation

cernicc
Copy link
Member

@cernicc cernicc commented Jan 16, 2025

Description

The PR implements the retrieval server and client. The server is running a p2p node that listens for block requests. The client runs a temporary p2p node that is used to request the needed blocks by the provider. The new blocks are requested until we receive a whole unixfs DAG tree.

The next step is integrating the server with the storage provider.

Important points for reviewers

Start a simple server as a proof of concept:
cargo run --package polka-storage-retrieval --example simple_server --all-features

Request content from the server:
cargo run --bin polka-fetch -- --provider /ip4/127.0.0.1/tcp/8989 --output ./download.car --payload-cid bafybeiefli7iugocosgirzpny4t6yxw5zehy6khtao3d252pbf352xzx5q

Note: For content extraction, you should use go-car. The reason is that our mater crate currently doesn't know how to extract content from the encoded unixfs dag tree.

Checklist

  • Make sure that you described what this change does.
  • Have you tested this solution?
  • Were there any alternative implementations considered?
  • Did you document new (or modified) APIs?

@cernicc cernicc self-assigned this Jan 16, 2025
storage-retrieval/cli/src/main.rs Outdated Show resolved Hide resolved
storage-retrieval/cli/src/main.rs Outdated Show resolved Hide resolved
storage-retrieval/lib/examples/simple_server.rs Outdated Show resolved Hide resolved
storage-retrieval/lib/src/server.rs Show resolved Hide resolved
storage-retrieval/lib/src/lib.rs Outdated Show resolved Hide resolved
storage-retrieval/lib/src/client.rs Show resolved Hide resolved
storage-retrieval/lib/src/server.rs Outdated Show resolved Hide resolved
storage-retrieval/lib/src/client.rs Show resolved Hide resolved
storage-retrieval/lib/src/client.rs Outdated Show resolved Hide resolved
storage-retrieval/lib/src/client.rs Outdated Show resolved Hide resolved
@cernicc cernicc requested review from aidan46 and jmg-duarte January 17, 2025 09:12
@jmg-duarte jmg-duarte added the ready for review Review is needed label Jan 17, 2025
This was linked to issues Jan 17, 2025
@cernicc cernicc enabled auto-merge (squash) January 17, 2025 10:13
@cernicc cernicc merged commit 60b42db into develop Jan 17, 2025
5 checks passed
@cernicc cernicc deleted the feat/retrieval_server_client branch January 17, 2025 10:15
@th7nder th7nder mentioned this pull request Jan 23, 2025
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ready for review Review is needed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement retrieval cli app Implement retrieval server/client
3 participants