You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
With the new shard download, we have Content-Range resumable downloads with integrity checks so we should be able to give a list of candidate download URLs (in list of priority) and exo can try downloading from each one in turn.
This should be an easy way to get P2P downloads working - each node exposes a HTTP endpoint to serve models to others with Content-Range support. Should just work.
we probably want to keep downloading from the same URL but periodically (e.g. every 5 secs) check if a better candidate is available. any error means try next candidate. also we would want to make sure we don't have partial writes to the file that can corrupt it - this wouldn't ever lead to a corrupt download since we have an integrity check but it might lead to infinite download loop
The text was updated successfully, but these errors were encountered:
With the new shard download, we have Content-Range resumable downloads with integrity checks so we should be able to give a list of candidate download URLs (in list of priority) and exo can try downloading from each one in turn.
This should be an easy way to get P2P downloads working - each node exposes a HTTP endpoint to serve models to others with Content-Range support. Should just work.
we probably want to keep downloading from the same URL but periodically (e.g. every 5 secs) check if a better candidate is available. any error means try next candidate. also we would want to make sure we don't have partial writes to the file that can corrupt it - this wouldn't ever lead to a corrupt download since we have an integrity check but it might lead to infinite download loop
The text was updated successfully, but these errors were encountered: