referencefs: add streaming _open support #1778
Conversation
skshetry
force-pushed
the
referencefs-stream-open
branch
2 times, most recently
from
666d9b5 to
a598a9e
Compare
fsspec/implementations/reference.py
Outdated
| part_or_url, start0, end0 = self._cat_common(path) | ||
| if isinstance(part_or_url, bytes): | ||
| return io.BytesIO(part_or_url[start0:end0]) | ||
|
|
||
| protocol, _ = split_protocol(part_or_url) | ||
| if start0 is None and end0 is None: | ||
| return self.fss[protocol]._open( |
There was a problem hiding this comment.
Ideally, we could put these in ReferenceFile, but I avoided adding unnecessary redirection. That does mean we will do _cat_common twice if it ends up going to the ReferenceFile. It does not look very critical, so I think that's acceptable(?).
That said, I don't have a strong opinion on this and can move it if deemed necessary.
There was a problem hiding this comment.
This is fine. Maybe add a comment to say what it's doing.
There was a problem hiding this comment.
Added a small comment. PTAL.
There was a problem hiding this comment.
This looks totally good to me.
There is apparently something being picked up by codespell.
The dask installation failure in the downstream run would be fixed by removing dask-expr from the environment. Please do fix this.
The error in the s3fs might already be fixed, but is in any case due to a recent release of botocore and nothing to do with this PR.
fsspec/implementations/reference.py
Outdated
| part_or_url, start0, end0 = self._cat_common(path) | ||
| if isinstance(part_or_url, bytes): | ||
| return io.BytesIO(part_or_url[start0:end0]) | ||
|
|
||
| protocol, _ = split_protocol(part_or_url) | ||
| if start0 is None and end0 is None: | ||
| return self.fss[protocol]._open( |
There was a problem hiding this comment.
This is fine. Maybe add a comment to say what it's doing.
skshetry
force-pushed
the
referencefs-stream-open
branch
from
a598a9e to
4ed21b1
Compare
…fsspec < 2025.2.0 Following changes from fsspec/filesystem_spec#1778.
Closes #1771.