Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add multi-storage-client backend for file open #1455

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

jayya2
Copy link

@jayya2 jayya2 commented Feb 15, 2025

This PR adds support for the Multi-Storage Client (MSC) backend to handle object storage access in Lhotse. The changes include:

Features

  • New MSCIOBackend for handling MSC protocol URLs
  • URL transformation from existing protocols (e.g., s3://) to MSC format via
    • LHOTSE_MSC_OVERRIDE_PROTOCOLS env for supported protocols, e.g. s3:// -> msc://
    • LHOTSE_MSC_PROFILE env for profile/bucket name overrides, e.g. msc://my-bucket -> msc://my-profile

Implementation Details

  • Added URL transformation utilities for bucket/profile name handling
  • Integrated MSC backend into the default IO backend chain
  • Added unit tests for MSC functionality

Configuration

MSC behavior can be configured through environment variables:

  • LHOTSE_MSC_OVERRIDE_PROTOCOLS: Comma-separated list of protocols to override (e.g., "s3,gs")
  • LHOTSE_MSC_PROFILE: Profile name to use for bucket override

Dependencies

  • Requires multistorageclient package to be installed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant