feat : load driver by uri like file:/// http:/// etc #582

AntoineMontane · 2025-01-25T17:21:30Z

solve #581

pi0 · 2025-01-27T11:25:30Z

This is a nice idea however as it has bundle size implications, I would keep it completely separate.

Can you please as first step, deduce diff Generate a new entry in dist that is an object mapping from URI scheme like "file", to driver object. (there should be no changes in driver src files)

(we also need to introduce a new subpath export like unstorage/full to export anything related to this from)

AntoineMontane · 2025-01-27T21:23:08Z

True, i was so displeased by the bundling issue i rewrote from scratch (i'm always amazed how good dynamic imports look from far away ;-) )
Now the driverFromUri takes a list of imported driver which means the bundling is left to the library consumer. More flexibility, more control, less issues ...

import http from "unstorage/drivers/http"
import memory from "unstorage/drivers/memory"

driverFromUri(uri, [http, memory])

We could still provide the "full bundle" if needed one way or another, or profiles: "node", "browser", "azure", etc.

BTW i saw that mongodb was not declared as peer / optional and presented issue with the full bundle in my consumer project. fs was also an issue client side ... that was the last straw for the dynamic import tentative.

src/drivers/fs-lite.ts

test/drivers/fs.test.ts

AntoineMontane · 2025-01-28T16:37:46Z

I think i should go back to the previous approach to export the regexp from each driver module. And keep the new one for the loader. What do you think of this ?

src/loader/all.ts

pi0 · 2025-01-29T17:43:07Z

src/loader/index.ts

+import { coerceQuery } from "./utils";
+import type { Driver, DriverFactory } from "../types";
+
+const RE = /^(?<protocol>[^:?]+)(:(?<base>[^?]*))?(\?(?<query>.*))?$/;


We should use new URL as parser.

new URL misses the ability to express relative urls:

> new URL('file:abc/def') URL { href: 'file:///abc/def', origin: 'null', protocol: 'file:', username: '', password: '', host: '', hostname: '', port: '', pathname: '/abc/def', search: '', searchParams: URLSearchParams {}, hash: '' }

> new URL('file:/abc/def') URL { href: 'file:///abc/def', origin: 'null', protocol: 'file:', username: '', password: '', host: '', hostname: '', port: '', pathname: '/abc/def', search: '', searchParams: URLSearchParams {}, hash: '' }

> new URL('http:/abc/def') URL { href: 'http://abc/def', origin: 'http://abc', protocol: 'http:', username: '', password: '', host: 'abc', hostname: 'abc', port: '', pathname: '/def', search: '', searchParams: URLSearchParams {}, hash: '' }

i have tries with ufo.parseUrl without much success either

url-parse library looks to be ok ; yet, i'm not sure it's a good idea to bring in another url parsing library

pi0 · 2025-01-29T17:44:29Z

test/loader.test.ts

+describe("loader", () => {
+  it("load driver", async () => {
+    const driver = loadFromUri(
+      'test:abc?string=def&number=1&boolean=true&array=[2,3,4]&object={"h":5,"i":6,"j":"7"}',


For being a valid URL, it should be something like this:

test:/abc?string=def&number=1&boolean=true&array=[2,3,4]&object={"h":5,"i":6,"j":"7"}

http url is a subset of all url (take data: as an example of url not having the slash). And we want to be able to express relative url not starting with /. Ex with http

http:relative/path => base: "relative/path"

http:/absolute/path => base: "/absolute/path"

http://fqdn/path => base: "http://fqdn/path"

pi0 · 2025-01-29T23:32:24Z

Unstorage is primarily targeting server runtimes. Supporting relative URLs is really an edge case that possible in browser-only environments.

(and we should have one entry exporting dynamic imports inside it / no extra browser.ts)

AntoineMontane · 2025-01-30T07:48:24Z

Is it wrong to support browser runtimes ? In fact i started my development especially for this cross-environment ability.
Relative url for file: or db+sqlite: is quite possible & not an edge case either

pi0 · 2025-01-30T08:51:15Z

js runtime support depends on drivers (if they support broeser they do) — we shouldn’t make an exception entey for browser.

re parsing, it kinda makes sense for some drivers but also incompatible with URL standards. To be honest i really have to locally play with some ideas my self at this point.

Please ping me when your work finished later when i could will try to work on it.

AntoineMontane · 2025-01-30T09:01:15Z

A RFC about the generality of url for reflexion :
https://datatracker.ietf.org/doc/html/rfc1738#section-2.1

I'm almost done for the generic loader ( but the part we are discussing on). I will work on some drivers (s3, ...) that i plan to use

AntoineMontane · 2025-01-30T09:17:45Z

Is it possible someway to mark node:fs external so that it doesn't represent an issue at client bundle build ?

pi0 · 2025-01-30T09:29:35Z

that's a bundler-specific config people should do.

Also, we need to have exactly syntax below in the generated list:

() => import("unstorage/drivers/name")

this is because when using import(unstorage/drivers/${name}) bundler cannot statically analyze and has to scan all possible patterns.

AntoineMontane · 2025-01-30T10:01:19Z

I've

refactored uri to url,
renamed protocol to scheme
refactored the factoryloader to dynamicaly import with raw string
dropped the browser bundle

It would be less deceptive to only include in the all bundle drivers that were verified or reworked to correctly handle the url protocol (+ i had to guess some schemes without completly understanding what it could mean)

AntoineMontane · 2025-01-30T12:35:08Z

Hey @pi0 Do you have a good advice to handle driver import location in the all bundle ?
I don't want to import from unstorage/drivers/ because it's not available until build and adds a step before testing.
If i import relatively from ../drivers/, it works in test but fails in production as the path is different (../../drivers/)
What was the point to output the build of drivers outside of the ./dist driectory ?

pi0 · 2025-02-03T09:38:46Z

src/drivers/http.ts

+  const base =
+    opts.base.startsWith("//") && opts.scheme
+      ? `${opts.scheme}:${opts.base}`
+      : opts.base;
+
+  const r = (key: string = "") => joinURL(base!, key.replace(/:/g, "/"));

  const rBase = (key: string = "") =>
-    joinURL(opts.base!, (key || "/").replace(/:/g, "/"), ":");
+    joinURL(base!, (key || "/").replace(/:/g, "/"), ":");


Please split this into a new PR with a clear description of new behavior

pi0 · 2025-02-03T09:40:07Z

src/drivers/s3.ts

@@ -36,7 +48,7 @@ export interface S3DriverOptions {
  /**
   * The name of the bucket.
   */
-  bucket: string;
+  bucket?: string;


endpoint should not be deprecated because of a new feature. If you like to introduce sessionToken please open one PR for it.

If you want to introduce base, another PR.

pi0 · 2025-02-03T09:40:25Z

src/loader/all.ts

(TODO for myself): this should be auto generated

AntoineMontane requested a review from pi0 as a code owner January 25, 2025 17:21

AntoineMontane force-pushed the main branch 3 times, most recently from 6099abe to d9bee91 Compare January 26, 2025 14:17

AntoineMontane marked this pull request as draft January 26, 2025 14:18

pi0 changed the title ~~Feature URI : load driver by uri like file:/// http:/// etc~~ feat : load driver by uri like file:/// http:/// etc Jan 27, 2025

AntoineMontane force-pushed the main branch from d9bee91 to f760b22 Compare January 27, 2025 21:18

pi0 reviewed Jan 28, 2025

View reviewed changes

src/drivers/fs-lite.ts Outdated Show resolved Hide resolved

pi0 reviewed Jan 28, 2025

View reviewed changes

test/drivers/fs.test.ts Outdated Show resolved Hide resolved

AntoineMontane force-pushed the main branch from f760b22 to d4b3662 Compare January 29, 2025 17:24

pi0 reviewed Jan 29, 2025

View reviewed changes

src/loader/all.ts Outdated Show resolved Hide resolved

pi0 reviewed Jan 29, 2025

View reviewed changes

AntoineMontane force-pushed the main branch 4 times, most recently from 14699a7 to 5bf8819 Compare January 29, 2025 19:00

chore(url): move DriverFactory type to types

50cc28a

AntoineMontane force-pushed the main branch from 5bf8819 to f2cdaab Compare January 30, 2025 09:53

AntoineMontane force-pushed the main branch from f2cdaab to 1a94fd2 Compare January 30, 2025 10:51

AntoineMontane force-pushed the main branch 3 times, most recently from 0fc8557 to 5cb7e08 Compare January 30, 2025 11:23

AntoineMontane force-pushed the main branch from 5cb7e08 to b6cd98d Compare February 1, 2025 20:39

Antoine Monnet added 2 commits February 3, 2025 08:38

feat(url): new loadFromUrl(url, {drivers}) function

6aa49d3

feat(url): add a bundle for all drivers

a39085c

AntoineMontane force-pushed the main branch from 8fd7419 to cfe6470 Compare February 3, 2025 07:38

AntoineMontane marked this pull request as ready for review February 3, 2025 07:39

pi0 reviewed Feb 3, 2025

View reviewed changes

src/loader/all.ts

Copy link

Member

pi0 Feb 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(TODO for myself): this should be auto generated

pi0 marked this pull request as draft February 3, 2025 09:40

Antoine Monnet added 7 commits February 4, 2025 13:53

doc: add URL loader docs

ff7985b

feat(url): adapt http driver for loadFromUrl

836f669

feat(s3): configure from standard environment variables

5276b3d

feat(s3): use sessionToken if available

763f2f2

fix(s3): remove endpoint check

d3e5a26

feat(url): options "base" replace endpoint

4861b2d

feat(s3): add options to set headers in setItem

6e49e27

AntoineMontane force-pushed the main branch from cfe6470 to 6e49e27 Compare February 11, 2025 22:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat : load driver by uri like file:/// http:/// etc #582

feat : load driver by uri like file:/// http:/// etc #582

AntoineMontane commented Jan 25, 2025

pi0 commented Jan 27, 2025 •

edited

Loading

AntoineMontane commented Jan 27, 2025 •

edited

Loading

AntoineMontane commented Jan 28, 2025

pi0 Jan 29, 2025

AntoineMontane Jan 29, 2025 •

edited

Loading

AntoineMontane Jan 30, 2025

AntoineMontane Jan 30, 2025

pi0 Jan 29, 2025

AntoineMontane Jan 29, 2025 •

edited

Loading

pi0 commented Jan 29, 2025

AntoineMontane commented Jan 30, 2025 •

edited

Loading

pi0 commented Jan 30, 2025

AntoineMontane commented Jan 30, 2025 •

edited

Loading

AntoineMontane commented Jan 30, 2025

pi0 commented Jan 30, 2025 •

edited

Loading

AntoineMontane commented Jan 30, 2025 •

edited

Loading

AntoineMontane commented Jan 30, 2025 •

edited

Loading

pi0 Feb 3, 2025

pi0 Feb 3, 2025

pi0 Feb 3, 2025

feat : load driver by uri like file:/// http:/// etc #582

Are you sure you want to change the base?

feat : load driver by uri like file:/// http:/// etc #582

Conversation

AntoineMontane commented Jan 25, 2025

pi0 commented Jan 27, 2025 • edited Loading

AntoineMontane commented Jan 27, 2025 • edited Loading

AntoineMontane commented Jan 28, 2025

pi0 Jan 29, 2025

Choose a reason for hiding this comment

AntoineMontane Jan 29, 2025 • edited Loading

Choose a reason for hiding this comment

AntoineMontane Jan 30, 2025

Choose a reason for hiding this comment

AntoineMontane Jan 30, 2025

Choose a reason for hiding this comment

pi0 Jan 29, 2025

Choose a reason for hiding this comment

AntoineMontane Jan 29, 2025 • edited Loading

Choose a reason for hiding this comment

pi0 commented Jan 29, 2025

AntoineMontane commented Jan 30, 2025 • edited Loading

pi0 commented Jan 30, 2025

AntoineMontane commented Jan 30, 2025 • edited Loading

AntoineMontane commented Jan 30, 2025

pi0 commented Jan 30, 2025 • edited Loading

AntoineMontane commented Jan 30, 2025 • edited Loading

AntoineMontane commented Jan 30, 2025 • edited Loading

pi0 Feb 3, 2025

Choose a reason for hiding this comment

pi0 Feb 3, 2025

Choose a reason for hiding this comment

pi0 Feb 3, 2025

Choose a reason for hiding this comment

pi0 commented Jan 27, 2025 •

edited

Loading

AntoineMontane commented Jan 27, 2025 •

edited

Loading

AntoineMontane Jan 29, 2025 •

edited

Loading

AntoineMontane Jan 29, 2025 •

edited

Loading

AntoineMontane commented Jan 30, 2025 •

edited

Loading

AntoineMontane commented Jan 30, 2025 •

edited

Loading

pi0 commented Jan 30, 2025 •

edited

Loading

AntoineMontane commented Jan 30, 2025 •

edited

Loading

AntoineMontane commented Jan 30, 2025 •

edited

Loading