Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

joinp: refactor --cache=schema option #2370

Merged
merged 2 commits into from
Dec 23, 2024
Merged

Conversation

jqnatividad
Copy link
Collaborator

    --cache-schema <arg>   Create and cache Polars schema JSON files.
                           ‎ -2: treat all columns as String. A Polars schema file is created & cached.
                           ‎ -1: treat all columns as String. No Polars schema file is created.
                             0: do not cache Polars schema.
                             1: cache Polars schema
                                If set to 1 and the schema file/s do not exist, it will check if a
                                stats cache is available. If so, it will use it to derive a Polars
                                schema and save it. If there's no stats cache, it will infer the
                                schema using --infer-len and save the inferred schemas. 
                                Each schema file will have the same file stem as the corresponding
                                input file, with the extension ".pschema.json"
                                (data.csv's Polars schema file will be data.pschema.json)
                                If the file/s exists, it will load the schema instead of inferring it
                                (ignoring --infer-len) and attempt to use it for each corresponding
                                Polars "table" with the same file stem.

closes #2369

instead of just a bool option, give user the ability to control the polars schema created & cached
@jqnatividad jqnatividad merged commit 5964047 into master Dec 23, 2024
13 checks passed
@jqnatividad jqnatividad deleted the 2369-joinp_cache-schema branch December 23, 2024 15:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

qsv joinp --cache-schema Polars error: could not parse 0 as dtype bool
1 participant