Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dbt_utils.get_filtered_columns_in_relation has inconsistent type between parsing and runtime #969

Open
4 tasks
cvm-a opened this issue Nov 26, 2024 · 0 comments
Labels
bug Something isn't working triage

Comments

@cvm-a
Copy link

cvm-a commented Nov 26, 2024

Describe the bug

dbt_utils.get_filtered_columns_in_relation returns '' during parsing, but returns a list during runtime. This makes it very hard to use this with conditional concatenation or with with filters, since the type changes from parsing and execution.
This results in errors like can only concatenate str (not "list") to str

Steps to reproduce

When creating a macro that takes two relations, one of which is optional, and trying to get all the column names, we may wan tot concatenate the list of column names like

{% macro all_columns (table_ref1, table_ref2=none) -%}
{{
    return (
        dbt_utils.get_filtered_columns_in_relation(table_ref1) +
       (dbt_utils.get_filtered_columns_in_relation(table_ref2) if table_ref2 is not none else [])                            
)
}}
{%- endmacro %}


Expected results

The concatenation should just work -- I should get the columns of both table_ref1 and table_ref2 if table_ref2 is not none, otherwise, I should only get the columns of table_ref1.

Actual results

during parsing, dbt_utils.get_filtered_columns_in_relation(table_ref1) is just '', so we get an error can only concatenate str (not "list") to str. If I change the else to '' from [] then it will raise can only concatenate list (not "str") to list as a Compilation error for the specific model selected.

Screenshots and log output

System information

The contents of your packages.yml file:
packages:

  • package: dbt-labs/dbt_utils
    version: 1.1.1

Which database are you using dbt with?

  • postgres
  • redshift
  • [ *] bigquery
  • snowflake
  • other (specify: ____________)

The output of dbt --version:

Core:
  - installed: 1.8.7
  - latest:    1.8.9 - Update available!

  Your version of dbt-core is out of date!
  You can find instructions for upgrading here:
  https://docs.getdbt.com/docs/installation

Plugins:
  - bigquery: 1.8.3 - Up to date!

Additional context


has return('')

Are you interested in contributing the fix?

sure I would replace that with return([])

@cvm-a cvm-a added bug Something isn't working triage labels Nov 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working triage
Projects
None yet
Development

No branches or pull requests

1 participant