Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Schema not automatically created in secondary warehouse #233

Closed
cheyney-w opened this issue Oct 11, 2024 · 3 comments
Closed

Schema not automatically created in secondary warehouse #233

cheyney-w opened this issue Oct 11, 2024 · 3 comments

Comments

@cheyney-w
Copy link

If you attempt to materialize a model in a second warehouse by setting its database property (and you've already materialized at least one model in your target database), you receive the following error:

('42000', '[42000] [Microsoft][ODBC Driver 18 for SQL Server][SQL Server]The specified schema name "dbt_mycustomschema" either does not exist or you do not have permission to use it. (2760) (SQLMoreResults)')

It seems that the fabric__list_schemas macro in dbt/include/fabric/macros/adapters/metadata.sql always runs against the target database, not the database configured for the relation. I think this could be resolved by adding a USE {{database}} statement immediately above the select ... from sys.schemas statement:

    USE {{database}};
    select  name as [schema]
    from sys.schemas {{ information_schema_hints() }} {{ apply_label() }}

Reproduction steps

  1. Create two warehouses in the same workspace: WH_Primary and WH_Secondary
  2. Configure the profiles target to connect to WH_Primary and the schema to be dbt_mycustomschema
  3. Create a model and run dbt. Confirm the model is successfully materialized in WH_Primary under dbt_mycustomschema
  4. Create a second model and configure its database to WH_Secondary.
  5. Run dbt again. You'll get the error above.

This is because when the adapter attempts to materialize the second model and checks for the existence of the schema, it checks in the WH_Primary database (in which it exists because of step 3) and skips creating it.

@prdpsvs
Copy link
Collaborator

prdpsvs commented Nov 28, 2024

I think adding USE database statement should work, however, there are few catalog queries which require additional work.
And I do not know the impact of this change with table/view/snapshot/clone materializations.

I will not prioritize this for V1.8.8. However, will start looking into this soon.

@boxysean
Copy link

I am working with a team who is also experiencing the exact same issue.

prdpsvs added a commit that referenced this issue Dec 23, 2024
* configuring local tests to run on user credentials. Dropping a relation correctly based on its type

* Addressing #243, #221, #228, #229, #232, #235 issues

* Updated test helper with ephemeral

* Updated unit tests

* Updated get_pyodbc_attrs_before_credentials method

* include only lines that start with order by

* Addressed issue #249,#240,#238,#233,#180,#168,#186,#52

* Ensure Testview Name Uniqueness with MD5 of Model Name and Invocation

* Updating integration tests
@prdpsvs
Copy link
Collaborator

prdpsvs commented Dec 23, 2024

Addressed this change in #251. All integration tests are passing. Test and let me know if you still this issue. Please re-open if required.

@prdpsvs prdpsvs closed this as completed Dec 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants