-
Notifications
You must be signed in to change notification settings - Fork 104
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
clean up relation usage and default behavior #92
Closed
Closed
Changes from all commits
Commits
Show all changes
23 commits
Select commit
Hold shift + click to select a range
200ada7
auto-render relation names without database
dataders 139807e
fix import path
dataders a2b820e
need correct import order
dataders 2dae934
Merge branch 'master' of https://github.com/dbt-msft/dbt-sqlserver in…
dataders 7dac89f
to make error clearer
dataders 7dacfca
typo
dataders 3b785e4
changes that will make it into next release
dataders 1f952c7
throw back to #34
dataders 25a4253
will dispatch to sqlserver__ version anyway
dataders 56d4969
Merge branch 'posssibly_drop_incremental' of https://github.com/dbt-m…
dataders 46d8a02
safer bet for quoting
dataders 18903b4
re-enable tests
dataders ddd0b2a
do statement "do something"
dataders fe7514a
not needed
dataders 939e7f0
EXEC is needed after all
dataders 4d077c7
typo
dataders 1f11488
TSQL does not require parenthesizing the SELECT
dataders b49725f
document change
dataders 1d5ca8b
leave quoting as is
dataders 83c9b6b
Merge branch 'master' of https://github.com/dbt-msft/dbt-sqlserver in…
dataders 93202ce
simpler reference
dataders b392d71
clarify operation; minimize code
dataders 8d5f219
fix indentation
dataders File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,21 @@ | ||
from typing import Optional | ||
|
||
from dataclasses import dataclass | ||
|
||
from dbt.adapters.base.relation import BaseRelation, Policy | ||
from dbt.exceptions import RuntimeException | ||
|
||
|
||
|
||
|
||
@dataclass | ||
class SQLServerIncludePolicy(Policy): | ||
database: bool = False | ||
schema: bool = True | ||
identifier: bool = True | ||
|
||
|
||
@dataclass(frozen=True, eq=False, repr=False) | ||
class SQLServerRelation(BaseRelation): | ||
include_policy: SQLServerIncludePolicy = SQLServerIncludePolicy() | ||
quote_character: str = '"' | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -44,9 +44,9 @@ | |
{% macro sqlserver__create_schema(relation) -%} | ||
{% call statement('create_schema') -%} | ||
USE [{{ relation.database }}] | ||
IF NOT EXISTS (SELECT * FROM sys.schemas WHERE name = '{{ relation.without_identifier().schema }}') | ||
IF NOT EXISTS (SELECT * FROM sys.schemas WHERE name = '{{ relation.schema }}') | ||
BEGIN | ||
EXEC('CREATE SCHEMA {{ relation.without_identifier().schema }}') | ||
EXEC('CREATE SCHEMA {{ relation.schema }}') | ||
END | ||
{% endcall %} | ||
{% endmacro %} | ||
|
@@ -61,7 +61,7 @@ | |
{%- set schema_relation = adapter.get_relation(database=relation.database, | ||
schema=relation.schema, | ||
identifier=table) -%} | ||
{% do drop_relation(schema_relation) %} | ||
{% do adapter.drop_relation(schema_relation) %} | ||
{%- endfor %} | ||
|
||
{% call statement('drop_schema') -%} | ||
|
@@ -79,26 +79,13 @@ | |
{%- else -%} invalid target name | ||
{% endif %} | ||
{% call statement('drop_relation', auto_begin=False) -%} | ||
if object_id ('{{ relation.include(database=False) }}','{{ object_id_type }}') is not null | ||
if object_id ('{{ relation }}','{{ object_id_type }}') is not null | ||
begin | ||
drop {{ relation.type }} {{ relation.include(database=False) }} | ||
drop {{ relation.type }} {{ relation }} | ||
end | ||
{%- endcall %} | ||
{% endmacro %} | ||
|
||
{% macro sqlserver__drop_relation_script(relation) -%} | ||
{% if relation.type == 'view' -%} | ||
{% set object_id_type = 'V' %} | ||
{% elif relation.type == 'table'%} | ||
{% set object_id_type = 'U' %} | ||
{%- else -%} invalid target name | ||
{% endif %} | ||
if object_id ('{{ relation.include(database=False) }}','{{ object_id_type }}') is not null | ||
begin | ||
drop {{ relation.type }} {{ relation.include(database=False) }} | ||
end | ||
{% endmacro %} | ||
|
||
{% macro sqlserver__check_schema_exists(information_schema, schema) -%} | ||
{% call statement('check_schema_exists', fetch_result=True, auto_begin=False) -%} | ||
--USE {{ database_name }} | ||
|
@@ -108,19 +95,19 @@ | |
{% endmacro %} | ||
|
||
{% macro sqlserver__create_view_as(relation, sql) -%} | ||
create view {{ relation.schema }}.{{ relation.identifier }} as | ||
{# TSQL does not require parenthesizing SELECT statement #} | ||
create view {{ relation }} as | ||
{{ sql }} | ||
{% endmacro %} | ||
|
||
|
||
{% macro sqlserver__rename_relation(from_relation, to_relation) -%} | ||
{% call statement('rename_relation') -%} | ||
EXEC sp_rename '{{ from_relation.schema }}.{{ from_relation.identifier }}', '{{ to_relation.identifier }}' | ||
EXEC sp_rename '{{ from_relation }}', '{{ to_relation.identifier }}' | ||
IF EXISTS( | ||
SELECT * | ||
FROM sys.indexes | ||
WHERE name='{{ from_relation.schema }}_{{ from_relation.identifier }}_cci' and object_id = OBJECT_ID('{{ from_relation.schema }}.{{ to_relation.identifier }}')) | ||
EXEC sp_rename N'{{ from_relation.schema }}.{{ to_relation.identifier }}.{{ from_relation.schema }}_{{ from_relation.identifier }}_cci', N'{{ from_relation.schema }}_{{ to_relation.identifier }}_cci', N'INDEX' | ||
WHERE name='{{ from_relation.schema }}_{{ from_relation.identifier }}_cci' and object_id = OBJECT_ID('{{ from_relation }}')) | ||
EXEC sp_rename N'{{ from_relation }}.{{ from_relation.schema }}_{{ from_relation.identifier }}_cci', N'{{ from_relation.schema }}_{{ to_relation.identifier }}_cci', N'INDEX' | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why are we prefixing with the entire relation now? That includes double-quoted schema and relation names, which is creating really bizarre output in my run:
|
||
{%- endcall %} | ||
{% endmacro %} | ||
|
||
|
@@ -133,42 +120,38 @@ | |
sys.indexes WHERE name = '{{cci_name}}' | ||
AND object_id=object_id('{{relation_name}}') | ||
) | ||
DROP index {{full_relation}}.{{cci_name}} | ||
DROP index {{relation}}.{{cci_name}} | ||
CREATE CLUSTERED COLUMNSTORE INDEX {{cci_name}} | ||
ON {{full_relation}} | ||
ON {{relation}} | ||
{% endmacro %} | ||
|
||
{% macro sqlserver__create_table_as(temporary, relation, sql) -%} | ||
{%- set as_columnstore = config.get('as_columnstore', default=true) -%} | ||
{% set tmp_relation = relation.incorporate( | ||
path={"identifier": relation.identifier.replace("#", "") ~ '_temp_view'}, | ||
type='view')-%} | ||
{%- set temp_view_sql = sql.replace("'", "''") -%} | ||
{%- set as_columnstore = config.get('as_columnstore', default=true) -%} | ||
{% set tmp_relation = relation.incorporate( | ||
path={"identifier": relation.identifier.replace("#", "") ~ '_temp_view'}, | ||
type='view') -%} | ||
|
||
{{ sqlserver__drop_relation_script(tmp_relation) }} | ||
{% do adapter.drop_relation(tmp_relation) %} | ||
{% do adapter.drop_relation(relation) %} | ||
|
||
{{ sqlserver__drop_relation_script(relation) }} | ||
{% set view_sql = create_view_as(tmp_relation, sql) %} | ||
{%- set view_sql_escpd = view_sql.replace("'", "''") -%} | ||
EXEC('{{ view_sql_escpd }}'); | ||
|
||
EXEC('create view {{ tmp_relation.schema }}.{{ tmp_relation.identifier }} as | ||
{{ temp_view_sql }} | ||
'); | ||
SELECT * INTO {{ relation }} FROM | ||
{{ tmp_relation }} | ||
|
||
SELECT * INTO {{ relation.schema }}.{{ relation.identifier }} FROM | ||
{{ tmp_relation.schema }}.{{ tmp_relation.identifier }} | ||
{% do adapter.drop_relation(tmp_relation) %} | ||
|
||
{% if not temporary and as_columnstore -%} | ||
{{ sqlserver__create_clustered_columnstore_index(relation) }} | ||
{% endif %} | ||
|
||
{{ sqlserver__drop_relation_script(tmp_relation) }} | ||
|
||
{% if not temporary and as_columnstore -%} | ||
{{ sqlserver__create_clustered_columnstore_index(relation) }} | ||
{% endif %} | ||
|
||
{% endmacro %}_ | ||
{% endmacro %} | ||
|
||
{% macro sqlserver__insert_into_from(to_relation, from_relation) -%} | ||
{%- set full_to_relation = to_relation.schema ~ '.' ~ to_relation.identifier -%} | ||
{%- set full_from_relation = from_relation.schema ~ '.' ~ from_relation.identifier -%} | ||
|
||
SELECT * INTO {{full_to_relation}} FROM {{full_from_relation}} | ||
SELECT * INTO {{to_relation}} FROM {{from_relation}} | ||
|
||
{% endmacro %} | ||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mikaelene this is the one thing I'm not sure about. can you forsee any issues using
"
instead of'
to quote relation parts?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am not a guru in quoting :-). Did we have
'
before?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah we did, inexplicably though, the code change when I change it back to single quote. You're right that we should think about this change maybe breaking existing snapshot or incremental tables...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I will definitely have to change this back to include
database=True
by default to make cross-database selects work.