added reconcile sql server code #868
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Here we modify some existing tests to improve coverage of TSQL and fix some bugs in code generation or add additional tests.
…atabrickslabs#785) This should speed up test targets and inner devloop
…abs#786) PR databrickslabs#548 unfortunately incorrectly removed support for parenthesized expressions. This was not caught because we had no coverage tests for expression precedence so an expression such as: ```sql (a + b) * c ``` Generates a syntax error. The PR also: - removed `bracketExpression` from the `expr` rule, but left it in the grammar despite being unused (removing it would have pointed out its real use in the IR generator) - Incorrectly duplicated `COLON id` Here we reinstate expression precedence correctly, add test coverage for the grammar changes that were made, and correct the duplication of `COLON id`
…w frame explicit (databrickslabs#792) Co-authored-by: Serge Smertin <[email protected]>
…D OPTION (databrickslabs#788) The TSQL grammar for `CREATE DATABASE` and `CREATE DATABASE SCOPED OPTION` was not consistent with the TSQL documentation. Here we correct the grammar errors. While it was originally my intention to implement these all the way through to code gen, the DDL needs some serious work, and I would rather implement the grammar for it all in a series of grammar only PRs, then come back through and implement all the way to codegen in a subsequent set of PRs. The reason being that as each DDL statement is fixed at the grammar level, all the other DDL statements will gradually simplify at the same time and I do not wish to keep changing grammar then have to tweak previously defined code gen.
…ickslabs#796) Here we correct the existing grammar for CREATE TABLE in all its forms such as graph nodes, CTAS and FILETABLE.
…atabrickslabs#793) Adds additional instructions for making ANTLR grammar changes, including explicit rules and checks to make.
Here we implement a linting tool that is customized towards the antlr4
grammars in this repo.
As a first step, we identify any rules defined in a parser grammar that
are not referenced anywhere else in the grammar, as well as referenced
rules that are undefined. Note that we do not process include file in
this PR and so this could result in false negatives if rules in one
grammar are referenced in an included grammar. At this point we have no
included grammars in our main grammars and hence this is as yet, not an
issue.
Note that this PR adds a new github action which eventually will fail
the build if it find issues in the grammars. As the new linter finds
errors in both our parser grammars, we will need to fix those errors
before allowing the lint to fail the build.
The linter produces output akin to:
```
Issues found in grammar: /Users/jim.idle/databricks/remorph/core/src/main/antlr4/com/databricks/labs/remorph/parsers/tsql/TSqlParser.g4
Orphaned rules (rules defined but never used):
nouse defined at line 1675:
nouse
: DEFAULT_LANGUAGE EQ (id | STRING)
insertColumnNameList defined at line 3321:
insertColumnNameList: insertColumnId (COMMA insertColumnId)*
;
entityNameForParallelDw defined at line 3290:
entityNameForParallelDw: schemaDatabase = id | schema = id DOT objectName = id
;
deprecatedTableHint defined at line 3211:
deprecatedTableHint: LPAREN tableHint RPAREN
;
createDatabaseScopedCredential defined at line 1665:
createDatabaseScopedCredential
: CREATE DATABASE SCOPED CREDENTIAL id WITH IDENTITY EQ STRING (SECRET EQ STRING)?
udtMethodArguments defined at line 3014:
udtMethodArguments
: LPAREN argument += executeVarString (COMMA argument += executeVarString)* RPAREN
entityNameForAzureDw defined at line 3287:
entityNameForAzureDw: schema = id | schema = id DOT objectName = id
;
indexValue defined at line 3223:
indexValue: id | INT
;
sybaseLegacyHint defined at line 3214:
sybaseLegacyHint: HOLDLOCK | NOHOLDLOCK | READPAST | SHARED
;
Issues found in grammar: /Users/jim.idle/databricks/remorph/core/src/main/antlr4/com/databricks/labs/remorph/parsers/snowflake/SnowflakeParser.g4
Orphaned rules (rules defined but never used):
valuesBuilder defined at line 118:
valuesBuilder: VALUES L_PAREN exprList R_PAREN (COMMA L_PAREN exprList R_PAREN)?
;
jsonPath defined at line 3145:
jsonPath: jsonPathElem (DOT jsonPathElem)*
;
bracketExpression defined at line 3276:
bracketExpression: L_PAREN exprList R_PAREN | L_PAREN subquery R_PAREN
;
make: *** [antlr-lint] Error 1
```
* Added widgets Aggregates Reconcile metrics
closes databrickslabs#748 * Added create and insert statement(100) TPC-DS dataset
…s#802) The new ANTLR linter points out a number of unused rules in the TSQL grammar. Here we resolve the linting errors, generally by deleting unused rules.
…slabs#804) The new ANTLR linter points out a number of unused rules in the Snowflake grammar. Here we resolve the linting errors, generally by deleting unused rules.
Here, we greatly improve the grammar constructs for create and alter and additionally, make parsing of `id . id` orthogonal instead of having 17 rules to parse the same construct. We also remove the mountain of superfluous labels given to many tokens and rules, which just add bloat to the generated code. As a result, the error recovery mechanism is more accurate in identifying the source of syntax errors as there is much less ambiguity. The size of the generated parser file is also 'smaller' at 99387 lines vs 107354 - we intend to continue reducing this.
TODO: - [x] NOTICE file - [x] Snowflake runner - [x] infra permissions Follow-up PR: - [ ] File framework - [ ] CSV-to-Databricks comparison - [ ] unify snowflake logger. See [docs](https://docs.snowflake.com/en/developer-guide/jdbc/jdbc-configure#simple-logging-facade-for-java-org-slf4j) <img width="485" alt="image" src="https://github.com/user-attachments/assets/3c3b8287-51f5-4eac-9f37-fe172c36969e">
Previously, `NATURAL` and `CROSS` joins were not properly handled, this change fixes it.
…ssion generators (databrickslabs#809) GeneratorContext is something that is explicitly passed down through entire chain, so let's use it to inject logical plan generator when we need it to generate subquery expressions.
…ckslabs#810) This PR increases coverage of snowflake data types.
This PR adds simplistic SQL formatter into sql transpilation chain, so that it's easier to compare results
After a fairly big reorg on ddl statements, slight changes to the grammar rules caused a regression in the number of TSQL statements that we can parse and therefore transpile. This PR corrects those oversights. Also improves coverages test results.
Having 300+ folders in a single folder is unmaintainable. This PR moves files around in a bit more nested structure, as well as reformatting complex cases.
This Patches CI failures in main branch
Co-authored-by: sundar.shankar <[email protected]> Co-authored-by: SundarShankar89 <[email protected]>
The 0.14.0 release depends on Databricks SDK 0.37.0, but doesn't declare this. As such it can't be installed due to our dependency on Databricks SDK 0.29.0. Reference: databrickslabs/lsql#330
This PR updates the TSQL parsing so that the following form of UNION is supported: ```sql (SELECT a from b) UNION [ALL] SELECT x from y ``` Resolves databrickslabs#1127.
In this PR we are handling both STRPOS and ARRAY_AVERAGE functions.
`STRPOS:`
--presto SQL:
SELECT strpos('Hello world', 'l', 2);
--databricks sql:
SELECT LOCATE('l', 'Hello world', 2);
`ARRAY_AVERAGE:`
Databricks doesn't have any built-in function to calculate the average
of an array, so we are handling it using nested functions available in
Databricks. It handles nulls, Integer and Double.
--presto SQL:
rray_average(arr)
--databricks SQL:
AGGREGATE(
FILTER(arr, x -> x IS NOT NULL),
NAMED_STRUCT('sum', CAST(0 AS DOUBLE), 'cnt', 0),
(acc, x) -> NAMED_STRUCT('sum', acc.sum + x, 'cnt', acc.cnt + 1),
acc -> TRY_DIVIDE(acc.sum, acc.cnt)
)
…kslabs#1228) Support multiple columns in order by clause in for ARRAYAGG. Resolves databrickslabs#1176
…abrickslabs#1218) Improve the way `TO_TIME` and `TO_TIMESTAMP` are translated. - When called with an explicit format, said format is properly translated (for instance, SF's format `YYYY-MM-DD HH24:MITZH:TZM` gets translated into DB's `yyyy-MM-dd'T'HH:mmZZZ`), be it a constant or a variable. - When SF's TO_TIMESTAMP is called with the `TO_TIMESTAMP ( <numeric_expr> , <scale> )` we simply truncate `numeric_expr` to a number-of-seconds-since-epoch, as it seems to be the best we can do. - When TO_TIMESTAMP is called with no explicit format, we do our best as to handle the automatic formats that SF handles but DB doesn't by trying each explicitly until one matches - the `DY` format part (corresponding to `EEE` in DB), ie. day-of-week, isn't allowed anymore since Spark 3.0. Since it isn't relevant in figuring out a specific point in time, we simply ignore it in both translated format and input string (resulting in huge amount of SQL being produced, see `tests/resources/functional/snowflake/core_engine/functions/conversion/to_timestamp_variable_format.sql` for an example)
closes databrickslabs#1025 Add code for Alter Table Add multiple Columns. Also added a fix for the default constraint previously it is translated to `GENERATED ALWAYS AS` --------- Co-authored-by: SundarShankar89 <[email protected]>
* Added support for json_size function in presto and alternative in Databricks using SQL functions * Fixed `is not null` error for json_extract in Databricks generator. Added a new test case for this issue.
Doesn't actually fix anything, rather adds tests that show that a fix might not be required Progresses databrickslabs#976 Supersedes databrickslabs#1223 which was lacking gpg signature
…bs#1248) This PR trivially refactors the situations where we are using cases classes without parameters to be case objects (as is idiomatic in Scala).
This PR aims to fix the memory leak causing failure by raising parse errors immediately.
Generator does not currently enclose subqueries in parenthesis, thus generating incorrect code such as the following: `SELECT * FROM SELECT * FROM t WHERE a > 'a' WHERE a > 'b' ` This PR fixes the issue such that the generated code is now: `SELECT * FROM (SELECT * FROM t WHERE a > 'a') WHERE a > 'b' ` It takes special care of the `.. IN(SELECT...)` pattern, by avoiding doubling enlosing parenthesis Test cases are included databrickslabs#1233 but not added here because they require changes unrelated to this PR
…kslabs#1249) This PR refactors the situations where we use this: ```scala x.tail.foldLeft(x.head) ``` to use the equivalent operation that does this: ```scala x.reduceLeft ``` Following a review comment discussing the safety of reduce in these situations[^1], some further changes have been made to the surrounding code such that the behaviour remains the same but the safety can be determined via trivial inspection without needing to refer to the grammar. [^1]: Not a new concern; the previous code had the same safety issue due to the use of `.head`.
## Changes This PR implements support for parsing set operations with TSql: `UNION [ALL]`, `EXCEPT` and `INTERSECT`. The grammar previously supported these but they were not being converted to the IR. ### Linked issues Resolves databrickslabs#1126. Resolves databrickslabs#1102. ### Tests - [x] added unit tests - [x] added transpiler tests - [x] added functional tests
…atabrickslabs#1250) Added support for format_datetime function in presto to Databricks <img width="838" alt="Screenshot 2024-11-27 at 5 09 44 PM" src="https://github.com/user-attachments/assets/5eef2df5-757c-4107-a2a7-28b3f57f970d"> <img width="972" alt="Screenshot 2024-11-27 at 5 09 38 PM" src="https://github.com/user-attachments/assets/70499579-09a8-4f34-b6fb-062b5bbb4424">
This PR drops an unnecessary unit test.
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 4 to 5. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/codecov/codecov-action/releases">codecov/codecov-action's releases</a>.</em></p> <blockquote> <h2>v5.0.0</h2> <h2>v5 Release</h2> <p><code>v5</code> of the Codecov GitHub Action will use the <a href="https://github.com/codecov/wrapper">Codecov Wrapper</a> to encapsulate the <a href="https://github.com/codecov/codecov-cli">CLI</a>. This will help ensure that the Action gets updates quicker.</p> <h3>Migration Guide</h3> <p>The <code>v5</code> release also coincides with the opt-out feature for tokens for public repositories. In the <code>Global Upload Token</code> section of the settings page of an organization in codecov.io, you can set the ability for Codecov to receive a coverage reports from any source. This will allow contributors or other members of a repository to upload without needing access to the Codecov token. For more details see <a href="https://docs.codecov.com/docs/codecov-tokens#uploading-without-a-token">how to upload without a token</a>.</p> <blockquote> <p>[!WARNING]<br /> <strong>The following arguments have been changed</strong></p> <ul> <li><code>file</code> (this has been deprecated in favor of <code>files</code>)</li> <li><code>plugin</code> (this has been deprecated in favor of <code>plugins</code>)</li> </ul> </blockquote> <p>The following arguments have been added:</p> <ul> <li><code>binary</code></li> <li><code>gcov_args</code></li> <li><code>gcov_executable</code></li> <li><code>gcov_ignore</code></li> <li><code>gcov_include</code></li> <li><code>report_type</code></li> <li><code>skip_validation</code></li> <li><code>swift_project</code></li> </ul> <p>You can see their usage in the <code>action.yml</code> <a href="https://github.com/codecov/codecov-action/blob/main/action.yml">file</a>.</p> <h2>What's Changed</h2> <ul> <li>chore(deps): bump to eslint9+ and remove eslint-config-google by <a href="https://github.com/thomasrockhu-codecov"><code>@thomasrockhu-codecov</code></a> in <a href="https://redirect.github.com/codecov/codecov-action/pull/1591">codecov/codecov-action#1591</a></li> <li>build(deps-dev): bump <code>@octokit/webhooks-types</code> from 7.5.1 to 7.6.1 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/codecov/codecov-action/pull/1595">codecov/codecov-action#1595</a></li> <li>build(deps-dev): bump typescript from 5.6.2 to 5.6.3 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/codecov/codecov-action/pull/1604">codecov/codecov-action#1604</a></li> <li>build(deps-dev): bump <code>@typescript-eslint/parser</code> from 8.8.0 to 8.8.1 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/codecov/codecov-action/pull/1601">codecov/codecov-action#1601</a></li> <li>build(deps): bump <code>@actions/core</code> from 1.11.0 to 1.11.1 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/codecov/codecov-action/pull/1597">codecov/codecov-action#1597</a></li> <li>build(deps): bump github/codeql-action from 3.26.9 to 3.26.11 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/codecov/codecov-action/pull/1596">codecov/codecov-action#1596</a></li> <li>build(deps-dev): bump <code>@typescript-eslint/eslint-plugin</code> from 8.8.0 to 8.8.1 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/codecov/codecov-action/pull/1600">codecov/codecov-action#1600</a></li> <li>build(deps-dev): bump eslint from 9.11.1 to 9.12.0 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/codecov/codecov-action/pull/1598">codecov/codecov-action#1598</a></li> <li>build(deps): bump github/codeql-action from 3.26.11 to 3.26.12 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/codecov/codecov-action/pull/1609">codecov/codecov-action#1609</a></li> <li>build(deps): bump actions/checkout from 4.2.0 to 4.2.1 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/codecov/codecov-action/pull/1608">codecov/codecov-action#1608</a></li> <li>build(deps): bump actions/upload-artifact from 4.4.0 to 4.4.3 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/codecov/codecov-action/pull/1607">codecov/codecov-action#1607</a></li> <li>build(deps-dev): bump <code>@typescript-eslint/parser</code> from 8.8.1 to 8.9.0 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/codecov/codecov-action/pull/1612">codecov/codecov-action#1612</a></li> <li>build(deps-dev): bump <code>@typescript-eslint/eslint-plugin</code> from 8.8.1 to 8.9.0 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/codecov/codecov-action/pull/1611">codecov/codecov-action#1611</a></li> <li>build(deps-dev): bump <code>@typescript-eslint/eslint-plugin</code> from 8.9.0 to 8.10.0 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/codecov/codecov-action/pull/1615">codecov/codecov-action#1615</a></li> <li>build(deps-dev): bump eslint from 9.12.0 to 9.13.0 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/codecov/codecov-action/pull/1618">codecov/codecov-action#1618</a></li> <li>build(deps): bump github/codeql-action from 3.26.12 to 3.26.13 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/codecov/codecov-action/pull/1617">codecov/codecov-action#1617</a></li> <li>build(deps-dev): bump <code>@typescript-eslint/parser</code> from 8.9.0 to 8.10.0 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/codecov/codecov-action/pull/1614">codecov/codecov-action#1614</a></li> <li>build(deps-dev): bump <code>@typescript-eslint/eslint-plugin</code> from 8.10.0 to 8.11.0 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/codecov/codecov-action/pull/1620">codecov/codecov-action#1620</a></li> <li>build(deps-dev): bump <code>@typescript-eslint/parser</code> from 8.10.0 to 8.11.0 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/codecov/codecov-action/pull/1619">codecov/codecov-action#1619</a></li> <li>build(deps-dev): bump <code>@types/jest</code> from 29.5.13 to 29.5.14 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/codecov/codecov-action/pull/1622">codecov/codecov-action#1622</a></li> <li>build(deps): bump actions/checkout from 4.2.1 to 4.2.2 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/codecov/codecov-action/pull/1625">codecov/codecov-action#1625</a></li> <li>build(deps): bump github/codeql-action from 3.26.13 to 3.27.0 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/codecov/codecov-action/pull/1624">codecov/codecov-action#1624</a></li> <li>build(deps-dev): bump <code>@typescript-eslint/eslint-plugin</code> from 8.11.0 to 8.12.1 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/codecov/codecov-action/pull/1626">codecov/codecov-action#1626</a></li> <li>build(deps-dev): bump <code>@typescript-eslint/eslint-plugin</code> from 8.12.1 to 8.12.2 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/codecov/codecov-action/pull/1629">codecov/codecov-action#1629</a></li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md">codecov/codecov-action's changelog</a>.</em></p> <blockquote> <h2>4.0.0-beta.2</h2> <h3>Fixes</h3> <ul> <li><a href="https://redirect.github.com/codecov/codecov-action/issues/1085">#1085</a> not adding -n if empty to do-upload command</li> </ul> <h2>4.0.0-beta.1</h2> <p><code>v4</code> represents a move from the <a href="https://github.com/codecov/uploader">universal uploader</a> to the <a href="https://github.com/codecov/codecov-cli">Codecov CLI</a>. Although this will unlock new features for our users, the CLI is not yet at feature parity with the universal uploader.</p> <h3>Breaking Changes</h3> <ul> <li>No current support for <code>aarch64</code> and <code>alpine</code> architectures.</li> <li>Tokenless uploading is unsuported</li> <li>Various arguments to the Action have been removed</li> </ul> <h2>3.1.4</h2> <h3>Fixes</h3> <ul> <li><a href="https://redirect.github.com/codecov/codecov-action/issues/967">#967</a> Fix typo in README.md</li> <li><a href="https://redirect.github.com/codecov/codecov-action/issues/971">#971</a> fix: add back in working dir</li> <li><a href="https://redirect.github.com/codecov/codecov-action/issues/969">#969</a> fix: CLI option names for uploader</li> </ul> <h3>Dependencies</h3> <ul> <li><a href="https://redirect.github.com/codecov/codecov-action/issues/970">#970</a> build(deps-dev): bump <code>@types/node</code> from 18.15.12 to 18.16.3</li> <li><a href="https://redirect.github.com/codecov/codecov-action/issues/979">#979</a> build(deps-dev): bump <code>@types/node</code> from 20.1.0 to 20.1.2</li> <li><a href="https://redirect.github.com/codecov/codecov-action/issues/981">#981</a> build(deps-dev): bump <code>@types/node</code> from 20.1.2 to 20.1.4</li> </ul> <h2>3.1.3</h2> <h3>Fixes</h3> <ul> <li><a href="https://redirect.github.com/codecov/codecov-action/issues/960">#960</a> fix: allow for aarch64 build</li> </ul> <h3>Dependencies</h3> <ul> <li><a href="https://redirect.github.com/codecov/codecov-action/issues/957">#957</a> build(deps-dev): bump jest-junit from 15.0.0 to 16.0.0</li> <li><a href="https://redirect.github.com/codecov/codecov-action/issues/958">#958</a> build(deps): bump openpgp from 5.7.0 to 5.8.0</li> <li><a href="https://redirect.github.com/codecov/codecov-action/issues/959">#959</a> build(deps-dev): bump <code>@types/node</code> from 18.15.10 to 18.15.12</li> </ul> <h2>3.1.2</h2> <h3>Fixes</h3> <ul> <li><a href="https://redirect.github.com/codecov/codecov-action/issues/718">#718</a> Update README.md</li> <li><a href="https://redirect.github.com/codecov/codecov-action/issues/851">#851</a> Remove unsupported path_to_write_report argument</li> <li><a href="https://redirect.github.com/codecov/codecov-action/issues/898">#898</a> codeql-analysis.yml</li> <li><a href="https://redirect.github.com/codecov/codecov-action/issues/901">#901</a> Update README to contain correct information - inputs and negate feature</li> <li><a href="https://redirect.github.com/codecov/codecov-action/issues/955">#955</a> fix: add in all the extra arguments for uploader</li> </ul> <h3>Dependencies</h3> <ul> <li><a href="https://redirect.github.com/codecov/codecov-action/issues/819">#819</a> build(deps): bump openpgp from 5.4.0 to 5.5.0</li> <li><a href="https://redirect.github.com/codecov/codecov-action/issues/835">#835</a> build(deps): bump node-fetch from 3.2.4 to 3.2.10</li> <li><a href="https://redirect.github.com/codecov/codecov-action/issues/840">#840</a> build(deps): bump ossf/scorecard-action from 1.1.1 to 2.0.4</li> <li><a href="https://redirect.github.com/codecov/codecov-action/issues/841">#841</a> build(deps): bump <code>@actions/core</code> from 1.9.1 to 1.10.0</li> <li><a href="https://redirect.github.com/codecov/codecov-action/issues/843">#843</a> build(deps): bump <code>@actions/github</code> from 5.0.3 to 5.1.1</li> <li><a href="https://redirect.github.com/codecov/codecov-action/issues/869">#869</a> build(deps): bump node-fetch from 3.2.10 to 3.3.0</li> <li><a href="https://redirect.github.com/codecov/codecov-action/issues/872">#872</a> build(deps-dev): bump jest-junit from 13.2.0 to 15.0.0</li> <li><a href="https://redirect.github.com/codecov/codecov-action/issues/879">#879</a> build(deps): bump decode-uri-component from 0.2.0 to 0.2.2</li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/codecov/codecov-action/commit/968872560f81e7bdde9272853e65f2507c0eca7c"><code>9688725</code></a> Update README.md</li> <li><a href="https://github.com/codecov/codecov-action/commit/2112eaec1bedbdabc7e93d5312449d0d62b07c60"><code>2112eae</code></a> chore(deps): bump wrapper to 0.0.23 (<a href="https://redirect.github.com/codecov/codecov-action/issues/1644">#1644</a>)</li> <li><a href="https://github.com/codecov/codecov-action/commit/193421c5b3d1aca4209c9754f224ca0d85729414"><code>193421c</code></a> fixL use the correct source (<a href="https://redirect.github.com/codecov/codecov-action/issues/1642">#1642</a>)</li> <li><a href="https://github.com/codecov/codecov-action/commit/6018df70b05b191502ce08196e76e30ea3578615"><code>6018df7</code></a> fix: update container builds (<a href="https://redirect.github.com/codecov/codecov-action/issues/1640">#1640</a>)</li> <li><a href="https://github.com/codecov/codecov-action/commit/eff1a643d6887ee5935d4ca343e9076dc377d416"><code>eff1a64</code></a> fix: add missing vars (<a href="https://redirect.github.com/codecov/codecov-action/issues/1638">#1638</a>)</li> <li><a href="https://github.com/codecov/codecov-action/commit/4582d54fd3d27d9130327cdb51361c32016fa400"><code>4582d54</code></a> Update README.md (<a href="https://redirect.github.com/codecov/codecov-action/issues/1639">#1639</a>)</li> <li><a href="https://github.com/codecov/codecov-action/commit/bb7467c2bce05781760a0964d48e35e96ee59505"><code>bb7467c</code></a> feat: use wrapper (<a href="https://redirect.github.com/codecov/codecov-action/issues/1621">#1621</a>)</li> <li><a href="https://github.com/codecov/codecov-action/commit/1d6059880cab9176d33e31e0f1ab076b20495f5e"><code>1d60598</code></a> build(deps-dev): bump <code>@typescript-eslint/eslint-plugin</code> from 8.12.2 to 8.13.0 ...</li> <li><a href="https://github.com/codecov/codecov-action/commit/e587ce276eb45f1fcd960de3c01c83119213efca"><code>e587ce2</code></a> build(deps-dev): bump <code>@typescript-eslint/parser</code> from 8.12.2 to 8.13.0 (<a href="https://redirect.github.com/codecov/codecov-action/issues/1635">#1635</a>)</li> <li><a href="https://github.com/codecov/codecov-action/commit/e43f28e103e52bb26d252b5a97fcdfa06175321e"><code>e43f28e</code></a> build(deps-dev): bump <code>@typescript-eslint/parser</code> from 8.11.0 to 8.12.2 (<a href="https://redirect.github.com/codecov/codecov-action/issues/1628">#1628</a>)</li> <li>Additional commits viewable in <a href="https://github.com/codecov/codecov-action/compare/v4...v5">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: SundarShankar89 <[email protected]> Co-authored-by: Andrew Snare <[email protected]>
SnowFlake supports both SUBSTR and SUBSTRING, see https://docs.snowflake.com/fr/sql-reference/functions/substr. This PR fills the gap for missing SUBSTR Supersedes databrickslabs#1226 which was lacking gpg signature --------- Co-authored-by: Andrew Snare <[email protected]>
Bumps [sqlglot](https://github.com/tobymao/sqlglot) from 25.30.0 to 25.32.1. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/tobymao/sqlglot/blob/main/CHANGELOG.md">sqlglot's changelog</a>.</em></p> <blockquote> <h2>[v25.32.1] - 2024-11-27</h2> <h3>:bug: Bug Fixes</h3> <ul> <li><a href="https://github.com/tobymao/sqlglot/commit/954d8fd12740071e0951d1df3a405a4b9634868d"><code>954d8fd</code></a> - parse DEFAULT in VALUES clause into a Var <em>(PR <a href="https://redirect.github.com/tobymao/sqlglot/pull/4448">#4448</a> by <a href="https://github.com/georgesittas"><code>@georgesittas</code></a>)</em> <ul> <li>:arrow_lower_right: <em>fixes issue <a href="https://redirect.github.com/tobymao/sqlglot/issues/4446">#4446</a> opened by <a href="https://github.com/ddh-5230"><code>@ddh-5230</code></a></em></li> </ul> </li> <li><a href="https://github.com/tobymao/sqlglot/commit/73afd0f435b7e7ccde831ee311c9a76c14797fdc"><code>73afd0f</code></a> - <strong>bigquery</strong>: Make JSONPathTokenizer more lenient for new standards <em>(PR <a href="https://redirect.github.com/tobymao/sqlglot/pull/4447">#4447</a> by <a href="https://github.com/VaggelisD"><code>@VaggelisD</code></a>)</em> <ul> <li>:arrow_lower_right: <em>fixes issue <a href="https://redirect.github.com/tobymao/sqlglot/issues/4441">#4441</a> opened by <a href="https://github.com/patricksurry"><code>@patricksurry</code></a></em></li> </ul> </li> </ul> <h2>[v25.32.0] - 2024-11-22</h2> <h3>:boom: BREAKING CHANGES</h3> <ul> <li> <p>due to <a href="https://github.com/tobymao/sqlglot/commit/0eed45cce82681bfbafc8bfb78eb2a1bce86ae53"><code>0eed45c</code></a> - Add support for ATTACH/DETACH statements <em>(PR <a href="https://redirect.github.com/tobymao/sqlglot/pull/4419">#4419</a> by <a href="https://github.com/VaggelisD"><code>@VaggelisD</code></a>)</em>:</p> <p>Add support for ATTACH/DETACH statements (<a href="https://redirect.github.com/tobymao/sqlglot/issues/4419">#4419</a>)</p> </li> <li> <p>due to <a href="https://github.com/tobymao/sqlglot/commit/da48b68a4f1fa6a754fa2a0a789564675d59546f"><code>da48b68</code></a> - Tokenize hints as comments <em>(PR <a href="https://redirect.github.com/tobymao/sqlglot/pull/4426">#4426</a> by <a href="https://github.com/VaggelisD"><code>@VaggelisD</code></a>)</em>:</p> <p>Tokenize hints as comments (<a href="https://redirect.github.com/tobymao/sqlglot/issues/4426">#4426</a>)</p> </li> <li> <p>due to <a href="https://github.com/tobymao/sqlglot/commit/fe3539464a153b1c0bf46975d6221dee48a48f02"><code>fe35394</code></a> - fix datetime coercion in the canonicalize rule <em>(PR <a href="https://redirect.github.com/tobymao/sqlglot/pull/4431">#4431</a> by <a href="https://github.com/georgesittas"><code>@georgesittas</code></a>)</em>:</p> <p>fix datetime coercion in the canonicalize rule (<a href="https://redirect.github.com/tobymao/sqlglot/issues/4431">#4431</a>)</p> </li> <li> <p>due to <a href="https://github.com/tobymao/sqlglot/commit/fddcd3dfc264a645909686c201d2288c0adf9047"><code>fddcd3d</code></a> - bump sqlglotrs to 0.3.0 <em>(commit by <a href="https://github.com/georgesittas"><code>@georgesittas</code></a>)</em>:</p> <p>bump sqlglotrs to 0.3.0</p> </li> </ul> <h3>:sparkles: New Features</h3> <ul> <li><a href="https://github.com/tobymao/sqlglot/commit/0eed45cce82681bfbafc8bfb78eb2a1bce86ae53"><code>0eed45c</code></a> - <strong>duckdb</strong>: Add support for ATTACH/DETACH statements <em>(PR <a href="https://redirect.github.com/tobymao/sqlglot/pull/4419">#4419</a> by <a href="https://github.com/VaggelisD"><code>@VaggelisD</code></a>)</em></li> <li><a href="https://github.com/tobymao/sqlglot/commit/2db757dfec9ded26572b8e9a71dcc8ea8a2382fe"><code>2db757d</code></a> - <strong>bigquery</strong>: Support FEATURES_AT_TIME <em>(PR <a href="https://redirect.github.com/tobymao/sqlglot/pull/4430">#4430</a> by <a href="https://github.com/VaggelisD"><code>@VaggelisD</code></a>)</em> <ul> <li>:arrow_lower_right: <em>addresses issue <a href="https://redirect.github.com/tobymao/sqlglot/issues/4428">#4428</a> opened by <a href="https://github.com/YuvrajSoni-Ksolves"><code>@YuvrajSoni-Ksolves</code></a></em></li> </ul> </li> <li><a href="https://github.com/tobymao/sqlglot/commit/fc591ae2fa80be5821cb53d78906afe8e5505654"><code>fc591ae</code></a> - <strong>risingwave</strong>: add support for SINK, SOURCE & other DDL properties <em>(PR <a href="https://redirect.github.com/tobymao/sqlglot/pull/4387">#4387</a> by <a href="https://github.com/lin0303-siyuan"><code>@lin0303-siyuan</code></a>)</em></li> <li><a href="https://github.com/tobymao/sqlglot/commit/a2bde2e03e9ef8650756bf304db35b4876746d1f"><code>a2bde2e</code></a> - <strong>mysql</strong>: improve transpilability of CHAR[ACTER]_LENGTH <em>(commit by <a href="https://github.com/georgesittas"><code>@georgesittas</code></a>)</em></li> <li><a href="https://github.com/tobymao/sqlglot/commit/0acc248361f49f68f17d799cbaf6b3de06c57f7e"><code>0acc248</code></a> - <strong>snowflake</strong>: Support CREATE ... WITH TAG <em>(PR <a href="https://redirect.github.com/tobymao/sqlglot/pull/4434">#4434</a> by <a href="https://github.com/asikowitz"><code>@asikowitz</code></a>)</em> <ul> <li>:arrow_lower_right: <em>addresses issue <a href="https://redirect.github.com/tobymao/sqlglot/issues/4427">#4427</a> opened by <a href="https://github.com/asikowitz"><code>@asikowitz</code></a></em></li> </ul> </li> <li><a href="https://github.com/tobymao/sqlglot/commit/37863ffd747cad9c2b9bed60119cc1551faeffda"><code>37863ff</code></a> - <strong>snowflake</strong>: Transpile non-UNNEST exp.GenerateDateArray refs <em>(PR <a href="https://redirect.github.com/tobymao/sqlglot/pull/4433">#4433</a> by <a href="https://github.com/VaggelisD"><code>@VaggelisD</code></a>)</em></li> </ul> <h3>:bug: Bug Fixes</h3> <ul> <li><a href="https://github.com/tobymao/sqlglot/commit/83ee97b34cd0fe269b4820f15147d1ed7523612e"><code>83ee97b</code></a> - <strong>parser</strong>: Do not parse window function arg as exp.Column <em>(PR <a href="https://redirect.github.com/tobymao/sqlglot/pull/4415">#4415</a> by <a href="https://github.com/VaggelisD"><code>@VaggelisD</code></a>)</em> <ul> <li>:arrow_lower_right: <em>fixes issue <a href="https://redirect.github.com/tobymao/sqlglot/issues/4410">#4410</a> opened by <a href="https://github.com/merlindso"><code>@merlindso</code></a></em></li> </ul> </li> <li><a href="https://github.com/tobymao/sqlglot/commit/b22e0c8680b0ee5a382e57904b698bf21a94f782"><code>b22e0c8</code></a> - <strong>parser</strong>: Extend DESCRIBE parser for MySQL FORMAT & statements <em>(PR <a href="https://redirect.github.com/tobymao/sqlglot/pull/4417">#4417</a> by <a href="https://github.com/VaggelisD"><code>@VaggelisD</code></a>)</em> <ul> <li>:arrow_lower_right: <em>fixes issue <a href="https://redirect.github.com/tobymao/sqlglot/issues/4414">#4414</a> opened by <a href="https://github.com/AhlamHani"><code>@AhlamHani</code></a></em></li> </ul> </li> <li><a href="https://github.com/tobymao/sqlglot/commit/d1d2ae7d1514abc9477d275352e5e126509157c6"><code>d1d2ae7</code></a> - <strong>duckdb</strong>: Allow count arg on exp.ArgMax & exp.ArgMin <em>(PR <a href="https://redirect.github.com/tobymao/sqlglot/pull/4413">#4413</a> by <a href="https://github.com/aersam"><code>@aersam</code></a>)</em> <ul> <li>:arrow_lower_right: <em>fixes issue <a href="https://redirect.github.com/tobymao/sqlglot/issues/4412">#4412</a> opened by <a href="https://github.com/aersam"><code>@aersam</code></a></em></li> </ul> </li> <li><a href="https://github.com/tobymao/sqlglot/commit/e3c45d5ec0ae6827e4b0bcfb047aeac131379732"><code>e3c45d5</code></a> - presto reset session closes <a href="https://redirect.github.com/tobymao/sqlglot/pull/4421">#4421</a> <em>(commit by <a href="https://github.com/tobymao"><code>@tobymao</code></a>)</em></li> <li><a href="https://github.com/tobymao/sqlglot/commit/fd81f1bfee9a566b8df8bb501828c20bd72ac481"><code>fd81f1b</code></a> - more presto commands <em>(commit by <a href="https://github.com/tobymao"><code>@tobymao</code></a>)</em></li> <li><a href="https://github.com/tobymao/sqlglot/commit/da48b68a4f1fa6a754fa2a0a789564675d59546f"><code>da48b68</code></a> - Tokenize hints as comments <em>(PR <a href="https://redirect.github.com/tobymao/sqlglot/pull/4426">#4426</a> by <a href="https://github.com/VaggelisD"><code>@VaggelisD</code></a>)</em> <ul> <li>:arrow_lower_right: <em>fixes issue <a href="https://redirect.github.com/tobymao/sqlglot/issues/4425">#4425</a> opened by <a href="https://github.com/mkmoisen"><code>@mkmoisen</code></a></em></li> </ul> </li> <li><a href="https://github.com/tobymao/sqlglot/commit/69d4a8ccdf5954f293acbdf61c420b72dde5b8af"><code>69d4a8c</code></a> - <strong>tsql</strong>: Map weekday to %w <em>(PR <a href="https://redirect.github.com/tobymao/sqlglot/pull/4438">#4438</a> by <a href="https://github.com/VaggelisD"><code>@VaggelisD</code></a>)</em> <ul> <li>:arrow_lower_right: <em>fixes issue <a href="https://redirect.github.com/tobymao/sqlglot/issues/4435">#4435</a> opened by <a href="https://github.com/travispaice"><code>@travispaice</code></a></em></li> </ul> </li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/tobymao/sqlglot/commit/73afd0f435b7e7ccde831ee311c9a76c14797fdc"><code>73afd0f</code></a> fix(bigquery): Make JSONPathTokenizer more lenient for new standards (<a href="https://redirect.github.com/tobymao/sqlglot/issues/4447">#4447</a>)</li> <li><a href="https://github.com/tobymao/sqlglot/commit/954d8fd12740071e0951d1df3a405a4b9634868d"><code>954d8fd</code></a> Fix: parse DEFAULT in VALUES clause into a Var (<a href="https://redirect.github.com/tobymao/sqlglot/issues/4448">#4448</a>)</li> <li><a href="https://github.com/tobymao/sqlglot/commit/74dc39ba8649fd8292c97c82088b39b08f531702"><code>74dc39b</code></a> docs: update API docs, CHANGELOG.md for v25.32.0 [skip ci]</li> <li><a href="https://github.com/tobymao/sqlglot/commit/fddcd3dfc264a645909686c201d2288c0adf9047"><code>fddcd3d</code></a> Chore!: bump sqlglotrs to 0.3.0</li> <li><a href="https://github.com/tobymao/sqlglot/commit/6aea9f346ef8f91467e1d5da5a3f94cf862b44fe"><code>6aea9f3</code></a> fix: Refactor NORMALIZE_FUNCTIONS flag usage (<a href="https://redirect.github.com/tobymao/sqlglot/issues/4437">#4437</a>)</li> <li><a href="https://github.com/tobymao/sqlglot/commit/fe3539464a153b1c0bf46975d6221dee48a48f02"><code>fe35394</code></a> Fix(optimizer)!: fix datetime coercion in the canonicalize rule (<a href="https://redirect.github.com/tobymao/sqlglot/issues/4431">#4431</a>)</li> <li><a href="https://github.com/tobymao/sqlglot/commit/41d6a13ccfb28fbcf772fd43ea17da3b36567e67"><code>41d6a13</code></a> fix: add return type (<a href="https://redirect.github.com/tobymao/sqlglot/issues/4440">#4440</a>)</li> <li><a href="https://github.com/tobymao/sqlglot/commit/b24aced2dbb7e471d2dd0eb830ea4f2e24f9d267"><code>b24aced</code></a> Refactor(snowflake): clean up [WITH] TAG property / constraint (<a href="https://redirect.github.com/tobymao/sqlglot/issues/4439">#4439</a>)</li> <li><a href="https://github.com/tobymao/sqlglot/commit/37863ffd747cad9c2b9bed60119cc1551faeffda"><code>37863ff</code></a> feat(snowflake): Transpile non-UNNEST exp.GenerateDateArray refs (<a href="https://redirect.github.com/tobymao/sqlglot/issues/4433">#4433</a>)</li> <li><a href="https://github.com/tobymao/sqlglot/commit/69d4a8ccdf5954f293acbdf61c420b72dde5b8af"><code>69d4a8c</code></a> fix(tsql): Map weekday to %w (<a href="https://redirect.github.com/tobymao/sqlglot/issues/4438">#4438</a>)</li> <li>Additional commits viewable in <a href="https://github.com/tobymao/sqlglot/compare/v25.30.0...v25.32.1">compare view</a></li> </ul> </details> <br /> <details> <summary>Most Recent Ignore Conditions Applied to This Pull Request</summary> | Dependency Name | Ignore Conditions | | --- | --- | | sqlglot | [>= 24.a, < 25] | | sqlglot | [>= 25.31.dev0, < 25.32] | </details> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: SundarShankar89 <[email protected]>
This PR adjusts the way transpilation results are checked during tests so that the formatted values can be easily checked in a debugger. (IntelliJ/PyCharm have better ways of comparing output than the side-by-side view we display.)
|
raised a fresh PR with all latest code #1403 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Reconcile currently supports only oracle , snowflake and databricks .
Have added code to integrate SQL Server in Remorph Reconcile