Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: Snowflake support - statement #127

Merged
merged 3 commits into from
Nov 2, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion packages/doc/docs/connectors.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ We support the following data warehouses to connect with, you can choose multipl
| ------------------------------------- | ------ | ----------- | ----------------- |
| [PostgreSQL](./connectors/postgresql) | ✅ Yes | ✅ Yes | ❌ No |
| [DuckDB](./connectors/duckdb) | ✅ Yes | ✅ Yes | ❌ No |
| Snowflake | | | |
| [Snowflake](./connectors/snowflake) | ✅ Yes | ✅ Yes | ❌ No |
| BigQuery | | | |

\* Fetching rows only when we need them, it has better performance with large query results.
Expand Down
66 changes: 66 additions & 0 deletions packages/doc/docs/connectors/snowflake.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
# Snowflake

Connect with your Snowflake servers via the official [Node.js Driver](https://docs.snowflake.com/en/user-guide/nodejs-driver.html).

## Installation

1. Install package

```bash
npm i @vulcan-sql/extension-driver-snowflake
```

:::info
If you run VulcanSQL with Docker, you should use the command `vulcan-install @vulcan-sql/extension-driver-snowflake` instead.

:::

2. Update `vulcan.yaml`, and enable the extension.

```yaml
extensions:
...
// highlight-next-line
snow: '@vulcan-sql/extension-driver-snowflake' # Add this line
```

3. Create a new profile in `profiles.yaml` or in your profile files. For example:

```yaml
- name: snow # profile name
type: snowflake
connection:
account: 'xy123456.ap-northeast-1.aws'
username: xxxx
password: xxxx
warehouse: test
min: 0
max: 3
allow: '*'
```

## Configuration

Please check [arguments of Snowflake SDK](https://docs.snowflake.com/en/user-guide/nodejs-driver-use.html#setting-the-connection-options) and [options of Generic Pool](https://github.com/coopernurse/node-pool) for further information.

| Name | Required | Default | Description |
| -------------------- | -------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| account | Y | | Your account [identifier](https://docs.snowflake.com/en/user-guide/admin-account-identifier.html). |
| username | Y | | he login name for your Snowflake user or your Identity Provider (e.g. your login name for Okta). |
| application | N | VulcanSQL | Specifies the name of the client application connecting to Snowflake. |
| authenticator | N | SNOWFLAKE | Specifies the authenticator to use for verifying user login credentials. |
| password | N | | Password for the user. Set this option if you set the authenticator option to SNOWFLAKE or the Okta URL endpoint for your Okta account. |
| token | N | | Specifies the OAuth token to use for authentication. Set this option if you set the authenticator option to OAUTH. |
| privateKey | N | | Specifies the private key (in PEM format) for key pair authentication |
| privateKeyPath | N | | Specifies the local path to the private key file (e.g. rsa_key.p8) |
| privateKeyPass | N | | Specifies the passcode to decrypt the private key file, if the file is encrypted. |
| database | N | | The default database to use for the session after connecting. |
| warehouse | N | | The default virtual warehouse to use for the session after connecting. Used for performing queries, loading data, etc. |
| timeout | N | 60000 | Number of milliseconds to keep the connection alive with no response. |
| role | N | | The default security role to use for the session after connecting. |
| schema | N | | The default schema to use for the session after connecting. |
| max | N | 1 | Maximum number of connection to create at any given time. |
| min | N | 0 | Minimum number of connection to keep at any given time. If this is set >= max, the pool will silently set the min to equal max. |
| maxWaitingClients | N | 0 | Maximum number of queued connections allowed. |
| acquireTimeoutMillis | N | 0 | Max milliseconds a request will wait for a connection before timing out, if supplied should non-zero positive integer. |
| destroyTimeoutMillis | N | 0 | Max milliseconds a connection will wait for closing before timing out, if supplied should non-zero positive integer. |
12 changes: 3 additions & 9 deletions packages/doc/sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -150,15 +150,9 @@ const sidebars = {
link: { type: 'doc', id: 'connectors' },
items: [
{
type: 'doc',
id: 'connectors/postgresql',
label: 'PostgreSQL',
},
{
type: 'doc',
id: 'connectors/duckdb',
label: 'DuckDB',
},
type: 'autogenerated',
dirName: 'connectors'
}
],
},
'deployment',
Expand Down
18 changes: 18 additions & 0 deletions packages/extension-driver-snowflake/.eslintrc.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
{
"extends": ["../../.eslintrc.json"],
"ignorePatterns": ["!**/*"],
"overrides": [
{
"files": ["*.ts", "*.tsx", "*.js", "*.jsx"],
"rules": {}
},
{
"files": ["*.ts", "*.tsx"],
"rules": {}
},
{
"files": ["*.js", "*.jsx"],
"rules": {}
}
]
}
78 changes: 78 additions & 0 deletions packages/extension-driver-snowflake/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
# extension-driver-snowflake

[snowflake](https://www.snowflake.com/en/) driver for VulcanSQL.

## Install

1. Install package

```sql
npm i @vulcan-sql/extension-driver-snowflake
```

2. Update `vulcan.yaml`, enable the extension.

```yaml
extensions:
snowflake: '@vulcan-sql/extension-driver-snowflake'
```

3. Create a new profile in `profiles.yaml` or in your profiles' paths.
Please check [arguments of Snowflake SDK](https://docs.snowflake.com/en/user-guide/nodejs-driver-use.html#setting-the-connection-options) and [options of Generic Pool](https://github.com/coopernurse/node-pool) for further information.

```yaml
- name: snow # profile name
type: snowflake
connection:
# Required: Your account identifier. https://docs.snowflake.com/en/user-guide/admin-account-identifier.html
account: 'xy12345.ap-northeast-1.aws'
# Required: The login name for your Snowflake user or your Identity Provider (e.g. your login name for Okta).
username: 'vulcan'
# Optional: Specifies the name of the client application connecting to Snowflake.
application: 'VulcanSQL'
# Optional: Specifies the authenticator to use for verifying user login credentials.
authenticator: 'SNOWFLAKE'
# Optional: Password for the user. Set this option if you set the authenticator option to SNOWFLAKE or the Okta URL endpoint for your Okta account.
password: 'pass'
# Optional: Specifies the OAuth token to use for authentication. Set this option if you set the authenticator option to OAUTH.
token: ''
# Optional: Specifies the private key (in PEM format) for key pair authentication
privateKey: ''
# Optional: Specifies the local path to the private key file (e.g. rsa_key.p8)
privateKeyPath: ''
# Optional: Specifies the passcode to decrypt the private key file, if the file is encrypted.
privateKeyPass: ''
# Optional: The default database to use for the session after connecting.
database: ''
# Optional: The default virtual warehouse to use for the session after connecting. Used for performing queries, loading data, etc.
warehouse: ''
# Optional: Number of milliseconds to keep the connection alive with no response. Default: 60000 (1 minute).
timeout: 60000
# Optional: The default security role to use for the session after connecting.
role: ''
# Optional: The default schema to use for the session after connecting.
schema: ''
# Optional: Maximum number of connection to create at any given time. (default=1)
max: 1
# Optional: Minimum number of connection to keep at any given time. If this is set >= max, the pool will silently set the min to equal max. (default=0)
min: 0
# Optional: Maximum number of queued connections allowed.
maxWaitingClients: 0
# Optional: Max milliseconds a request will wait for a connection before timing out. (default no limit), if supplied should non-zero positive integer.
acquireTimeoutMillis: 0
# Optional: Max milliseconds a connection will wait for closing before timing out. (default no limit), if supplied should non-zero positive integer.
destroyTimeoutMillis: 0
```

## Testing

```bash
nx test extension-driver-snowflake
```

To run test, the following environment variables are required:

- SNOWFLAKE_ACCOUNT
- SNOWFLAKE_USER
- SNOWFLAKE_PASSWORD
- SNOWFLAKE_WAREHOUSE
14 changes: 14 additions & 0 deletions packages/extension-driver-snowflake/jest.config.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
module.exports = {
displayName: 'extension-driver-snowflake',
preset: '../../jest.preset.ts',
globals: {
'ts-jest': {
tsconfig: '<rootDir>/tsconfig.spec.json',
},
},
transform: {
'^.+\\.[tj]s$': 'ts-jest',
},
moduleFileExtensions: ['ts', 'js', 'html'],
coverageDirectory: '../../coverage/packages/extension-driver-snowflake',
};
35 changes: 35 additions & 0 deletions packages/extension-driver-snowflake/package.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
{
"name": "@vulcan-sql/extension-driver-snowflake",
"description": "Snowflake driver for VulcanSQL",
"version": "0.3.0",
"type": "commonjs",
"publishConfig": {
"access": "public"
},
"keywords": [
"vulcan",
"vulcan-sql",
"data",
"sql",
"database",
"data-warehouse",
"data-lake",
"api-builder",
"snowflake",
"snow"
],
"repository": {
"type": "git",
"url": "https://github.com/Canner/vulcan.git"
},
"license": "MIT",
"dependencies": {
"snowflake-sdk": "^1.6.14"
},
"devDependencies": {
"@types/snowflake-sdk": "^1.6.8"
},
"peerDependencies": {
"@vulcan-sql/core": "~0.3.0-0"
}
}
85 changes: 85 additions & 0 deletions packages/extension-driver-snowflake/project.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
{
"root": "packages/extension-driver-snowflake",
"sourceRoot": "packages/extension-driver-snowflake/src",
"targets": {
"build": {
"executor": "@nrwl/workspace:run-commands",
"options": {
"command": "yarn ts-node ./tools/scripts/replaceAlias.ts extension-driver-snowflake"
},
"dependsOn": [
{
"projects": "self",
"target": "tsc"
},
{
"projects": "self",
"target": "install-dependencies"
}
]
},
"tsc": {
"executor": "@nrwl/js:tsc",
"outputs": ["{options.outputPath}"],
"options": {
"outputPath": "dist/packages/extension-driver-snowflake",
"main": "packages/extension-driver-snowflake/src/index.ts",
"tsConfig": "packages/extension-driver-snowflake/tsconfig.lib.json",
"assets": ["packages/extension-driver-snowflake/*.md"],
"buildableProjectDepsInPackageJsonType": "dependencies"
},
"dependsOn": [
{
"projects": "dependencies",
"target": "build"
},
{
"projects": "self",
"target": "install-dependencies"
}
]
},
"lint": {
"executor": "@nrwl/linter:eslint",
"outputs": ["{options.outputFile}"],
"options": {
"lintFilePatterns": ["packages/extension-driver-snowflake/**/*.ts"]
}
},
"test": {
"executor": "@nrwl/jest:jest",
"outputs": ["coverage/packages/extension-driver-snowflake"],
"options": {
"jestConfig": "packages/extension-driver-snowflake/jest.config.ts",
"passWithNoTests": true
},
"dependsOn": [
{
"projects": "self",
"target": "install-dependencies"
}
]
},
"publish": {
"executor": "@nrwl/workspace:run-commands",
"options": {
"command": "node ../../../tools/scripts/publish.mjs {args.tag} {args.version}",
"cwd": "dist/packages/extension-driver-snowflake"
},
"dependsOn": [
{
"projects": "self",
"target": "build"
}
]
},
"install-dependencies": {
"executor": "@nrwl/workspace:run-commands",
"options": {
"command": "yarn",
"cwd": "packages/extension-driver-snowflake"
}
}
},
"tags": []
}
3 changes: 3 additions & 0 deletions packages/extension-driver-snowflake/src/index.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
export * from './lib/snowflakeDataSource';
import { SnowflakeDataSource } from './lib/snowflakeDataSource';
export default [SnowflakeDataSource];
Loading