-
-
Notifications
You must be signed in to change notification settings - Fork 35
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #390 from slingdata-io/1.2.21
1.2.21
- Loading branch information
Showing
72 changed files
with
1,147 additions
and
777 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1,2 @@ | ||
|
|
||
| See https://github.com/slingdata-io/sling-cli/ for more details. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -183,108 +183,88 @@ Here are the links of the official development builds, which are the latest buil | |
|
|
||
| Then you should be able to run `sling --help` from command line. | ||
|
|
||
| ## Running a Extract-Load Task | ||
|
|
||
| ### CLI | ||
| ## Contributing | ||
|
|
||
| ```shell | ||
| sling run --src-conn POSTGRES_URL --src-stream myschema.mytable \ | ||
| --tgt-conn SNOWFLAKE_URL --tgt-object yourschema.yourtable \ | ||
| --mode full-refresh | ||
| ``` | ||
| We welcome contributions to improve Sling! Here are some guidelines to help you get started. | ||
|
|
||
| Or passing a yaml/json string or file | ||
| ### Branch Naming Convention | ||
|
|
||
| ```shell | ||
| sling run -c ' | ||
| source: | ||
| conn: $POSTGRES_URL | ||
| stream: myschema.mytable | ||
| target: | ||
| conn: $SNOWFLAKE_URL | ||
| object: yourschema.yourtable | ||
| mode: full-refresh | ||
| ' | ||
| # OR | ||
| sling run -c /path/to/config.json | ||
| When creating a new branch for your contribution, please use the following naming convention: | ||
|
|
||
| - `feature/your-feature-name` for new features | ||
| - `bugfix/issue-description` for bug fixes | ||
| - `docs/update-description` for documentation updates | ||
|
|
||
| ### Testing Guidelines | ||
|
|
||
| Sling has three main test suites: Database, File and CLI. When contributing, please ensure that your changes pass the relevant tests. | ||
|
|
||
| #### Running Tests | ||
|
|
||
| To run the full test suite, run below. However you'd need to define all the needed connections as shown [here](https://github.com/slingdata-io/sling-cli/blob/main/cmd/sling/sling_test.go#L52-L83), so it's recommended to target specific tests instead. | ||
|
|
||
| ```sh | ||
| ./scripts/build.sh | ||
| ./scripts/test.sh | ||
| ``` | ||
|
|
||
| ### From Lib | ||
| #### Targeting Specific Tests | ||
|
|
||
| ```go | ||
| package main | ||
| You can target specific tests or suites using environment variables: | ||
|
|
||
| import ( | ||
| "log" | ||
| 1. Database Suite: | ||
| ```sh | ||
| cd cmd/sling | ||
| go test -v -run TestSuiteDatabasePostgres # run all Postgres tests | ||
| TESTS="1-3" go test -v -run TestSuiteDatabasePostgres # run Postgres tests 1, 2, 3 | ||
| ``` | ||
|
|
||
| "github.com/slingdata-io/sling-cli/core/sling" | ||
| ) | ||
| 2. File Suite: | ||
| ```sh | ||
| cd cmd/sling | ||
| go test -v -run TestSuiteFileS3 # run all S3 tests | ||
| TESTS="1,2,3" go test -v -run TestSuiteFileS3 # run S3 tests 1, 2, 3 | ||
| ``` | ||
|
|
||
| func main() { | ||
| // cfgStr can be JSON or YAML | ||
| cfgStr := ` | ||
| source: | ||
| conn: $POSTGRES_URL | ||
| stream: myschema.mytable | ||
| 3. CLI Suite: | ||
| ```sh | ||
| cd cmd/sling | ||
| export SLING_BIN=../../sling | ||
| go test -v -run TestCLI # run all CLI tests | ||
| TESTS="31+" go test -v -run TestCLI # run CLI tests 31 and all subsequent tests | ||
| ``` | ||
|
|
||
| target: | ||
| conn: $SNOWFLAKE_URL | ||
| object: yourschema.yourtable | ||
| You can specify individual test numbers, ranges, or use the '+' suffix to run all tests from a certain number: | ||
|
|
||
| mode: full-refresh | ||
| ` | ||
| cfg, err := sling.NewConfig(cfgStr) | ||
| if err != nil { | ||
| log.Fatal(err) | ||
| } | ||
| - `TESTS="1,2,3"`: Run tests 1, 2, and 3 | ||
| - `TESTS="1-5"`: Run tests 1 through 5 | ||
| - `TESTS="3+"`: Run test 3 and all subsequent tests | ||
|
|
||
| err = sling.Sling(cfg) | ||
| if err != nil { | ||
| log.Fatal(err) | ||
| } | ||
| } | ||
| #### Test Suites Overview | ||
|
|
||
| ``` | ||
| 1. **Database Suite**: Tests database-related functionality. | ||
| - Located in: `cmd/sling/sling_test.go` | ||
| - Configuration: `cmd/sling/tests/suite.db.template.tsv` | ||
|
|
||
| ## Config Schema | ||
|
|
||
| An example. Put this in https://jsonschema.net/ | ||
|
|
||
| `--src-conn`/`source.conn` and `--tgt-conn`/`target.conn` can be a name or URL of a folder: | ||
| - `MY_PG` (connection ref in db, profile or env) | ||
| - `$MY_PG` (connection ref in env) | ||
| - `postgresql://user:[email protected]:5432/database` | ||
| - `s3://my_bucket/my_folder/file.csv` | ||
| - `gs://my_google_bucket/my_folder/file.json` | ||
| - `file:///tmp/my_folder/file.csv` (local storage) | ||
|
|
||
| `--src-stream`/`source.stream` can be an object name to stream from: | ||
| - `TABLE1` | ||
| - `SCHEMA1.TABLE2` | ||
| - `OBJECT_NAME` | ||
| - `select * from SCHEMA1.TABLE3` | ||
| - `/path/to/file.sql` (if source conn is DB) | ||
|
|
||
| `--tgt-object`/`target.object` can be an object name to write to: | ||
| - `TABLE1` | ||
| - `SCHEMA1.TABLE2` | ||
|
|
||
| ### Example as JSON | ||
|
|
||
| ```json | ||
| { | ||
| "source": { | ||
| "conn": "MY_PG_URL", | ||
| "stream": "select * from my_table", | ||
| "options": {} | ||
| }, | ||
| "target": { | ||
| "conn": "s3://my_bucket/my_folder/new_file.csv", | ||
| "options": { | ||
| "header": false | ||
| } | ||
| } | ||
| } | ||
| ``` | ||
| 2. **File Suite**: Tests file system operations. | ||
| - Located in: `cmd/sling/sling_test.go` | ||
| - Configuration: `cmd/sling/tests/suite.file.template.tsv` | ||
|
|
||
| 3. **CLI Suite**: Tests command-line interface functionality. | ||
| - Located in: `cmd/sling/sling_cli_test.go` | ||
| - Configuration: `cmd/sling/tests/suite.cli.tsv` | ||
|
|
||
| ### Adding New Tests | ||
|
|
||
| When introducing new features or addressing bugs, it's essential to incorporate relevant tests, focusing mainly on the CLI suite file located at `cmd/sling/suite.cli.tsv`. The database and file suites serve as templates applicable across all connectors, making them more sensitive to modifications. Therefore, any changes to these suites will be managed internally. | ||
|
|
||
| When adding new test entries in the CLI suite file, feel free to create a new replication file in folder `cmd/sling/tests/replications`, or a corresponding source file in the `cmd/sling/tests/files` directory. Also include the expected output or the number of expected rows/streams in the new test entry. | ||
|
|
||
| ### Pull Request Process | ||
|
|
||
| 1. Ensure your code adheres to the existing style and passes all tests. | ||
| 2. Update the README.md with details of changes to the interface, if applicable. | ||
| 3. Create a Pull Request with a clear title and description. | ||
|
|
||
| Thank you for contributing to Sling! | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.