diff --git a/.github/ISSUE_TEMPLATE/bug-report.yml b/.github/ISSUE_TEMPLATE/bug-report.yml index 17c2c545..39ea753b 100644 --- a/.github/ISSUE_TEMPLATE/bug-report.yml +++ b/.github/ISSUE_TEMPLATE/bug-report.yml @@ -1,7 +1,7 @@ name: šŸž Bug description: Report a bug or an issue you've found within the dbt package title: "[Bug] " -labels: ["bug", "triage"] +labels: ["type:bug"] body: - type: markdown attributes: @@ -35,6 +35,12 @@ body: description: A concise description of what you expected to happen. validations: required: true + - type: textarea + attributes: + label: Possible solution + description: Were you able to investigate and/or discover a potential fix to this bug in your investigation? If so, it would be much appreciated if you could submit code samples to show us how your fix resolved this issue. + validations: + required: false - type: textarea attributes: label: dbt Project configurations @@ -61,6 +67,19 @@ body: - other (mention it in "Additional Context") validations: required: true + - type: dropdown + id: orchestration_type + attributes: + label: How are you running this dbt package? + multiple: true + options: + - Quickstart + - Fivetran Transformations + - dbt Coreā„¢ + - dbt Cloudā„¢ + - other (mention it in "Additional Context") + validations: + required: true - type: textarea attributes: label: dbt Version @@ -83,6 +102,6 @@ body: description: Our team will assess this issue and let you know if we will add it to a future sprint. However, if you would like to expedite the solution, we encourage you to contribute to the package via a PR. Our team will then work with you to approve and merge your contributions as soon as possible. options: - label: Yes. - - label: Yes, but I will need assistance and will schedule time during our [office hours](https://calendly.com/fivetran-solutions-team/fivetran-solutions-team-office-hours) for guidance + - label: Yes, but I will need assistance. - label: No. required: false \ No newline at end of file diff --git a/.github/ISSUE_TEMPLATE/feature-request.yml b/.github/ISSUE_TEMPLATE/feature-request.yml index a1d28bbb..529e9bc5 100644 --- a/.github/ISSUE_TEMPLATE/feature-request.yml +++ b/.github/ISSUE_TEMPLATE/feature-request.yml @@ -1,7 +1,7 @@ name: šŸŽ‰ Feature description: Suggest a new feature for the Fivetran dbt package title: "[Feature] <title>" -labels: ["enhancement"] +labels: ["type:enhancement"] body: - type: markdown attributes: @@ -20,6 +20,13 @@ body: description: A clear and concise description of what you want to happen and why you want the new feature. validations: required: true + - type: textarea + attributes: + label: How would you implement this feature? + description: | + How would you build out this feature with your existing data? Any code examples you can provide to help accelerate development on this issue? + validations: + required: true - type: textarea attributes: label: Describe alternatives you've considered @@ -34,7 +41,7 @@ body: description: Our team will assess this feature and let you know if we will add it to a future sprint. However, if you would like to expedite the feature, we encourage you to contribute to the package via a PR. Our team will then work with you to approve and merge your contributions as soon as possible. options: - label: Yes. - - label: Yes, but I will need assistance and will schedule time during your [office hours](https://calendly.com/fivetran-solutions-team/fivetran-solutions-team-office-hours) for guidance. + - label: Yes, but I will need assistance. - label: No. required: false - type: textarea diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index b4e7e8e0..d4d1335b 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -19,6 +19,12 @@ <!--- To select a checkbox you simply need to add an "x" with no spaces between the brackets (eg. [x] Yes). --> - [ ] Yes +**There are commonly additional maintenance changes required before this will be ready for an upcoming release. Are you comfortable with the Fivetran team making a few commits directly to your branch?** +<!--- If you select Yes this will help expedite your PR in case there are small changes required before approval. We encourage you not to use this branch in a production environment as we may make additional updates. --> +<!--- If you select No, we will not make any changes directly to your branch and will either communicate any planned changes via the PR thread or will merge your PR into a separate branch so we may make changes without modifying your branch.. --> +- [ ] Yes +- [ ] No + **Provide an emoji that best describes your current mood** <!--- For a complete list of markdown compatible emojis check our this git repo (https://gist.github.com/rxaviers/7360908) --> :dancer: diff --git a/.gitignore b/.gitignore index ee0ba26d..a899762f 100644 --- a/.gitignore +++ b/.gitignore @@ -5,6 +5,4 @@ logs/ .gitignore .DS_Store dbt_packages/ -integration_tests/.DS_Store -integration_tests/seeds/.DS_Store -integration_tests/package-lock.yml \ No newline at end of file +package-lock.yml \ No newline at end of file diff --git a/.quickstart/quickstart.yml b/.quickstart/quickstart.yml index d98ff7c8..b05d4d28 100644 --- a/.quickstart/quickstart.yml +++ b/.quickstart/quickstart.yml @@ -9,6 +9,16 @@ table_variables: fivetran_platform_using_user: - user +public_models: [ + "fivetran_platform__audit_table", + "fivetran_platform__audit_user_activity", + "fivetran_platform__connector_daily_events", + "fivetran_platform__connector_status", + "fivetran_platform__mar_table_history", + "fivetran_platform__schema_changelog", + "fivetran_platform__usage_mar_destination_history" +] + destination_configurations: databricks: dispatch: diff --git a/README.md b/README.md index 3a4b1d3d..280d5c13 100644 --- a/README.md +++ b/README.md @@ -4,17 +4,15 @@ <img src="https://img.shields.io/badge/License-Apache%202.0-blue.svg" /></a> <a alt="dbt-core"> <img src="https://img.shields.io/badge/dbt_Coreā„¢_version->=1.3.0_<2.0.0-orange.svg" /></a> - <a alt="Maintained?"> - <img src="https://img.shields.io/badge/Maintained%3F-yes-green.svg" /></a> - <a alt="PRs"> - <img src="https://img.shields.io/badge/Contributions-welcome-blueviolet" /></a> + <a alt="Maintained and PRs Welcomed"> + <img src="https://img.shields.io/badge/Maintained_and_PRs_Welcomed%3F-yes-green.svg" /></a> <a alt="Fivetran Quickstart Compatible" href="https://fivetran.com/docs/transformations/dbt/quickstart"> <img src="https://img.shields.io/badge/Fivetran_Quickstart_Compatible%3F-yes-green.svg" /></a> </p> # Fivetran Platform dbt Package ([Docs](https://fivetran.github.io/dbt_fivetran_log/)) -# šŸ“£ What does this dbt package do? +## What does this dbt package do? - Generates a comprehensive data dictionary of your Fivetran Platform connector (previously called Fivetran Log) data via the [dbt docs site](https://fivetran.github.io/dbt_fivetran_log/) - Produces staging models in the format described by [this ERD](https://fivetran.com/docs/logs/fivetran-platform#schemainformation) which clean, test, and prepare your Fivetran data from our free [Fivetran Platform connector](https://fivetran.com/docs/logs/fivetran-platform) and generates analysis ready end models. @@ -27,7 +25,7 @@ <!--section="fivetran_platform_transformation_model"--> Refer to the table below for a detailed view of all models materialized by default within this package. Additionally, check out our [docs site](https://fivetran.github.io/dbt_fivetran_log/#!/overview/fivetran_platform?g_v=1&g_e=seeds) for more details about these models. -## Models +### Models | **model** | **description** | | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | @@ -40,12 +38,12 @@ Refer to the table below for a detailed view of all models materialized by defau | [fivetran_platform__audit_user_activity](https://fivetran.github.io/dbt_fivetran_log/#!/model/model.fivetran_log.fivetran_platform__audit_user_activity) | Each record represents a user-triggered action in your Fivetran instance. This model is intended for audit-trail purposes, as it can be very helpful when trying to trace a user action to a [log event](https://fivetran.com/docs/logs#logeventlist) such as a schema change, sync frequency update, manual update, broken connection, etc. | <!--section-end--> -# šŸŽÆ How do I use the dbt package? -## Step 1: Pre-Requisites +## How do I use the dbt package? +### Step 1: Pre-Requisites - **Connector**: Have the Fivetran Platform connector syncing data into your warehouse. - **Database support**: This package has been tested on **BigQuery**, **Snowflake**, **Redshift**, **Postgres**, **Databricks**, and **SQL Server**. Ensure you are using one of these supported databases. -### Databricks Dispatch Configuration +#### Databricks Dispatch Configuration If you are using a Databricks destination with this package you will need to add the below (or a variation of the below) dispatch configuration within your `dbt_project.yml`. This is required in order for the package to accurately search for macros within the `dbt-labs/spark_utils` then the `dbt-labs/dbt_utils` packages respectively. ```yml dispatch: @@ -53,7 +51,7 @@ dispatch: search_order: ['spark_utils', 'dbt_utils'] ``` -### Database Incremental Strategies +#### Database Incremental Strategies For models in this package that are materialized incrementally, they are configured to work with the different strategies available to each supported warehouse. For **BigQuery** and **Databricks All Purpose Cluster runtime** destinations, we have chosen `insert_overwrite` as the default strategy, which benefits from the partitioning capability. @@ -63,8 +61,12 @@ For **Snowflake**, **Redshift**, and **Postgres** databases, we have chosen `del > Regardless of strategy, we recommend that users periodically run a `--full-refresh` to ensure a high level of data quality. -## Step 2: Installing the Package -Include the following Fivetran Platform package version range in your `packages.yml` +### Step 2: Running the Package +#### Option 1: As a Fivetran Quickstart Data Model +Fivetran offers this dbt package as a one-click managed solution via the [Quickstart Data Model product](https://fivetran.com/docs/transformations/quickstart). You can take advantage of this by logging into your Fivetran account, navigate to your Fivetran Platform connector, and select the Transformations tab. From the Transformations tab you can add the Quickstart transformation and let Fivetran manage the execution and orchestration. + +#### Option 2: As a dbt package in your existing dbt project +To run these models as a dbt package in your existing dbt project you will include the following Fivetran Platform package version range in your `packages.yml` > Check [dbt Hub](https://hub.getdbt.com/) for the latest installation instructions, or [read the dbt docs](https://docs.getdbt.com/docs/package-management) for more information on installing packages. ```yaml packages: @@ -74,7 +76,7 @@ packages: > Note that although the source connector is now "Fivetran Platform", the package retains the old name of "fivetran_log". -## Step 3: Define Database and Schema Variables +### Step 3: Define Database and Schema Variables By default, this package will run using your target database and the `fivetran_log` schema. If this is not where your Fivetran Platform data is (perhaps your fivetran platform schema is `fivetran_platform`), add the following configuration to your root `dbt_project.yml` file: ```yml @@ -83,7 +85,7 @@ vars: fivetran_platform_schema: your_schema_name # default is fivetran_log ``` -## Step 4: Disable Models for Non Existent Sources +### Step 4: Disable Models for Non Existent Sources If you do not leverage Fivetran RBAC, then you will not have the `user` or `destination_membership` sources. It's also possible you might not have any To disable the corresponding functionality in the package, you must add the following variable(s) to your root `dbt_project.yml` file. By default, all variables are assumed to be `true`: ```yml @@ -92,9 +94,9 @@ vars: fivetran_platform_using_user: false # this will disable only the user logic ``` -## (Optional) Step 5: Additional Configurations +### (Optional) Step 5: Additional Configurations -### Change the Build Schema +#### Change the Build Schema By default this package will build the Fivetran staging models within a schema titled (<target_schema> + `_stg_fivetran_platform`) and the Fivetran Platform final models within your <target_schema> + `_fivetran_platform` in your target database. If this is not where you would like you Fivetran staging and final models to be written to, add the following configuration to your root `dbt_project.yml` file: ```yml @@ -105,21 +107,21 @@ models: +schema: my_new_staging_models_schema # leave blank for just the target_schema ``` -### Change the Source Table References +#### Change the Source Table References If an individual source table has a different name than expected (see this projects [dbt_project.yml](https://github.com/fivetran/dbt_fivetran_log/blob/main/dbt_project.yml) variable declarations for expected names), provide the name of the table as it appears in your warehouse to the respective variable as identified below: ```yml vars: fivetran_platform_<default_table_name>_identifier: your_table_name ``` -## (Optional) Step 6: Orchestrate your models with Fivetran Transformations for dbt Coreā„¢ +### (Optional) Step 6: Orchestrate your models with Fivetran Transformations for dbt Coreā„¢ <details><summary>Expand for details</summary> <br> Fivetran offers the ability for you to orchestrate your dbt project through [Fivetran Transformations for dbt Coreā„¢](https://fivetran.com/docs/transformations/dbt). Refer to the linked docs for more information on how to setup your project for orchestration through Fivetran. </details> -# šŸ” Does this package have dependencies? +## Does this package have dependencies? This dbt package is dependent on the following dbt packages. Please be aware that these dependencies are installed by default within this package. For more information on the below packages, refer to the [dbt hub](https://hub.getdbt.com/) site. > **If you have any of these dependent packages in your own `packages.yml` I highly recommend you remove them to ensure there are no package version conflicts.** ```yml @@ -137,16 +139,15 @@ packages: version: [">=0.9.0", "<1.0.0"] ``` -# šŸ™Œ How is this package maintained and can I contribute? -## Package Maintenance +## How is this package maintained and can I contribute? +### Package Maintenance The Fivetran team maintaining this package **only** maintains the latest version of the package. We highly recommend you stay consistent with the [latest version](https://hub.getdbt.com/fivetran/fivetran_log/latest/) of the package and refer to the [CHANGELOG](https://github.com/fivetran/dbt_fivetran_log/blob/main/CHANGELOG.md) and release notes for more information on changes across versions. -## Contributions +### Contributions These dbt packages are developed by a small team of analytics engineers at Fivetran. However, the packages are made better by community contributions! We highly encourage and welcome contributions to this package. Check out [this post](https://discourse.getdbt.com/t/contributing-to-a-dbt-package/657) on the best workflow for contributing to a package! -# šŸŖ Are there any resources available? +## Are there any resources available? - If you encounter any questions or want to reach out for help, please refer to the [GitHub Issue](https://github.com/fivetran/dbt_fivetran_log/issues/new/choose) section to find the right avenue of support for you. -- If you would like to provide feedback to the dbt package team at Fivetran, or would like to request a future dbt package to be developed, then feel free to fill out our [Feedback Form](https://www.surveymonkey.com/r/DQ7K7WW). -- Have questions or want to be part of the community discourse? Create a post in the [Fivetran community](https://community.fivetran.com/t5/user-group-for-dbt/gh-p/dbt-user-group) and our team along with the community can join in on the discussion! +- If you would like to provide feedback to the dbt package team at Fivetran, or would like to request a future dbt package to be developed, then feel free to let us know via the [Fivetran Support Portal](https://support.fivetran.com/hc/en-us). \ No newline at end of file