bugfix/string-agg-removal (#119)

* bugfix/string-agg-removal * regen docs and changelog update * readme version bump * Apply suggestions from code review Co-authored-by: Renee Li <[email protected]> * review updates and docs regen * docs regen --------- Co-authored-by: Renee Li <[email protected]>
fivetran · Mar 18, 2024 · 1856dd4 · 1856dd4
1 parent 2fe07aa
commit 1856dd4
Show file tree

Hide file tree

Showing 10 changed files with 33 additions and 34 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,3 +1,18 @@
+# dbt_fivetran_log v1.7.0
+[PR #119](https://github.com/fivetran/dbt_fivetran_log/pull/119) includes the following updates:
+
+## 🚨 Breaking Changes 🚨: Bug Fixes
+- The following fields have been deprecated (removed) as these fields proved to be problematic across warehouses due to the end size of the fields.
+  - `errors_since_last_completed_sync`
+  - `warnings_since_last_completed_sync`
+> Note: If you found these fields to be relevant, you may still reference the error/warning messages from within the underlying `log` table.
+- The `fivetran_platform_using_sync_alert_messages` variable has been removed as it is no longer necessary due to the above changes.
+
+## Feature Updates
+- The following fields have been added to display the number of error/warning messages sync last completed sync. These fields are intended to substitute the information from deprecated fields listed above.
+  - `number_errors_since_last_completed_sync`
+  - `number_warnings_since_last_completed_sync`
+
 # dbt_fivetran_log v1.6.0
 [PR #117](https://github.com/fivetran/dbt_fivetran_log/pull/117) includes the following updates as a result of users encountering numeric counts exceeding the limit of a standard integer. Therefore, these fields were required to be cast as `bigint` in order to avoid "integer out of range" errors:
 

diff --git a/README.md b/README.md
@@ -66,10 +66,10 @@ Include the following Fivetran Platform package version range in your `packages.
 ```yaml
 packages:
   - package: fivetran/fivetran_log
-    version: [">=1.6.0", "<1.7.0"]
+    version: [">=1.7.0", "<1.8.0"]
 ```
 
-> Note that altough the source connector is now "Fivetran Platform", the package retains the old name of "fivetran_log".
+> Note that although the source connector is now "Fivetran Platform", the package retains the old name of "fivetran_log".
 
 ## Step 3: Define Database and Schema Variables
 By default, this package will run using your target database and the `fivetran_log` schema. If this is not where your Fivetran Platform data is (perhaps your fivetran platform schema is `fivetran_platform`), add the following configuration to your root `dbt_project.yml` file:
@@ -91,14 +91,6 @@ vars:
 
 ## (Optional) Step 5: Additional Configurations
 
-### Configuring Fivetran Error and Warning Messages
-Some users may wish to exclude Fivetran error and warnings messages from the final `fivetran_platform__connector_status` model due to the length of the message. To disable the `errors_since_last_completed_sync` and `warnings_since_last_completed_sync` fields from the final model you may add the following variable to you to your root `dbt_project.yml` file. By default, this variable is assumed to be `true`:
-
-```yml
-vars:
-    fivetran_platform_using_sync_alert_messages: false # this will disable only the sync alert messages within the connector status model
-```
-
 ### Change the Build Schema
 By default this package will build the Fivetran staging models within a schema titled (<target_schema> + `_stg_fivetran_platform`)  and the Fivetran Platform final models within your <target_schema> + `_fivetran_platform` in your target database. If this is not where you would like you Fivetran staging and final models to be written to, add the following configuration to your root `dbt_project.yml` file:
 

diff --git a/dbt_project.yml b/dbt_project.yml
@@ -1,6 +1,6 @@
 config-version: 2
 name: 'fivetran_log'
-version: '1.6.0'
+version: '1.7.0'
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 
 models:

diff --git a/docs/catalog.json b/docs/catalog.json
diff --git a/docs/index.html b/docs/index.html
diff --git a/docs/manifest.json b/docs/manifest.json
diff --git a/docs/run_results.json b/docs/run_results.json
diff --git a/integration_tests/dbt_project.yml b/integration_tests/dbt_project.yml
@@ -1,5 +1,5 @@
 name: 'fivetran_log_integration_tests'
-version: '1.6.0'
+version: '1.7.0'
 
 config-version: 2
 profile: 'integration_tests'

diff --git a/models/fivetran_platform.yml b/models/fivetran_platform.yml
@@ -53,17 +53,11 @@ models:
         description: > 
           The number of schema changes (creating tables or schemas and altering tables) in the past 30 days.
 
-      - name: errors_since_last_completed_sync
-        description: >
-          Aggregated line-separated list of error messages (in JSON format) raised synce the last 
-          sync completion. Included by default for non-SQL Server targets, but can be disabled by setting
-          the `fivetran_platform_using_sync_alert_messages` var to False.
+      - name: number_errors_since_last_completed_sync
+        description: Numeric metric indicating the number of error messages raised since the last sync completion.
 
-      - name: warnings_since_last_completed_sync
-        description: >
-          Aggregated line-separated list of warning messages (in JSON format) raised synce the last 
-          sync completion. Included by default for non-SQL Server targets, but can be disabled by setting
-          the `fivetran_platform_using_sync_alert_messages` var to False.
+      - name: number_warnings_since_last_completed_sync
+        description: Numeric metric indicating the number of warning messages raised since the last sync completion.
 
 
   - name: fivetran_platform__mar_table_history

diff --git a/models/fivetran_platform__connector_status.sql b/models/fivetran_platform__connector_status.sql
@@ -198,12 +198,9 @@ final as (
         connector_recent_logs.last_sync_started_at,
         connector_recent_logs.last_sync_completed_at,
         connector_recent_logs.set_up_at,
-        coalesce(schema_changes.number_of_schema_changes_last_month, 0) as number_of_schema_changes_last_month
-
-        {% if var('fivetran_platform_using_sync_alert_messages', true) and target.type != 'sqlserver' %}
-        , {{ fivetran_utils.string_agg("distinct case when connector_recent_logs.event_type = 'SEVERE' then connector_recent_logs.message_data else null end", "'\\n'") }} as errors_since_last_completed_sync
-        , {{ fivetran_utils.string_agg("distinct case when connector_recent_logs.event_type = 'WARNING' then connector_recent_logs.message_data else null end", "'\\n'") }} as warnings_since_last_completed_sync
-        {% endif %}
+        coalesce(schema_changes.number_of_schema_changes_last_month, 0) as number_of_schema_changes_last_month,
+        count(case when connector_recent_logs.event_type = 'SEVERE' then connector_recent_logs.message_data else null end) as number_errors_since_last_completed_sync,
+        count(case when connector_recent_logs.event_type = 'WARNING' then connector_recent_logs.message_data else null end) as number_warnings_since_last_completed_sync
 
     from connector_recent_logs
     left join schema_changes 
@@ -226,4 +223,5 @@ final as (
         schema_changes.number_of_schema_changes_last_month
 )
 
-select * from final
+select * 
+from final