You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
A shortcoming in our sync-wrds-location-db step function surfaced during our UAT testing for v2.1.8. The man_rnr_* pipeline was failing every run due to issues with the underlying wrds_location3 database as accessed via the VIZ db foreign table connections. The two main errors were:
The "nwm_feature_id" column of the external.full_crosswalk_view had data in it that exceeded the size of its "integer" dtype.
The "external.nwm_routelink_3_0_conus" table does not exist.
To be clear, these were occurring after syncing the underlying wrds_location3 database. After digging, I found that the reason was due to how foreign tables work in PostgreSQL. Essentially a copy of the foreign tables are made on the reference database. This means that if anything about those table schemas change (or if new tables need a foreign connection) then the tables need to be synced or re-synced.
We have code that resyncs these foreign tables as part of our automated Terraform deployment - Core/EC2/RDSBastion/scripts/utils/setup_foreign_tables.tftpl. But that is an ephemeral script that gets executed only once among the first steps of a deployment. We need that same code to be executed as the final step after the wrds_location3 database dump is deployed so that any tweak in the underlying databases can be reflected properly.
The manual step in the meantime to fix the above errors was to execute the following sql:
DROP FOREIGN TABLE external.full_crosswalk_view, nwm_routelink_3_0_conus;
IMPORT FOREIGN SCHEMA public LIMIT TO (full_crosswalk_view, nwm_routelink_3_0_conus) FROM SERVER wrds_location INTO external;
The text was updated successfully, but these errors were encountered:
A shortcoming in our sync-wrds-location-db step function surfaced during our UAT testing for v2.1.8. The
man_rnr_*
pipeline was failing every run due to issues with the underlying wrds_location3 database as accessed via the VIZ db foreign table connections. The two main errors were:To be clear, these were occurring after syncing the underlying wrds_location3 database. After digging, I found that the reason was due to how foreign tables work in PostgreSQL. Essentially a copy of the foreign tables are made on the reference database. This means that if anything about those table schemas change (or if new tables need a foreign connection) then the tables need to be synced or re-synced.
We have code that resyncs these foreign tables as part of our automated Terraform deployment - Core/EC2/RDSBastion/scripts/utils/setup_foreign_tables.tftpl. But that is an ephemeral script that gets executed only once among the first steps of a deployment. We need that same code to be executed as the final step after the wrds_location3 database dump is deployed so that any tweak in the underlying databases can be reflected properly.
The manual step in the meantime to fix the above errors was to execute the following sql:
The text was updated successfully, but these errors were encountered: