Mapping in Airbyte UI #50881
Merged
Mapping in Airbyte UI #50881
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
octavia-squidington-iii
added
the
area/documentation
teallarson
reviewed
Jan 6, 2025
teallarson
reviewed
Jan 7, 2025
teallarson
reviewed
Jan 21, 2025
| @@ -38,7 +40,7 @@ There are many reasons you might want to encrypt data. | |||
| - **Efficient retrieval**: Hashing can enable faster lookups in databases. | |||
| - **Anonymity and compliance**: Source datasets can contain personally identifiable information (PII). Anonymizing PII can help you meet data privacy regulations like GDPR and HIPAA. | |||
|
|
|||
| <!-- Probably need some guidance on the use of keys with Airbyte, esp. wrt external secrets managers and AES encryption --> | |||
| <!-- Probably need some guidance on the use of keys with Airbyte, esp. wrt external secrets managers and AES encryption, and the validation rules --> | |||
There was a problem hiding this comment.
@pedroslopez or I can probably answer whatever questions you have here. Is this blocking?
teallarson
reviewed
Jan 21, 2025
|
|
||
| Before you begin mapping fields, consider the following. | ||
|
|
||
| - If you set up mapping on an existing connection, Airbyte prompts you to clear your stream and the data in your destination. You almost certainly should do this. Not doing this could result in undesirable or unexpected behavior. For example, if you set up hashing on a field that contains personally identifiable information (PII), you want all data in that column to be hashed, not just new data. Clearing your destination accomplishes this. However, if you're dealing with an extremely large database, a complete resync might take a significant amount of time. |
There was a problem hiding this comment.
For most destinations, we will recommend a refresh from within the UI, not a clear. The only exception to this is if the destination does not support refreshes (only a few Marketplace destinations are this way).
However, for this operation you are right that users will probably want a clear unless their sync mode uses Full Refresh.
teallarson
reviewed
Jan 21, 2025
| Before you begin mapping fields, consider the following. | ||
|
|
||
| - If you set up mapping on an existing connection, Airbyte prompts you to clear your stream and the data in your destination. You almost certainly should do this. Not doing this could result in undesirable or unexpected behavior. For example, if you set up hashing on a field that contains personally identifiable information (PII), you want all data in that column to be hashed, not just new data. Clearing your destination accomplishes this. However, if you're dealing with an extremely large database, a complete resync might take a significant amount of time. | ||
| - Do not map a cursor or primary key field if the [sync mode](core-concepts/sync-modes) depends on it. Doing this disrupts Airbyte's ability to use them for incremental syncing and deduplication. |
There was a problem hiding this comment.
@pedroslopez did we decide that this would actually be disruptive or not? I can't recall.
teallarson
approved these changes
Jan 21, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
Adds mapping docs to a "good enough" level.
How
Add a new topic explaining how mapping works.
Review guide
/docs/using-airbyte/mappings.md
User Impact
The feature for Teams customers is documented.
Can this PR be safely reverted and rolled back?