TDL-14887: Respect field selection #67

hpatel41 · 2021-09-02T14:25:39Z

Description of change

TDL-14887: Respect field selection

Added code change for field selection and transformation.

Manual QA steps

Risks

Rollback steps

revert this branch

…xacttarget into use-transformation

karanpanchal-crest · 2021-09-07T09:54:23Z

tap_exacttarget/endpoints/content_areas.py

@@ -134,6 +138,8 @@ def sync_data(self):
                                     'ModifiedDate',
                                     content_area.get('ModifiedDate'))

-            singer.write_records(table, [content_area])
+            with Transformer() as transformer:


@harshpatel4crest Can you make this code generic among all the streams?

Done the code change

karanpanchal-crest · 2021-09-07T10:27:09Z

tests/test_exacttarget_field_selection.py

+                                   if message['action'] == 'upsert'][0]
+
+                if stream == 'list':
+                    expected_keys = expected_keys - {


@harshpatel4crest Why are we extracting fields like 'SendClassification', 'PartnerProperties', etc. from the expected kets?

Yes, they are not retrievable. So, added a comment regarding that.

karanpanchal-crest · 2021-09-07T10:27:23Z

tests/test_exacttarget_field_selection.py

+                        'SendClassification',
+                        'PartnerProperties'}
+                elif stream == 'subscriber':
+                    expected_keys = expected_keys - {


Same question as above

Yes, they are not retrievable. So, added a comment regarding that.

…xacttarget into use-transformation

karanpanchal-crest · 2021-09-08T10:28:33Z

tap_exacttarget/dao.py

@@ -59,6 +59,12 @@ def get_catalog_keys(self):
    def parse_object(self, obj):
        return project(obj, self.get_catalog_keys())

+    # a function to write records with applying transformation
+    def write_records_with_transform(self, record, catalog, table): # pylint: disable=no-self-use


@harshpatel4crest Can you please try resolving this pylint error instead of disabling it

…xacttarget into use-transformation

kspeer825

Tests approved 👍

…xacttarget into use-transformation

KrisPersonal

For all endpoints, please separate out schema from .py file to schema and json file.

KrisPersonal

For all endpoints, please separate out schema from .py file to schema and json file.

KrisPersonal · 2021-10-07T18:13:22Z

We have to ensure the extracted .json schema files are the same as the generated schemas. With the schema extracted, it’ll be easier to focus on the sync-specific code in each of those files

hpatel41 · 2021-10-08T10:54:40Z

We have to ensure the extracted .json schema files are the same as the generated schemas. With the schema extracted, it’ll be easier to focus on the sync-specific code in each of those files

Separated schemas from .py files into .json files.

hpatel41 · 2021-10-12T12:21:50Z

tap_exacttarget/endpoints/campaigns.py


 LOGGER = singer.get_logger()


 class CampaignDataAccessObject(DataAccessObject):

-    SCHEMA = with_properties({


Removed schema.
Similarly for other files too.

hpatel41 · 2021-10-12T12:22:46Z

tap_exacttarget/endpoints/campaigns.py

 import singer

 from tap_exacttarget.client import request
 from tap_exacttarget.dao import DataAccessObject
-from tap_exacttarget.schemas import with_properties


Removed unused import as the schema is removed.
Similarly for other files too.

hpatel41 · 2021-10-12T12:27:01Z

tap_exacttarget/schemas.py

@@ -1,64 +0,0 @@
-def with_properties(properties):


Removed this file, as the schema is separated from the .py file. Hence, created a definitions.json for the common (reference) fields used in the schema.

…nResetErrors (#71) * added backoff for certain errors * resolve pylint * updated decorator location * added unittests * added comment * added comments

* updated error message when generating auth_stub * made changes according to the comments * updated the code acording to the comments * updated the tap tester image * updated pylint and astroid to latest version * updated the code as, on updating tap tester image it was throwing cascading errors * updated config.yml file * updated the start date for integration tests as per the params * removed scenario as new tap-tester version does not support it * updated start date in the base file test

* added best practices * resolve pylint * resolve test failure * test: updated the test cases * test: updated some test cases * updated the code as per comments * resolve test case failure * updated the code as per comments * resolve integration test failure * resolve tes case failure * resolve test case failure * add data extension stream in test cases * added data extension incremental stream * test: run bookmark test * test: run bookmark test * test: run bookmark test, debug failing test * test: pylint resolve * test: run all data extenstion stream test in bookmark test * test: updated data extension code * updated the test to run data extension stream * run all tests * added sys.exit and updated pylint and astroid to latest versions * resolve pylint * updated the files as per comments * updated the code

…stream does not bookmark correctly (#75) * make keys automatic * pylint resolve * add full replication test case * added code change for data extension stream * pylint resolve * added comment * added comments * added comment in base file * updated discovery test and removed full replication test * updated the code * added a comment explaining subscriber and list subscriber syncing * added comments

harshpatel4_crest added 5 commits September 2, 2021 18:50

make keys automatic

da759d3

pylint resolve

23a93b3

add full replication test case

f3de6a9

use transformation

d08a706

resolve pylint

e71aaf3

hpatel41 requested a review from karanpanchal-crest September 2, 2021 14:29

harshpatel4_crest added 7 commits September 3, 2021 17:53

added code change for data extension stream

bf15242

Merge branch 'automatic-fields' of https://github.com/singer-io/tap-e…

6c70f73

…xacttarget into use-transformation

pylint resolve

f25c815

Merge branch 'automatic-fields' of https://github.com/singer-io/tap-e…

42714f7

…xacttarget into use-transformation

updated test case for data extension

bbf14a3

added comment

af3ea3b

Merge branch 'automatic-fields' of https://github.com/singer-io/tap-e…

237d8b1

…xacttarget into use-transformation

karanpanchal-crest reviewed Sep 7, 2021

View reviewed changes

harshpatel4_crest added 6 commits September 7, 2021 17:35

added comments

eef1780

Merge branch 'automatic-fields' of https://github.com/singer-io/tap-e…

9b20b19

…xacttarget into use-transformation

added comments and optimized that condition

5f179fc

added code change for tranformation function in base file

88a8630

pylint resolve

bf8b487

disabled pylint error

2b12baf

karanpanchal-crest reviewed Sep 8, 2021

View reviewed changes

harshpatel4_crest added 3 commits September 8, 2021 17:58

test: removed disable pylint code

e5a98cf

added comment in base file

fa07e1c

Merge branch 'automatic-fields' of https://github.com/singer-io/tap-e…

b544e52

…xacttarget into use-transformation

hpatel41 marked this pull request as ready for review September 9, 2021 08:00

hpatel41 requested a review from karanpanchal-crest September 9, 2021 08:03

updated comment

c204136

hpatel41 requested a review from cosimon September 13, 2021 13:00

hpatel41 requested review from luandy64 and zachharris1 September 23, 2021 11:09

added a comment explaining subscriber and list subscriber syncing

8ab8353

kspeer825 approved these changes Oct 4, 2021

View reviewed changes

harshpatel4_crest added 3 commits October 7, 2021 11:33

added comments

684f752

Merge branch 'automatic-fields' of https://github.com/singer-io/tap-e…

a59243d

…xacttarget into use-transformation

updated comment

4abadef

KrisPersonal requested changes Oct 7, 2021

View reviewed changes

harshpatel4_crest added 4 commits October 8, 2021 14:34

made separate files for schemas

0232535

resolve pylint

ab43995

resolve integration test

5864acc

corrected typo

2a2b20c

hpatel41 requested a review from KrisPersonal October 8, 2021 10:52

hpatel41 commented Oct 12, 2021

View reviewed changes

KrisPersonal approved these changes Oct 12, 2021

View reviewed changes

hpatel41 and others added 7 commits October 13, 2021 11:44

TDL-14621: Add retry logic to requests and TDL-14622: Retry Connectio…

0418198

…nResetErrors (#71) * added backoff for certain errors * resolve pylint * updated decorator location * added unittests * added comment * added comments

resolve merge cconflicts: crest-master branch

3cd7436

updated the code

fc9fa00

resolved unittest case error

0824e00

hpatel41 mentioned this pull request Oct 13, 2021

Crest Work to Merge #72

Merged

hpatel41 changed the base branch from automatic-fields to master October 14, 2021 04:51

KrisPersonal mentioned this pull request Oct 14, 2021

Bump to version 1.7.0 #80

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TDL-14887: Respect field selection #67

TDL-14887: Respect field selection #67

hpatel41 commented Sep 2, 2021

karanpanchal-crest Sep 7, 2021

hpatel41 Sep 7, 2021

karanpanchal-crest Sep 7, 2021

hpatel41 Sep 7, 2021

karanpanchal-crest Sep 7, 2021

hpatel41 Oct 12, 2021

karanpanchal-crest Sep 8, 2021

hpatel41 Sep 8, 2021

kspeer825 left a comment

KrisPersonal left a comment

KrisPersonal left a comment

KrisPersonal commented Oct 7, 2021

hpatel41 commented Oct 8, 2021

hpatel41 Oct 12, 2021 •

edited

Loading

hpatel41 Oct 12, 2021 •

edited

Loading

hpatel41 Oct 12, 2021 •

edited

Loading

TDL-14887: Respect field selection #67

Are you sure you want to change the base?

TDL-14887: Respect field selection #67

Conversation

hpatel41 commented Sep 2, 2021

Description of change

Manual QA steps

Risks

Rollback steps

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kspeer825 left a comment

Choose a reason for hiding this comment

KrisPersonal left a comment

Choose a reason for hiding this comment

KrisPersonal left a comment

Choose a reason for hiding this comment

KrisPersonal commented Oct 7, 2021

hpatel41 commented Oct 8, 2021

hpatel41 Oct 12, 2021 • edited Loading

Choose a reason for hiding this comment

hpatel41 Oct 12, 2021 • edited Loading

Choose a reason for hiding this comment

hpatel41 Oct 12, 2021 • edited Loading

Choose a reason for hiding this comment

hpatel41 Oct 12, 2021 •

edited

Loading

hpatel41 Oct 12, 2021 •

edited

Loading

hpatel41 Oct 12, 2021 •

edited

Loading