From d4d13c57ae0f3b32cc3d60717f2452da92f750e7 Mon Sep 17 00:00:00 2001 From: Jaikumar Soundara Rajan <> Date: Fri, 21 Dec 2018 15:51:30 +0530 Subject: [PATCH 1/7] Issue #SC-543 feat: Organisation External Id ETL Jobs documentation --- .../installation/org_channel_migration.md | 51 +++++++++++++++++++ .../org_external_identity_migration.md | 51 +++++++++++++++++++ developer-docs/installation/sync_org.md | 50 ++++++++++++++++++ 3 files changed, 152 insertions(+) create mode 100644 developer-docs/installation/org_channel_migration.md create mode 100644 developer-docs/installation/org_external_identity_migration.md create mode 100644 developer-docs/installation/sync_org.md diff --git a/developer-docs/installation/org_channel_migration.md b/developer-docs/installation/org_channel_migration.md new file mode 100644 index 0000000000..e3edab8cab --- /dev/null +++ b/developer-docs/installation/org_channel_migration.md @@ -0,0 +1,51 @@ +--- +title: Organisation channel migration +page_title: Organisation channel migration +description: Organisation channel migration +published: true +allowSearch: true +--- + +## Overview +Sunbird, from its release version 1.13 onwards, captures the channel details of the tenant organization for all sub-organisations, in the channel column of organisation table in cassandra database. Sub-organisations created prior to this release version will not have channel values, and thus to ensure data consistency, this migration has to be done to set the channel value for all the existing sub-organisations based on their root organisation ID. + +## Prerequisites + +To run the migration script, ensure you have: + +1. Access to cassandra database +2. A backup of sunbird keyspace in Cassandra DB. + +## Configuration Parameters +The following parameters needs to be passed as arguments for the channel value migration job + + S.No. | Parameter | Description | Example +-------|-----------|-------------|--------- +1 | sunbird_cassandra_server | Cassandra DB IP Address| 198.168.1.1 +2 | sunbird_cassandra_port | Cassandra DB Port Number | 9042 +3 | sunbird_cassandra_username* | Cassandra DB User Name | username +4 | sunbird_cassandra_password* | Cassandra DB Password | password +5 | sunbird_cassandra_keyspace | Cassandra DB Keyspace Name | demodb +6 | sunbird_org_channel_migration_log_file | Path to CSV file where migration logs are stored | \home\channel_migration_log.csv + +> Note: If authentication is not required, pass `""` for parameters, username, and password + +## Migration Script + +To migrate channel value for the organisations: + +1. Extract the archive file (sunbird-utils/cassandra-migration-etl/r1.13/OrgMigrationUpdateChannelBin.zip) that contains the script for channel value migration + +2. Run the following command to migrate the data +
 
+OrgMigrationUpdateChannel_run.sh --context_param sunbird_cassandra_server="{sunbird_cassandra_server}" --context_param sunbird_cassandra_port="{sunbird_cassandra_port}" --context_param sunbird_cassandra_username="{sunbird_cassandra_username}" --context_param sunbird_cassandra_password="{sunbird_cassandra_password}" --context_param sunbird_cassandra_keyspace="{sunbird_cassandra_keyspace}" --context_param sunbird_org_channel_migration_log_file="{sunbird_org_channel_migration_log_file}"
+
+3. On successful migration, the log is available in the configured file {sunbird_org_channel_migration_log_file} + +4. To cross-check whether all organisations have channel value, the following queries can be used + + - Query to fetch number of organisations + ```select count(*) from organisation;``` + - Query to fetch number of organisations with channel value + ```select count(channel) from organisation;``` + diff --git a/developer-docs/installation/org_external_identity_migration.md b/developer-docs/installation/org_external_identity_migration.md new file mode 100644 index 0000000000..edeefa0747 --- /dev/null +++ b/developer-docs/installation/org_external_identity_migration.md @@ -0,0 +1,51 @@ +--- +title: Organisation external identity migration +page_title: Organisation external identity migration +description: Organisation external identity migration +published: true +allowSearch: true +--- + +## Overview +Sunbird, from its release version 1.13 onwards, captures the external id details of the organisations in a separate table(org_external_identity) in cassandra database. External Id details of organisations created before this release(which are captured in organisation table) has to populated in this new table to ensure data consistency. This migration job will take care of that activity. + +## Prerequisites + +To run the migration script, ensure you have: + +1. Access to cassandra database +2. A backup of sunbird keyspace in Cassandra DB. + +## Configuration Parameters +The following parameters needs to be passed as arguments for the organisation external identity migration job + + S.No. | Parameter | Description | Example +-------|-----------|-------------|--------- +1 | sunbird_cassandra_server | Cassandra DB IP Address| 198.168.1.1 +2 | sunbird_cassandra_port | Cassandra DB Port Number | 9042 +3 | sunbird_cassandra_username* | Cassandra DB User Name | username +4 | sunbird_cassandra_password* | Cassandra DB Password | password +5 | sunbird_cassandra_keyspace | Cassandra DB Keyspace Name | demodb +6 | sunbird_org_externalid_migration_log_file | Path to CSV file where migration logs are stored | \home\externalid_migration_log.csv + +> Note: If authentication is not required, pass `""` for parameters, username, and password + +## Migration Script + +To migrate external identity value for the organisations: + +1. Extract the archive file (sunbird-utils/cassandra-migration-etl/r1.13/OrgExternalIdentityMigrationBin.zip) that contains the script for external identity migration + +2. Run the following command to migrate the data +
 
+OrgExternalIdentityMigration_run.sh --context_param sunbird_cassandra_server="{sunbird_cassandra_server}" --context_param sunbird_cassandra_port="{sunbird_cassandra_port}" --context_param sunbird_cassandra_username="{sunbird_cassandra_username}" --context_param sunbird_cassandra_password="{sunbird_cassandra_password}" --context_param sunbird_cassandra_keyspace="{sunbird_cassandra_keyspace}" --context_param sunbird_org_externalid_migration_log_file="{sunbird_org_externalid_migration_log_file}"
+
+3. On successful migration, the log is available in the configured file {sunbird_org_externalid_migration_log_file} + +4. To cross-check whether all external ids has been populated to the org_external_identity table, the following queries can be used + + - Query to fetch number of organisations with external id + ```select count(externalid) from organisation;``` + - Query to fetch number of records from org_external_identity table + ```select count(*) from org_external_identity;``` + diff --git a/developer-docs/installation/sync_org.md b/developer-docs/installation/sync_org.md new file mode 100644 index 0000000000..3c6ec8c390 --- /dev/null +++ b/developer-docs/installation/sync_org.md @@ -0,0 +1,50 @@ +--- +title: Syncing Organisation Data +page_title: Syncing Organisation Data +description: Details on how to sync organisation's data +keywords: sync, org sync, org sync job +allowSearch: true +published: true +--- + +## Overview + +In Sunbird, all write operations are done in Cassandra and read operations are done from Elasticsearch. After write operations are completed in Cassandra, the data is written asynchronously into Elasticsearch as well. +In case of Cassandra migration that affects all organisation's data (e.g. OrgExternalIdentityMigration, OrgChannelMigration, etc.), this job is used to sync all organisation's data from Cassandra to Elasticsearch. + +## Prerequisites + +To sync organisation data, you need: + +1. Access to Cassandra database +2. API Key to access the Sync API + +## Configuration Parameters + +Pass the following parameters as arguments for the organisation sync job: + + S.No. | Parameter | Description | Example +-------|-----------|-------------|--------- +1 | sunbird_cassandra_server | The IP address of the Cassandra DB. This parameter is used to identify the server on which the Cassandra DB runs. The system uses the details provided to connect to the database.| 198.168.1.1 +2 | sunbird_cassandra_port | The port number of the Cassandra DB. This parameter is used to identify the port on which the Cassandra DB runs. The system uses the details provided to connect to the database.| 9042 +3 | sunbird_cassandra_username* | The user name for the Cassandra DB. This parameter is used to authenticate the user accessing the DB. | abc@xyz.com +4 | sunbird_cassandra_password* | The password for the Cassandra DB. This parameter is used to authenticate the user accessing the DB.| password +5 | sunbird_sync_api_endpoint | [Sync API](http://docs.sunbird.org/latest/apis/datasyncapi/#tag/Data-Sync-API(s)) | {{domain}}/api/data/v1/index/sync +6 | sunbird_sync_api_key | The API Key to access the Sync API. | As23456789zws34567w234 +7 | sunbird_sync_block_size | The number of org records to be synced per API call. | 1000 +8 | sunbird_sync_sleep_time | The time interval in milliseconds between API calls. | 5000 + +> Note: If you do not need to authenticate access to the cassandra, pass `""` value for the **username** and **password** parameters. + +## Syncing Organisation Data + +To sync organisation data from Cassandra to Elasticsearch: + +1. Extract the archive file (sunbird-utils/cassandra-migration-etl/r1.13/OrgSyncBin.zip) that contains the script to sync the organisation's data. + +2. Run the following command to sync all organisation's data. + +```` +OrgSync_run.sh --context_param sunbird_cassandra_server="{sunbird_cassandra_server}" --context_param sunbird_cassandra_port="{sunbird_cassandra_port}" --context_param sunbird_cassandra_username="{sunbird_cassandra_username}" --context_param sunbird_cassandra_password="{sunbird_cassandra_password}" --context_param sunbird_sync_api_endpoint="{sunbird_sync_api_endpoint}" --context_param sunbird_sync_api_key="{sunbird_sync_api_key}" --context_param sunbird_sync_block_size="{sunbird_sync_block_size}" --context_param sunbird_sync_sleep_time="{sunbird_sync_sleep_time}" +```` +3. On completion, refer to the success and failure logs that are generated From 9fab8ee01ec6193b5a6184be79f8e5cd04582c7e Mon Sep 17 00:00:00 2001 From: Jaikumar Soundara Rajan <> Date: Tue, 8 Jan 2019 14:32:34 +0530 Subject: [PATCH 2/7] Issue #SC-828 feat: Documentation for Email Case Change Migration --- developer-docs/installation/sync_user.md | 13 +++- .../user_email_case_change_migration.md | 68 +++++++++++++++++++ 2 files changed, 79 insertions(+), 2 deletions(-) create mode 100644 developer-docs/installation/user_email_case_change_migration.md diff --git a/developer-docs/installation/sync_user.md b/developer-docs/installation/sync_user.md index 27c46fe946..57369418e3 100644 --- a/developer-docs/installation/sync_user.md +++ b/developer-docs/installation/sync_user.md @@ -33,8 +33,10 @@ Pass the following parameters as arguments for the user sync job: 6 | sunbird_user_sync_api_key | The API Key to access the Sync API. | As23456789zws34567w234 7 | sunbird_user_sync_block_size | The number of user records to be synced per API call. | 100 8 | sunbird_user_sync_sleep_time | The time interval in milliseconds between API calls. | 10000 +9 | sunbird_user_sync_userids_file | Path of csv file, with no headers, containing user ids to be synced | /home/userIds.csv > Note: If you do not need to authenticate access to the API, pass `""` value for the **username** and **password** parameters. +> Note: If sunbird_user_sync_userids_file is not passed, all users from cassandra will be synced to elastic search ## Syncing User Data @@ -42,9 +44,16 @@ To sync user data from Cassandra to Elasticsearch: 1. Extract the [archive file](https://github.com/project-sunbird/sunbird-utils/tree/master/cassandra-migration-etl/common/UserSyncBin.zip) that contains the script to sync the user's data. -2. Run the following command to sync all users data. +2. Run the following command to sync **all users** data. ```` UserSync_run.sh --context_param sunbird_cassandra_server="{sunbird_cassandra_server}" --context_param sunbird_cassandra_port="{sunbird_cassandra_port}" --context_param sunbird_cassandra_username="{sunbird_cassandra_username}" --context_param sunbird_cassandra_password="{sunbird_cassandra_password}" --context_param sunbird_user_sync_api_endpoint="{sunbird_user_sync_api_endpoint}" --context_param sunbird_user_sync_api_key="{sunbird_user_sync_api_key}" --context_param sunbird_user_sync_block_size="{sunbird_user_sync_block_size}" --context_param sunbird_user_sync_sleep_time="{sunbird_user_sync_sleep_time}" ```` -3. On completion, refer to the success and failure logs that are generated + +3. Run the following command to sync **specific users** data. + +```` +UserSync_run.sh --context_param sunbird_cassandra_server="{sunbird_cassandra_server}" --context_param sunbird_cassandra_port="{sunbird_cassandra_port}" --context_param sunbird_cassandra_username="{sunbird_cassandra_username}" --context_param sunbird_cassandra_password="{sunbird_cassandra_password}" --context_param sunbird_user_sync_api_endpoint="{sunbird_user_sync_api_endpoint}" --context_param sunbird_user_sync_api_key="{sunbird_user_sync_api_key}" --context_param sunbird_user_sync_block_size="{sunbird_user_sync_block_size}" --context_param sunbird_user_sync_sleep_time="{sunbird_user_sync_sleep_time}" --context_param sunbird_user_sync_userids_file="{sunbird_user_sync_userids_file}" +```` + +4. On completion, refer to the success and failure logs that are generated diff --git a/developer-docs/installation/user_email_case_change_migration.md b/developer-docs/installation/user_email_case_change_migration.md new file mode 100644 index 0000000000..85d5041750 --- /dev/null +++ b/developer-docs/installation/user_email_case_change_migration.md @@ -0,0 +1,68 @@ +--- +title: User Email Case-Change Migration +page_title: User Email Case-Change Migration +description: User Email Case-Change Migration +published: true +allowSearch: true +--- + +## Overview +Sunbird, from its release version 1.13 onwards, captures the email of the users in lower case. Email of the users onboarded prior to this release version might have upper case characters. To ensure data consistency across users, this migration to convert upper case characters to lower case, in email has to be done. + +## Prerequisites + +To run the migration script, ensure you have: + +1. Access to cassandra database +2. A backup of sunbird keyspace in Cassandra DB. + +## Configuration Parameters +The following parameters needs to be passed as arguments for the email case-change migration job + + S.No. | Parameter | Description | Example +-------|-----------|-------------|--------- +1 | sunbird_cassandra_server | Cassandra DB IP Address| 198.168.1.1 +2 | sunbird_cassandra_port | Cassandra DB Port Number | 9042 +3 | sunbird_cassandra_username* | Cassandra DB User Name | username +4 | sunbird_cassandra_password* | Cassandra DB Password | password +5 | sunbird_cassandra_keyspace | Cassandra DB Keyspace Name | demodb +6 | sunbird_encryption_algorithm | Encryption Algorithm to encrypt email | DES/AES/RSA +7 | sunbird_encryption_iteration_count | Encryption Iteration Count to encrypt email | 5 +8 | dry_run | if enabled, only report generation and not actual migration happens | true + +> Note: If authentication is not required, pass `""` for parameters, username, and password + +## Environment Variables +The following parameters needs to be set in environment variables + + S.No. | Parameter | Description | Example +-------|-----------|-------------|--------- +1 | sunbird_encryption_key | Encryption Key to encrypt email | Encryption Key +2 | sunbird_cipher_key | Cipher Key to encrypt email | Cipher Key + +## Migration Script + +1. Extract the archive file (sunbird-utils/cassandra-migration-etl/r1.14/UserEmailCaseChangeMigrationBin.zip) + +2. Navigate to UserEmailCaseChangeMigrationBin/UserEmailCaseChangeMigration/ which contains the script file - UserEmailCaseChangeMigration_run.sh + +3. For dry-run, run the following command +
 
+UserEmailCaseChangeMigration_run.sh --context_param sunbird_cassandra_server="{sunbird_cassandra_server}" --context_param sunbird_cassandra_port="{sunbird_cassandra_port}" --context_param sunbird_cassandra_username="{sunbird_cassandra_username}" --context_param sunbird_cassandra_password="{sunbird_cassandra_password}" --context_param sunbird_cassandra_keyspace="{sunbird_cassandra_keyspace}" --context_param sunbird_encryption_algorithm="{sunbird_encryption_algorithm}" --context_param sunbird_encryption_iteration_count="{sunbird_encryption_iteration_count}" --context_param dry_run=true
+
+ +4. For email case-change migration, run the following command +
 
+UserEmailCaseChangeMigration_run.sh --context_param sunbird_cassandra_server="{sunbird_cassandra_server}" --context_param sunbird_cassandra_port="{sunbird_cassandra_port}" --context_param sunbird_cassandra_username="{sunbird_cassandra_username}" --context_param sunbird_cassandra_password="{sunbird_cassandra_password}" --context_param sunbird_cassandra_keyspace="{sunbird_cassandra_keyspace}" --context_param sunbird_encryption_algorithm="{sunbird_encryption_algorithm}" --context_param sunbird_encryption_iteration_count="{sunbird_encryption_iteration_count}" --context_param dry_run=false
+
+ +5. On completion of dry-run/migration, following logs will be generated and the logs path will be shown + + S.No. | Log File | Description | Example +-------|-----------|-------------|--------- +1 | emailsWithUpperCase_timestamp.csv | Email Ids available for case-change migration | emailsWithUpperCase_1546937413175.csv +2 | toSync.csv | User Ids to be synced(This file will be generated only in migration) | toSync.csv + +6. In case of migration, UserSync job has to be run for user ids captured in toSync.csv file to sync the migrated data from cassandra to elastic search + +7. Dry-run of the job can be run again to verify if any user's email still has upper case characters \ No newline at end of file From 4a1181d8d64ddc42c7380a4f2c58b19f4152689a Mon Sep 17 00:00:00 2001 From: Jaikumar Soundara Rajan <> Date: Tue, 8 Jan 2019 14:42:47 +0530 Subject: [PATCH 3/7] Revert "Issue #SC-828 feat: Documentation for Email Case Change Migration" This reverts commit 9fab8ee01ec6193b5a6184be79f8e5cd04582c7e. --- developer-docs/installation/sync_user.md | 13 +--- .../user_email_case_change_migration.md | 68 ------------------- 2 files changed, 2 insertions(+), 79 deletions(-) delete mode 100644 developer-docs/installation/user_email_case_change_migration.md diff --git a/developer-docs/installation/sync_user.md b/developer-docs/installation/sync_user.md index 57369418e3..27c46fe946 100644 --- a/developer-docs/installation/sync_user.md +++ b/developer-docs/installation/sync_user.md @@ -33,10 +33,8 @@ Pass the following parameters as arguments for the user sync job: 6 | sunbird_user_sync_api_key | The API Key to access the Sync API. | As23456789zws34567w234 7 | sunbird_user_sync_block_size | The number of user records to be synced per API call. | 100 8 | sunbird_user_sync_sleep_time | The time interval in milliseconds between API calls. | 10000 -9 | sunbird_user_sync_userids_file | Path of csv file, with no headers, containing user ids to be synced | /home/userIds.csv > Note: If you do not need to authenticate access to the API, pass `""` value for the **username** and **password** parameters. -> Note: If sunbird_user_sync_userids_file is not passed, all users from cassandra will be synced to elastic search ## Syncing User Data @@ -44,16 +42,9 @@ To sync user data from Cassandra to Elasticsearch: 1. Extract the [archive file](https://github.com/project-sunbird/sunbird-utils/tree/master/cassandra-migration-etl/common/UserSyncBin.zip) that contains the script to sync the user's data. -2. Run the following command to sync **all users** data. +2. Run the following command to sync all users data. ```` UserSync_run.sh --context_param sunbird_cassandra_server="{sunbird_cassandra_server}" --context_param sunbird_cassandra_port="{sunbird_cassandra_port}" --context_param sunbird_cassandra_username="{sunbird_cassandra_username}" --context_param sunbird_cassandra_password="{sunbird_cassandra_password}" --context_param sunbird_user_sync_api_endpoint="{sunbird_user_sync_api_endpoint}" --context_param sunbird_user_sync_api_key="{sunbird_user_sync_api_key}" --context_param sunbird_user_sync_block_size="{sunbird_user_sync_block_size}" --context_param sunbird_user_sync_sleep_time="{sunbird_user_sync_sleep_time}" ```` - -3. Run the following command to sync **specific users** data. - -```` -UserSync_run.sh --context_param sunbird_cassandra_server="{sunbird_cassandra_server}" --context_param sunbird_cassandra_port="{sunbird_cassandra_port}" --context_param sunbird_cassandra_username="{sunbird_cassandra_username}" --context_param sunbird_cassandra_password="{sunbird_cassandra_password}" --context_param sunbird_user_sync_api_endpoint="{sunbird_user_sync_api_endpoint}" --context_param sunbird_user_sync_api_key="{sunbird_user_sync_api_key}" --context_param sunbird_user_sync_block_size="{sunbird_user_sync_block_size}" --context_param sunbird_user_sync_sleep_time="{sunbird_user_sync_sleep_time}" --context_param sunbird_user_sync_userids_file="{sunbird_user_sync_userids_file}" -```` - -4. On completion, refer to the success and failure logs that are generated +3. On completion, refer to the success and failure logs that are generated diff --git a/developer-docs/installation/user_email_case_change_migration.md b/developer-docs/installation/user_email_case_change_migration.md deleted file mode 100644 index 85d5041750..0000000000 --- a/developer-docs/installation/user_email_case_change_migration.md +++ /dev/null @@ -1,68 +0,0 @@ ---- -title: User Email Case-Change Migration -page_title: User Email Case-Change Migration -description: User Email Case-Change Migration -published: true -allowSearch: true ---- - -## Overview -Sunbird, from its release version 1.13 onwards, captures the email of the users in lower case. Email of the users onboarded prior to this release version might have upper case characters. To ensure data consistency across users, this migration to convert upper case characters to lower case, in email has to be done. - -## Prerequisites - -To run the migration script, ensure you have: - -1. Access to cassandra database -2. A backup of sunbird keyspace in Cassandra DB. - -## Configuration Parameters -The following parameters needs to be passed as arguments for the email case-change migration job - - S.No. | Parameter | Description | Example --------|-----------|-------------|--------- -1 | sunbird_cassandra_server | Cassandra DB IP Address| 198.168.1.1 -2 | sunbird_cassandra_port | Cassandra DB Port Number | 9042 -3 | sunbird_cassandra_username* | Cassandra DB User Name | username -4 | sunbird_cassandra_password* | Cassandra DB Password | password -5 | sunbird_cassandra_keyspace | Cassandra DB Keyspace Name | demodb -6 | sunbird_encryption_algorithm | Encryption Algorithm to encrypt email | DES/AES/RSA -7 | sunbird_encryption_iteration_count | Encryption Iteration Count to encrypt email | 5 -8 | dry_run | if enabled, only report generation and not actual migration happens | true - -> Note: If authentication is not required, pass `""` for parameters, username, and password - -## Environment Variables -The following parameters needs to be set in environment variables - - S.No. | Parameter | Description | Example --------|-----------|-------------|--------- -1 | sunbird_encryption_key | Encryption Key to encrypt email | Encryption Key -2 | sunbird_cipher_key | Cipher Key to encrypt email | Cipher Key - -## Migration Script - -1. Extract the archive file (sunbird-utils/cassandra-migration-etl/r1.14/UserEmailCaseChangeMigrationBin.zip) - -2. Navigate to UserEmailCaseChangeMigrationBin/UserEmailCaseChangeMigration/ which contains the script file - UserEmailCaseChangeMigration_run.sh - -3. For dry-run, run the following command -
 
-UserEmailCaseChangeMigration_run.sh --context_param sunbird_cassandra_server="{sunbird_cassandra_server}" --context_param sunbird_cassandra_port="{sunbird_cassandra_port}" --context_param sunbird_cassandra_username="{sunbird_cassandra_username}" --context_param sunbird_cassandra_password="{sunbird_cassandra_password}" --context_param sunbird_cassandra_keyspace="{sunbird_cassandra_keyspace}" --context_param sunbird_encryption_algorithm="{sunbird_encryption_algorithm}" --context_param sunbird_encryption_iteration_count="{sunbird_encryption_iteration_count}" --context_param dry_run=true
-
- -4. For email case-change migration, run the following command -
 
-UserEmailCaseChangeMigration_run.sh --context_param sunbird_cassandra_server="{sunbird_cassandra_server}" --context_param sunbird_cassandra_port="{sunbird_cassandra_port}" --context_param sunbird_cassandra_username="{sunbird_cassandra_username}" --context_param sunbird_cassandra_password="{sunbird_cassandra_password}" --context_param sunbird_cassandra_keyspace="{sunbird_cassandra_keyspace}" --context_param sunbird_encryption_algorithm="{sunbird_encryption_algorithm}" --context_param sunbird_encryption_iteration_count="{sunbird_encryption_iteration_count}" --context_param dry_run=false
-
- -5. On completion of dry-run/migration, following logs will be generated and the logs path will be shown - - S.No. | Log File | Description | Example --------|-----------|-------------|--------- -1 | emailsWithUpperCase_timestamp.csv | Email Ids available for case-change migration | emailsWithUpperCase_1546937413175.csv -2 | toSync.csv | User Ids to be synced(This file will be generated only in migration) | toSync.csv - -6. In case of migration, UserSync job has to be run for user ids captured in toSync.csv file to sync the migrated data from cassandra to elastic search - -7. Dry-run of the job can be run again to verify if any user's email still has upper case characters \ No newline at end of file From 60668fab6c45d1431830b2913152f9d287f894d9 Mon Sep 17 00:00:00 2001 From: Basreena Date: Wed, 27 Feb 2019 10:38:12 +0530 Subject: [PATCH 4/7] Update org_channel_migration.md --- developer-docs/installation/org_channel_migration.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/developer-docs/installation/org_channel_migration.md b/developer-docs/installation/org_channel_migration.md index e3edab8cab..21d4686d6c 100644 --- a/developer-docs/installation/org_channel_migration.md +++ b/developer-docs/installation/org_channel_migration.md @@ -7,7 +7,7 @@ allowSearch: true --- ## Overview -Sunbird, from its release version 1.13 onwards, captures the channel details of the tenant organization for all sub-organisations, in the channel column of organisation table in cassandra database. Sub-organisations created prior to this release version will not have channel values, and thus to ensure data consistency, this migration has to be done to set the channel value for all the existing sub-organisations based on their root organisation ID. +Sunbird, from its release version 1.13 captures the channel details of the tenant organization for all sub-organisations. This data is captured in the channel column of the organisation table in cassandra database. Sub-organisations created prior to this release version will not have channel values. Thus to ensure data consistency this migration has to be done to set the channel value for all the existing sub-organisations based on their root organisation ID. ## Prerequisites @@ -17,7 +17,7 @@ To run the migration script, ensure you have: 2. A backup of sunbird keyspace in Cassandra DB. ## Configuration Parameters -The following parameters needs to be passed as arguments for the channel value migration job +The following parameters must be passed as arguments for the channel value migration job S.No. | Parameter | Description | Example -------|-----------|-------------|--------- @@ -40,9 +40,8 @@ To migrate channel value for the organisations: 
 
 OrgMigrationUpdateChannel_run.sh --context_param sunbird_cassandra_server="{sunbird_cassandra_server}" --context_param sunbird_cassandra_port="{sunbird_cassandra_port}" --context_param sunbird_cassandra_username="{sunbird_cassandra_username}" --context_param sunbird_cassandra_password="{sunbird_cassandra_password}" --context_param sunbird_cassandra_keyspace="{sunbird_cassandra_keyspace}" --context_param sunbird_org_channel_migration_log_file="{sunbird_org_channel_migration_log_file}"
-3. On successful migration, the log is available in the configured file {sunbird_org_channel_migration_log_file} -4. To cross-check whether all organisations have channel value, the following queries can be used +On successful migration, the log is available in the configured file {sunbird_org_channel_migration_log_file}. To cross-check whether all organisations have channel value, the following queries can be used - Query to fetch number of organisations ```select count(*) from organisation;``` From b0072f02957675f70dd1d1138d651774f975cdb9 Mon Sep 17 00:00:00 2001 From: Basreena Date: Wed, 27 Feb 2019 10:45:36 +0530 Subject: [PATCH 5/7] Update org_external_identity_migration.md --- .../installation/org_external_identity_migration.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/developer-docs/installation/org_external_identity_migration.md b/developer-docs/installation/org_external_identity_migration.md index edeefa0747..5f24eee96c 100644 --- a/developer-docs/installation/org_external_identity_migration.md +++ b/developer-docs/installation/org_external_identity_migration.md @@ -7,7 +7,7 @@ allowSearch: true --- ## Overview -Sunbird, from its release version 1.13 onwards, captures the external id details of the organisations in a separate table(org_external_identity) in cassandra database. External Id details of organisations created before this release(which are captured in organisation table) has to populated in this new table to ensure data consistency. This migration job will take care of that activity. +Sunbird, from its release version 1.13 onwards captures the external ID details of organisations in a separate table(org_external_identity) in cassandra database. Running the migration script moves the external ID details of organisations created before this release(which were captured in organisation table) to the new table to ensure data consistency. ## Prerequisites @@ -40,9 +40,8 @@ To migrate external identity value for the organisations: 
 
 OrgExternalIdentityMigration_run.sh --context_param sunbird_cassandra_server="{sunbird_cassandra_server}" --context_param sunbird_cassandra_port="{sunbird_cassandra_port}" --context_param sunbird_cassandra_username="{sunbird_cassandra_username}" --context_param sunbird_cassandra_password="{sunbird_cassandra_password}" --context_param sunbird_cassandra_keyspace="{sunbird_cassandra_keyspace}" --context_param sunbird_org_externalid_migration_log_file="{sunbird_org_externalid_migration_log_file}"
-3. On successful migration, the log is available in the configured file {sunbird_org_externalid_migration_log_file} -4. To cross-check whether all external ids has been populated to the org_external_identity table, the following queries can be used +On successful migration, the log is available in the configured file {sunbird_org_externalid_migration_log_file}. To cross-check whether all external ids has been populated to the org_external_identity table, the following queries can be used - Query to fetch number of organisations with external id ```select count(externalid) from organisation;``` From dc9770ef71dfdfb6cc452bfd1670be31040b8564 Mon Sep 17 00:00:00 2001 From: Basreena Date: Wed, 27 Feb 2019 15:50:29 +0530 Subject: [PATCH 6/7] Update sync_org.md --- developer-docs/installation/sync_org.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/developer-docs/installation/sync_org.md b/developer-docs/installation/sync_org.md index 3c6ec8c390..80a457f6f0 100644 --- a/developer-docs/installation/sync_org.md +++ b/developer-docs/installation/sync_org.md @@ -9,12 +9,12 @@ published: true ## Overview -In Sunbird, all write operations are done in Cassandra and read operations are done from Elasticsearch. After write operations are completed in Cassandra, the data is written asynchronously into Elasticsearch as well. -In case of Cassandra migration that affects all organisation's data (e.g. OrgExternalIdentityMigration, OrgChannelMigration, etc.), this job is used to sync all organisation's data from Cassandra to Elasticsearch. +In Sunbird, all write operations (transfer from data from CPU to a Memory Unit), are done in Cassandra and read operations (reading data from Memory Unit) are done through Elasticsearch. After write operations are completed in Cassandra, the data is written asynchronously into Elasticsearch. +On running a Cassandra migration all organisation data such as Organization External ID, Organization is affected. This document describes the script that can be run to sync organisation's data from Cassandra to Elasticsearch. ## Prerequisites -To sync organisation data, you need: +To sync organisation data, ensure you have: 1. Access to Cassandra database 2. API Key to access the Sync API @@ -47,4 +47,4 @@ To sync organisation data from Cassandra to Elasticsearch: ```` OrgSync_run.sh --context_param sunbird_cassandra_server="{sunbird_cassandra_server}" --context_param sunbird_cassandra_port="{sunbird_cassandra_port}" --context_param sunbird_cassandra_username="{sunbird_cassandra_username}" --context_param sunbird_cassandra_password="{sunbird_cassandra_password}" --context_param sunbird_sync_api_endpoint="{sunbird_sync_api_endpoint}" --context_param sunbird_sync_api_key="{sunbird_sync_api_key}" --context_param sunbird_sync_block_size="{sunbird_sync_block_size}" --context_param sunbird_sync_sleep_time="{sunbird_sync_sleep_time}" ```` -3. On completion, refer to the success and failure logs that are generated +On completion refer to the success and failure logs that are generated From 2750247fe591f7dcf74bb3a98574fc4a78514855 Mon Sep 17 00:00:00 2001 From: Basreena Date: Wed, 27 Feb 2019 15:52:42 +0530 Subject: [PATCH 7/7] Update sync_org.md --- developer-docs/installation/sync_org.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/developer-docs/installation/sync_org.md b/developer-docs/installation/sync_org.md index 80a457f6f0..39c82b50f8 100644 --- a/developer-docs/installation/sync_org.md +++ b/developer-docs/installation/sync_org.md @@ -9,8 +9,8 @@ published: true ## Overview -In Sunbird, all write operations (transfer from data from CPU to a Memory Unit), are done in Cassandra and read operations (reading data from Memory Unit) are done through Elasticsearch. After write operations are completed in Cassandra, the data is written asynchronously into Elasticsearch. -On running a Cassandra migration all organisation data such as Organization External ID, Organization is affected. This document describes the script that can be run to sync organisation's data from Cassandra to Elasticsearch. +In Sunbird, all transfers of data from CPU to a Memory Unit (all write operations) are done in Cassandra and all read operations (reading data from a Memory Unit) are done through Elasticsearch. Currently, after write operations are completed in Cassandra, the data is written asynchronously into Elasticsearch. +On running a Cassandra migration all organisation data such as Organization External ID, Organization are affected. This document describes the script that can be run to sync organisation's data from Cassandra to Elasticsearch. ## Prerequisites @@ -40,9 +40,9 @@ Pass the following parameters as arguments for the organisation sync job: To sync organisation data from Cassandra to Elasticsearch: -1. Extract the archive file (sunbird-utils/cassandra-migration-etl/r1.13/OrgSyncBin.zip) that contains the script to sync the organisation's data. +1. Extract the archive file (sunbird-utils/cassandra-migration-etl/r1.13/OrgSyncBin.zip) that contains the script to sync the organisation's data -2. Run the following command to sync all organisation's data. +2. Run the following command to sync all organisation's data ```` OrgSync_run.sh --context_param sunbird_cassandra_server="{sunbird_cassandra_server}" --context_param sunbird_cassandra_port="{sunbird_cassandra_port}" --context_param sunbird_cassandra_username="{sunbird_cassandra_username}" --context_param sunbird_cassandra_password="{sunbird_cassandra_password}" --context_param sunbird_sync_api_endpoint="{sunbird_sync_api_endpoint}" --context_param sunbird_sync_api_key="{sunbird_sync_api_key}" --context_param sunbird_sync_block_size="{sunbird_sync_block_size}" --context_param sunbird_sync_sleep_time="{sunbird_sync_sleep_time}"