Sync Datalad-Registry with datalad-usage-dashboard #287

candleindark · 2024-01-12T07:04:27Z

This PR does the following.

Implements a periodic Celery task for ensuring all active repos presented in datalad-usage-dashboard are registered in Datalad-Registry (excepts repos from OSF which will be supported later). This PR closes Automate (cron job script) getting updates from the datalad-usage-dashboard #271.
Provides a script to invoke the above Celery task on demand. Thus, an empty instance of Datalad-Registry can be populated with all the active repos presented in the usage dashboard by running this script.
Refactor some of the Pydantic models used by tools.populate.py so that these models can be used by the aforementioned Celery task as well.
responses will be added as a dependency for testing for the purpose of mocking out the requests library.

Note: Changes of this PR has been applied to the Typhon instance.

…oard

…ls.usage_dashboard`

With the one defined in `datalad_registry.tasks.utils.usage_dashboar`

For referring to the path of the dataset URLs resource on the DataLad Registry instance relative to the base API endpoint of the instance.

The definition is defined in `datalad_registry.blueprints.api.dataset_urls`

From `datalad_registry.blueprints.api.dataset_urls` to `datalad_registry.blueprints.api`

So that organization is consistent with the definition of `DATASET_URLS_PATH`

mypy does like the default to be just plain str

codecov · 2024-01-13T01:29:40Z

Codecov Report

Attention: 2 lines in your changes are missing coverage. Please review.

Comparison is base (ac489e8) 98.74% compared to head (9078f66) 98.73%.
Report is 1 commits behind head on master.

Files	Patch %	Lines
datalad_registry/tasks/utils/usage_dashboard.py	95.12%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #287      +/-   ##
==========================================
- Coverage   98.74%   98.73%   -0.02%     
==========================================
  Files          48       50       +2     
  Lines        2237     2366     +129     
==========================================
+ Hits         2209     2336     +127     
- Misses         28       30       +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

To specify invocation condition

For access to the DB

yarikoptic · 2024-01-16T21:18:31Z

datalad_registry/tasks/__init__.py

@@ -631,3 +636,101 @@ def chk_url_to_update(
        db.session.commit()

    return ChkUrlStatus.OK_UPDATED if is_record_updated else ChkUrlStatus.OK_CHK_ONLY
+
+
+class FailedSubmission(TypedDict):


why TypedDict and not dataclasses which seems to be more common construct pattern to use?

The reason I used type TypedDict is because a dictionary displays better in the Flower interface. I don't have fine control of how the result is displayed by Flower. Modifying __str__ or __repr__ didn't work, possibly due to serialization in the transmission from the worker service to the Flower service. Thus, I picked a type that displays more nicely by default.

yarikoptic · 2024-01-16T21:19:03Z

datalad_registry/tasks/__init__.py

+
+class FailedSubmission(TypedDict):
+    """
+    A TypedDict a failed submission of a repo URL to Datalad-Registry


not a sentence... not sure if worth stating TypedDict type here instead of purpose only

Yeah, stating TypedDict here is unnecessary. The docstring has been updated.

yarikoptic · 2024-01-16T21:19:44Z

@jwodder please review

candleindark · 2024-01-17T17:25:27Z

@yarikoptic Is it OK to merge now?

yarikoptic · 2024-01-17T21:43:05Z

sure... I didn't fully grasp why we need so many changes but if it accomplishes the drill -- let's proceed!

candleindark · 2024-01-18T01:37:27Z

sure... I didn't fully grasp why we need so many changes but if it accomplishes the drill -- let's proceed!

I think the additional changes you are referring to are those resulted from the refactoring of the tools/populate.py, testing with responses for mocking the requests calls, and the additional script to invoke the Celery tasks on demand.

candleindark added 29 commits January 5, 2024 13:41

Set up periodic task used for sync with usage dashboard

acc3469

Improve wording in comment

1bbe66a

Note exclusion of OSF repositories in usage_dashboard_sync

c7b1831

Add script to trigger syncing with usage dashboard on demand

38da6c5

Outline the steps in usage_dashboard_sync

bc509d1

Specify URL for JSON representation of repositories in usage dashboard

9ab4010

Define usage_dashboard.py for definitions for accessing usage dashb…

c46f48d

…oard

Move Status from tools.populate.py to `datalad_registry.tasks.uti…

cc6e682

…ls.usage_dashboard`

Update tests for the addition of the new periodic task

cae3e96

Add model templates for representing repos in dashboard

2fe3ddf

Define Pydantic model for git repo collection in datalad-usage-dashboard

9b48b40

Define Pydantic models for git repos in datalad-usage-dashboard

bc5ed27

Replace dashboard representation in tools/populate.py

a2523ff

With the one defined in `datalad_registry.tasks.utils.usage_dashboar`

Replace dashboard representation in tools/populate.py

8f9e2ce

With the one defined in `datalad_registry.tasks.utils.usage_dashboar`

Define clone_url in different repo representations

7a2db14

Utilize clone_url property of Repo in tools.populate.py

3e7a75f

Implement the usage_dashboard_sync task

8ea5301

Declare constant DATASET_URLS_PATH in dataset_urls.py

fdcc2ec

For referring to the path of the dataset URLs resource on the DataLad Registry instance relative to the base API endpoint of the instance.

Introduce DATALAD_REGISTRY_WEB_API_URL as config key

9b37095

Access repo submission URL through app configuration

7017923

Use DATASET_URLS_PATH definition in submit_urls.py

630e8d5

The definition is defined in `datalad_registry.blueprints.api.dataset_urls`

Move the definition of DATASET_URLS_PATH

23d844a

From `datalad_registry.blueprints.api.dataset_urls` to `datalad_registry.blueprints.api`

Define URL_METADATA_PATH in datalad_registry.blueprints.api

13c6aa5

So that organization is consistent with the definition of `DATASET_URLS_PATH`

Make DATALAD_REGISTRY_WEB_API_URL config optional in testing mode

88e456e

Improve comment and format

562a0d3

Add testing of DATALAD_REGISTRY_WEB_API_URL in TestCreateApp

0ae78e1

Update test_conf.py for DATALAD_REGISTRY_WEB_API_URL config key

48a750a

Merge branch 'master' into usage-dashboard-sync

d5f129e

Specify default for DATALAD_REGISTRY_WEB_API_URL as an AnyHttpUrl

bd517a2

mypy does like the default to be just plain str

candleindark added 2 commits January 12, 2024 20:06

Outline test_usage_dashboard_sync with first test case

5522910

Provide additional comment for sync_with_user_dashboard.py

41e4a8b

To specify invocation condition

candleindark added the enhancement New feature or request label Jan 15, 2024

candleindark added 7 commits January 15, 2024 10:04

Add responses as a dependency for testing

2e12029

Remove monkeypatch fixture in test_usage_dashboard_sync

5b3cfba

Use flask_app directly in test_usage_dashboard_sync

a32566c

For access to the DB

Modify test parameter values

19b9d89

Implement test_usage_dashboard_sync

a3eca12

Parametrize POST response status code in test_usage_dashboard_sync

2f3fbe0

Add additional test cases to test_usage_dashboard_sync

04345b8

candleindark marked this pull request as ready for review January 16, 2024 17:43

candleindark requested a review from yarikoptic January 16, 2024 18:14

yarikoptic reviewed Jan 16, 2024

View reviewed changes

Improve docstring of FailedSubmission

9078f66

jwodder approved these changes Jan 17, 2024

View reviewed changes

yarikoptic merged commit f6cb03c into datalad:master Jan 17, 2024
4 of 6 checks passed

candleindark deleted the usage-dashboard-sync branch January 18, 2024 01:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sync Datalad-Registry with datalad-usage-dashboard #287

Sync Datalad-Registry with datalad-usage-dashboard #287

candleindark commented Jan 12, 2024 •

edited

Loading

codecov bot commented Jan 13, 2024 •

edited

Loading

yarikoptic Jan 16, 2024

candleindark Jan 16, 2024 •

edited

Loading

yarikoptic Jan 16, 2024

candleindark Jan 16, 2024

yarikoptic commented Jan 16, 2024

candleindark commented Jan 17, 2024

yarikoptic commented Jan 17, 2024

candleindark commented Jan 18, 2024

Sync Datalad-Registry with datalad-usage-dashboard #287

Sync Datalad-Registry with datalad-usage-dashboard #287

Conversation

candleindark commented Jan 12, 2024 • edited Loading

codecov bot commented Jan 13, 2024 • edited Loading

Codecov Report

yarikoptic Jan 16, 2024

Choose a reason for hiding this comment

candleindark Jan 16, 2024 • edited Loading

Choose a reason for hiding this comment

yarikoptic Jan 16, 2024

Choose a reason for hiding this comment

candleindark Jan 16, 2024

Choose a reason for hiding this comment

yarikoptic commented Jan 16, 2024

candleindark commented Jan 17, 2024

yarikoptic commented Jan 17, 2024

candleindark commented Jan 18, 2024

candleindark commented Jan 12, 2024 •

edited

Loading

codecov bot commented Jan 13, 2024 •

edited

Loading

candleindark Jan 16, 2024 •

edited

Loading