feat: dagster pipeline to ingest edx.org course via API #1455
+179
−4
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What are the relevant tickets?
https://github.com/mitodl/hq/issues/6337
Description (What does it do?)
adding a new dagster asset edxorg_mitx_course_metadata to ingest the MITx course data from the edx.org course catalog API. This will be part of the existing automation edxorg_api_schedule
How can this be tested?
This can't tested with the new API credential in https://vault-qa.odl.mit.edu/ui/vault/secrets/secret-data/show/pipelines/edx/edxorg/edx-oauth-client (blocked by https://github.com/mitodl/hq/issues/6638)
Once the credential issue is solved, it should be
dagster dev -d src/ol_orchestrate/ -f src/ol_orchestrate/definitions/edx/edxorg_api_data_extract.py
You should see the logs like: