Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cordex extension and trying to build on higher level abstractions #64

Open
wants to merge 29 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
3c85233
typos
huard Oct 3, 2024
f1c5683
work on extension abstraction and cordex example
huard Oct 10, 2024
c0bdeb2
removed json from .gitignore. Add CORDEX6 json schema
huard Oct 10, 2024
3a4b6ef
merge
huard Oct 10, 2024
b7d9b75
embedding datacube and thredds extension in the base logic
huard Oct 10, 2024
98c6420
got it to work
huard Oct 10, 2024
33d78b9
cordex implementation
huard Oct 11, 2024
0212008
add missing item_geometry_model
huard Oct 11, 2024
04d94df
get the cordex extension to work with the stac-populator cli.
huard Oct 11, 2024
ac0cec2
add some notes and comments
huard Oct 11, 2024
03e5738
clean-up
huard Oct 11, 2024
8cceeab
remove break
huard Oct 11, 2024
8b583bb
suggestions from review
huard Oct 15, 2024
8db0847
added `apply` method to extension helpers
huard Oct 15, 2024
df35431
include schemas in installation source
huard Oct 15, 2024
c6e68e8
Put generic STAC item logic into BaseSTAC class
huard Oct 15, 2024
d76fdd8
docstring
huard Oct 16, 2024
adb1065
added xscen extension. automated the creation of an alias generator f…
huard Oct 16, 2024
fd49161
added license_type to xscen schema. Test over ncml
huard Oct 17, 2024
08ccd42
update README and CHANGES.
huard Oct 17, 2024
2fb0eb6
Merge branch 'xscen_extension' into cordex_extension
huard Oct 17, 2024
312694c
suggestions from Mischa
huard Oct 17, 2024
a847c3f
need to pass properties dict
huard Oct 17, 2024
72a8271
suggestions from review
huard Oct 17, 2024
4817d66
upgrade numpy type conversion function
huard Oct 15, 2024
2b43022
add extensions automatically
huard Oct 17, 2024
118810c
rename _extensions to _helpers
huard Oct 18, 2024
d1d838e
format test data jsons
huard Oct 18, 2024
6c1c809
merge
huard Nov 12, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,9 @@ build
*.jsonl
*.json
huard marked this conversation as resolved.
Show resolved Hide resolved

## Exclude schemas
!schemas/**/*.json

# Old Submodule Path
# Could be used locally
pyessv-archive/
Expand Down
3 changes: 2 additions & 1 deletion CHANGES.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,8 @@
* move argument parsing for logging options to the implementation code
* fix bug where logging options were being set incorrectly
* rename files to avoid potential naming conflicts with other packages (`logging` and `requests`)

* Add `cordex6` extension and `CORDEX-CMIP6_Ouranos` implementation. This includes a refactoring of base extension classes.
* Add an `xscen` extension demonstrating how to add properties to a STAC Item.

## [0.6.0](https://github.com/crim-ca/stac-populator/tree/0.6.0) (2024-02-22)

Expand Down
8 changes: 5 additions & 3 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ APP_NAME := STACpopulator
APP_VERSION ?= 0.6.0

DOCKER_COMPOSE_FILES := -f "$(APP_ROOT)/docker/docker-compose.yml"
COMPOSE_PROJECT_NAME := stac-populator
DOCKER_TAG := ghcr.io/crim-ca/stac-populator:$(APP_VERSION)

IMP_DIR := $(APP_NAME)/implementations
Expand Down Expand Up @@ -35,6 +36,7 @@ del-cmip6:
curl --location --request DELETE '$(STAC_HOST)/collections/CMIP6_UofT'
@echo ""


docker-start:
docker compose $(DOCKER_COMPOSE_FILES) up
starthost: docker-start
Expand All @@ -43,12 +45,12 @@ docker-stop:
docker compose $(DOCKER_COMPOSE_FILES) down
stophost: docker-stop

del_docker_volume: stophost
docker volume rm stac-db

docker-build:
docker build "$(APP_ROOT)" -f "$(APP_ROOT)/docker/Dockerfile" -t "$(DOCKER_TAG)"

del_docker_volume: stophost
docker volume rm stac-populator_stac-db

resethost: del_docker_volume starthost

install:
Expand Down
10 changes: 6 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,13 +19,15 @@ contain all the logic for constructing the STAC representation for an item in th

Provided implementations of `STACpopulatorBase`:

| Implementation | Description |
|--------------------------------|-------------------------------------------------------------------------------------------------------------------------|
| [CMIP6_UofT][CMIP6_UofT] | Crawls a THREDDS Catalog for CMIP6 NCML-annotated NetCDF references to publish corresponding STAC Collection and Items. |
| [DirectoryLoader][DirLoader] | Crawls a subdirectory hierarchy of pre-generated STAC Collections and Items to publish to a STAC API endpoint. |
| Implementation | Description |
|----------------------------------------------|-------------------------------------------------------------------------------------------------------------------------|
| [CMIP6_UofT][CMIP6_UofT] | Crawls a THREDDS Catalog for CMIP6 NCML-annotated NetCDF references to publish corresponding STAC Collection and Items. |
| [DirectoryLoader][DirLoader] | Crawls a subdirectory hierarchy of pre-generated STAC Collections and Items to publish to a STAC API endpoint. |
| [CORDEX-CMIP6_Ouranos][CORDEX-CMIP6_Ouranos] | Crawls a THREDDS Catalog for CORDEX-CMIP6 NetCDF references to publish corresponding STAC Collection and Items. |

[CMIP6_UofT]: STACpopulator/implementations/CMIP6_UofT/add_CMIP6.py
[DirLoader]: STACpopulator/implementations/DirectoryLoader/crawl_directory.py
[CORDEX-CMIP6_Ouranos]: STACpopulator/implementations/CORDEX-CMIP6_Ouranos/add_CORDEX-CMIP6.py

## Installation and Execution

Expand Down
Empty file.
Loading
Loading