Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merging jack_develop branch #8

Merged
merged 8 commits into from
Aug 26, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "NSDF Tutorial",
"image": "globalcomputinglab/somospie_openvisus:acmtutorial",
"name": "NSDF Tutorial - Session II",
"image": "globalcomputinglab/nsdf_tutorial:sessionII",
"containerEnv": {
"PROJ_LIB": "/opt/conda/envs/NSDF-Tutorial/share/proj"
},
Expand Down
25 changes: 25 additions & 0 deletions .devcontainer/session III/devcontainer.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
{
"name": "NSDF Tutorial - Session III",
"image": "globalcomputinglab/nsdf_tutorial:sessionIII",
"containerEnv": {
"PROJ_LIB": "/opt/conda/envs/NSDF-Tutorial/share/proj"
},
"customizations": {
"vscode": {
"extensions": [
"ms-python.python",
"ms-toolsai.jupyter",
"ms-vscode.cpptools",
"h5web.vscode-h5web",
"davidanson.vscode-markdownlint"
],
"settings": {
"C_Cpp.default.cppStandard": "c++17",
"C_Cpp.default.cStandard": "c99",
"python.defaultInterpreterPath": "/opt/conda/envs/NSDF-Tutorial/bin/python",
"python.languageServer": "Default",
"terminal.integrated.shell.linux": "/bin/bash"
}
}
}
}
17 changes: 13 additions & 4 deletions .github/workflows/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ on:
push:
branches:
- main
- jack_develop

jobs:
build:
Expand All @@ -18,13 +19,21 @@ jobs:
- name: Login to Docker Hub
run: echo ${{ secrets.DOCKERHUB_ACCESS_TOKEN }} | docker login -u ${{ secrets.DOCKERHUB_USERNAME }} --password-stdin

- name: Build and push register app
- name: Build and push register app session II
uses: docker/build-push-action@v3
with:
context: hands-on
context: hands-on/session II/
push: true
file: ./hands-on/Dockerfile
tags: ${{ secrets.DOCKERHUB_ORG }}/somospie_openvisus:acmtutorial
file: ./hands-on/session II/Materials/Dockerfile
tags: ${{ secrets.DOCKERHUB_ORG }}/nsdf_tutorial:sessionII

- name: Build and push register app session III
uses: docker/build-push-action@v3
with:
context: hands-on/session III
push: true
file: ./hands-on/session III/Dockerfile
tags: ${{ secrets.DOCKERHUB_ORG }}/nsdf_tutorial:sessionIII

env:
DOCKER_CLI_EXPERIMENTAL: enabled
Expand Down
5 changes: 5 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
.DS_store
hands-on/session\ II/Materials/files/tif_files/
hands-on/session\ II/Materials/files/log.log
hands-on/session\ II/Materials/idx_data/Tennessee_terrain_parameters/
hands-on/session\ II/Materials/GEOtiled/geotiled/src/__pycache__
24 changes: 12 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,18 +1,18 @@
# NSDF Tutorial: Using NSDF for End-to-End Analysis of Scientific Data.

## ACM Summer School Lecture, Barcelona, Spain, 2024.
# Enabling Scientific Discovery: Harnessing the Power of the National Science Data Fabric for Large-Scale Data Analysis

## Abstract

TBD
In this interactive half-day tutorial, participants explore the advanced applications of the National Science Data Fabric [(NSDF) services](https://nationalsciencedatafabric.org/) and comprehensive strategies for end-to-end scientific data analysis. The tutorial targets a broad audience, from researchers and students to developers and scientists, each finding valuable insights into managing and analyzing large datasets, with a particular focus on datasets exceeding 100TB.
Attendees gain hands-on experience constructing modular workflows, leveraging public and private data storage and streaming solutions, and deploying sophisticated visualization and analysis dashboards for scientific
discovery.
The tutorial highlights NSDF's role in supporting the VIS conference's themes by providing scalable solutions for advances in visualization and visual analytics. It covers various topics, from an overview of NSDF's capabilities to addressing common pain points in data analysis to intermediate hands-on exercises using NSDF services for Earth science data and advanced applications, including handling and visualizing massive datasets in domains requiring high-resolution data management.
Participants leave a deeper understanding of how NSDF services integrate into their research workflows to enhance data accessibility, sharing, and collaborative scientific discovery. This tutorial advances the knowledge of data-intensive computing and empowers attendees to harness the full potential of NSDF in their fields.

## Agenda

TBD

<!-- | Time | Session | Presenter |
| --- | --- | --- |
| 14:00 – 15:25 | [Part 1 : eFlows4HPC Lectures](slides/eFlows4HPC_ACM_2023.pdf) | Rosa M Badia (BSC) |
| 15:25 - 15:40 | Coffee break | |
| 15:40 - 16:55 | [Part 2: Hands-on session: How to build and run HPC Ready containers](hands-on/instructions.md) | Jorge Ejarque (BSC) |
| 16:55 - 17:00 | Tutorial conclusions | all presenters | -->
| **Session** | **Duration** | **Objective** |
| :---------: | :----------: | :----------------------------------------------------------------------------------------------------------------------------------------: |
| [I](./hands-on/session%20II/README.md) | 30 mins | This session begins with an overview of the NSDF and addresses users' challenges identified through interviews. |
| [II](./hands-on/session%20II/README.md) | 1 hour | This session offers a hands-on experience with NSDF services, focusing on visualization and dashboard creation for Earth science datasets. |
| [III](./hands-on/session%20III/README.md) | 1 hour | This session delves deeper into NSDF services tailored for the management and analysis of datasets exceeding 100TB. |
| [IV](./hands-on/session%20III/README.md) | 30 mins | This session concludes with an interactive Q&A, allowing attendees to discuss applications of NSDF in various research fields. |
2 changes: 0 additions & 2 deletions hands-on/.gitignore

This file was deleted.

36 changes: 0 additions & 36 deletions hands-on/CITATION.cff

This file was deleted.

156 changes: 0 additions & 156 deletions hands-on/Explore_Data.ipynb

This file was deleted.

Binary file not shown.
Loading
Loading