Skip to content

nsdf-fabric/nsdf-software-stack

Repository files navigation

Introduction

THis repository contains some NSDF libraries. The structure of the directories is the following:

  • ansible/. Contains all ansible rols to setup hosts and install software
  • nsdf/
    • catalog/ contains all the NSDF file catalog files.
    • cloud/ contains the NSDF cloud source files (i.e. creating VM programmatcally)
    • kernel/ contains the basic code used by all other folders e.g. code to upload/download files from S3
    • convert/ contains code OpenVisus specific to convert from/to other file formats
    • distributed/ contains code specific to Dask/Prefect to run tasks in parallel
    • fuse/ contains the code specific to mount Object Storage as a file system. Compares several alternatives.
    • material_science_workflow/ all the code specific to material science
    • tamu/ all the code/scripts etc relative to Tamu collaboration

Please refer to each ReadMe.md file in each folder.

Vault

You will need credentials to use NSDF accounts:

git clone [email protected]:nsdf-fabric/vault.git ~/.nsdf/vault
# assuming you will have a ` ~/.nsdf/vault/vault.yaml` that will be used by the nsdf.kernel module (see function `NormalizeEnv`)

Security checks before any commit

# trufflehog
sudo docker run -it -v "$PWD:/pwd"  trufflesecurity/trufflehog:latest filesystem --directory=/pwd

# gitleaks
sudo docker run -v $PWD:/pwd zricethezav/gitleaks:latest detect -v --source="/pwd"

# git secrets ` &&  && &&  && `

# pushd /tmp
# git clone https://github.com/awslabs/git-secrets
# cd git-secrets
# sudo make install
# popd
git secrets --scan -r
git secrets --scan-history 

To remove a leaked file:

FILENAME=nsdf/exxon/test-time.ipynb
git filter-branch -f --tree-filter 'rm -f $FILENAME' HEAD
git push origin --force --all

About

No description, website, or topics provided.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published