Skip to content

Latest commit

 

History

History
56 lines (41 loc) · 1.87 KB

ReadMe.md

File metadata and controls

56 lines (41 loc) · 1.87 KB

Introduction

THis repository contains some NSDF libraries. The structure of the directories is the following:

  • ansible/. Contains all ansible rols to setup hosts and install software
  • nsdf/
    • catalog/ contains all the NSDF file catalog files.
    • cloud/ contains the NSDF cloud source files (i.e. creating VM programmatcally)
    • kernel/ contains the basic code used by all other folders e.g. code to upload/download files from S3
    • convert/ contains code OpenVisus specific to convert from/to other file formats
    • distributed/ contains code specific to Dask/Prefect to run tasks in parallel
    • fuse/ contains the code specific to mount Object Storage as a file system. Compares several alternatives.
    • material_science_workflow/ all the code specific to material science
    • tamu/ all the code/scripts etc relative to Tamu collaboration

Please refer to each ReadMe.md file in each folder.

Vault

You will need credentials to use NSDF accounts:

git clone [email protected]:nsdf-fabric/vault.git ~/.nsdf/vault
# assuming you will have a ` ~/.nsdf/vault/vault.yaml` that will be used by the nsdf.kernel module (see function `NormalizeEnv`)

Security checks before any commit

# trufflehog
sudo docker run -it -v "$PWD:/pwd"  trufflesecurity/trufflehog:latest filesystem --directory=/pwd

# gitleaks
sudo docker run -v $PWD:/pwd zricethezav/gitleaks:latest detect -v --source="/pwd"

# git secrets ` &&  && &&  && `

# pushd /tmp
# git clone https://github.com/awslabs/git-secrets
# cd git-secrets
# sudo make install
# popd
git secrets --scan -r
git secrets --scan-history

To remove a leaked file:

FILENAME=nsdf/exxon/test-time.ipynb
git filter-branch -f --tree-filter 'rm -f $FILENAME' HEAD
git push origin --force --all