THis repository contains some NSDF libraries. The structure of the directories is the following:
ansible/
. Contains all ansible rols to setup hosts and install softwarensdf/
catalog/
contains all the NSDF file catalog files.cloud/
contains the NSDF cloud source files (i.e. creating VM programmatcally)kernel/
contains the basic code used by all other folders e.g. code to upload/download files from S3convert/
contains code OpenVisus specific to convert from/to other file formatsdistributed/
contains code specific to Dask/Prefect to run tasks in parallelfuse/
contains the code specific to mount Object Storage as a file system. Compares several alternatives.material_science_workflow/
all the code specific to material sciencetamu/
all the code/scripts etc relative to Tamu collaboration
Please refer to each ReadMe.md
file in each folder.
You will need credentials to use NSDF accounts:
git clone [email protected]:nsdf-fabric/vault.git ~/.nsdf/vault
# assuming you will have a ` ~/.nsdf/vault/vault.yaml` that will be used by the nsdf.kernel module (see function `NormalizeEnv`)
# trufflehog
sudo docker run -it -v "$PWD:/pwd" trufflesecurity/trufflehog:latest filesystem --directory=/pwd
# gitleaks
sudo docker run -v $PWD:/pwd zricethezav/gitleaks:latest detect -v --source="/pwd"
# git secrets ` && && && && `
# pushd /tmp
# git clone https://github.com/awslabs/git-secrets
# cd git-secrets
# sudo make install
# popd
git secrets --scan -r
git secrets --scan-history
To remove a leaked file:
FILENAME=nsdf/exxon/test-time.ipynb
git filter-branch -f --tree-filter 'rm -f $FILENAME' HEAD
git push origin --force --all