v1.0.0-beta1 release

v1.0.0-beta1 (2023-03-13)

To access data that is hosted on the NDA users must first create a Data Package through the NDA website and then use the NDA API to request data from those Data Packages. This release simplifies the process of downloading ABCC (Collection 3165) data from a Data Package once it has been created. The ABCC dataset is roughly 168TB with over 13 million files and is rapidly growing in size as more data is acquired. Due to the unusual suze if this collection the typical method of downloading data from the NDA via nda-tools has proven to be time consuming and prone to failure. As a result we are providing the community with this utility that was inspired by the nda-tools but designed specifically to download data from Collection 3165.

Upgrade Steps

• Make Data Package of Collection 3165 on NDA website w/ associated data files
• Make Data Package of Collection 3165 on NDA website w/o associated data files to download the datastructure_manifest.txt
• Clone repository and follow documentation

New Features

• Store NDA credentials with keyring
• NDA API integration
• Query before you download

Bug Fixes

• Replace decremented token generator

Performance Improvements

• Parallel batch download of presigned urls