Skip to content

Commit

Permalink
Hashing implemented & working
Browse files Browse the repository at this point in the history
  • Loading branch information
charles-turner-1 committed Jan 28, 2025
1 parent a874db8 commit 6c3d0e4
Showing 1 changed file with 15 additions and 5 deletions.
20 changes: 15 additions & 5 deletions src/access_nri_intake/experiment/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -229,10 +229,18 @@ def verify_ds_current(
)
return False

Check warning on line 230 in src/access_nri_intake/experiment/main.py

View check run for this annotation

Codecov / codecov/patch

src/access_nri_intake/experiment/main.py#L230

Added line #L230 was not covered by tests

warnings.warn(
"*** I haven't checked the hashes! Otherwise looks good bro ***",
category=DataStoreWarning,
)
expdir_manifest = Manifest("_")
expdir_manifest.add(experiment_files, hashfn="binhash")

Check warning on line 233 in src/access_nri_intake/experiment/main.py

View check run for this annotation

Codecov / codecov/patch

src/access_nri_intake/experiment/main.py#L232-L233

Added lines #L232 - L233 were not covered by tests

if not expdir_manifest.equals(mf):
warnings.warn(

Check warning on line 236 in src/access_nri_intake/experiment/main.py

View check run for this annotation

Codecov / codecov/patch

src/access_nri_intake/experiment/main.py#L235-L236

Added lines #L235 - L236 were not covered by tests
f"{Fore.YELLOW}Experiment directory and datastore do not match. Regenerating datastore...{Style.RESET_ALL}",
category=DataStoreWarning,
stacklevel=2,
)
return False

Check warning on line 241 in src/access_nri_intake/experiment/main.py

View check run for this annotation

Codecov / codecov/patch

src/access_nri_intake/experiment/main.py#L241

Added line #L241 was not covered by tests

print(f"{Fore.GREEN}Datastore integrity verified!{Style.RESET_ALL}")
return True

Check warning on line 244 in src/access_nri_intake/experiment/main.py

View check run for this annotation

Codecov / codecov/patch

src/access_nri_intake/experiment/main.py#L243-L244

Added lines #L243 - L244 were not covered by tests


Expand All @@ -244,10 +252,12 @@ def hash_catalog(
.$datastore_name.hash file in the catalog_dir. This will be used to check if the datastore
is current.
"""
cat_files = builder_instance.df.path.tolist()
cat_fullfiles = [str(Path(file).resolve()) for file in cat_files]

Check warning on line 256 in src/access_nri_intake/experiment/main.py

View check run for this annotation

Codecov / codecov/patch

src/access_nri_intake/experiment/main.py#L255-L256

Added lines #L255 - L256 were not covered by tests

mf = Manifest(str(catalog_dir / f".{datastore_name}.hash"))

Check warning on line 258 in src/access_nri_intake/experiment/main.py

View check run for this annotation

Codecov / codecov/patch

src/access_nri_intake/experiment/main.py#L258

Added line #L258 was not covered by tests

mf.add(builder_instance.df.path.tolist(), hashfn="binhash")
mf.add(cat_fullfiles, hashfn="binhash")

Check warning on line 260 in src/access_nri_intake/experiment/main.py

View check run for this annotation

Codecov / codecov/patch

src/access_nri_intake/experiment/main.py#L260

Added line #L260 was not covered by tests

mf.dump()
return None

Check warning on line 263 in src/access_nri_intake/experiment/main.py

View check run for this annotation

Codecov / codecov/patch

src/access_nri_intake/experiment/main.py#L262-L263

Added lines #L262 - L263 were not covered by tests

0 comments on commit 6c3d0e4

Please sign in to comment.