Reservoir Splitting (NOAA-OWP#38)

* configure initial set of template files with basic information * fix package list format in README.md * Update README.md * Update README.md * Delete bluencyan.gif Need to create non-proprietary version using, possibly, these packages: pip install imageio pip install imageio-ffmpeg pip install pygifsicle * initial commit to version 0.5 * Update README.md This will need further updating once in the OWP repository... * remove references to former location * switch non-license text files to lower case * incorporate notebooks readme into main readme * clean up Notebook will eventually go into notebooks folder * update readme about the source of these datafiles. * update readme about the source of these datafiles. * add masks (previously were omitted by .zip in .gitignore) * add traceback to execption print * added license-free gif image * add image and update urls * replace readme.txt with markdown file * remove all local and fork references. * expand instructions for contributing * minor update to readme.md * Remove WPC initialization * Remove WPC initialization * Remove WPC initialization also update naming of files, ensure execution is consistent. * convert to CSV write * backout commit with depthp loop * add helpful error if module not compiled * add waterbody parsing Several steps to go still * add waterbody parsing Several steps to go still * Fix parameter order for get_up_connections * Expand compose_reach call for readability * change compose_reaches to compose_networks * change compose_reaches to compose_networks * start integrating waterbodies * Add reach processing back into recursive call * ADD WPC to argument list python mc_sseg_stime_NOLOOP.py gives questionable results with this commit -- NEED TO CHECK * Fix WPC variable order * Change set.remove to set.discard Suggested by @groutr to avoid exception for missing input * Change set.remove to set.discard Suggested by @groutr to avoid exception for missing input * Split Networks at Water bodies This commit works with the Pocono test case and the Mainstems, but still breaks on the NHD CONUS Full resolution case. Debugging continues. * Fix condition spacing * Add files to test re-ordered recursive reach loop * Remove duplicates Junction counts and print statements were being duplicated A TODO: identifies an opportunity to further streamline * Clarify comment about broken networks * correct minor comment * Add files to test parallel scaling * reduce test case to only Pocono * Add Waterbodies to test dataset in Networkbuilder * Add source comment for full dataset pandas option * Switch reach test to debuglevel -1 These files still do not properly execute the CONUS NWM input when the Waterbody parsing is activated .... * incorporated reorganized recursive function The reorganized function allows simpler maintenance. The other two functions have been retained but are unreferenced in the source. At some point they can be removed. * removed now-extraneous files the reorganized functions are now incorporated into the primary function and are renamed recursive_reach_reach_new. * Remove terminal node from waterbody upstreams The internal argument was never being passed, but until a recent change, the default value happened to work. This commit makes sure we are not depending on the default value and also adds the explicit removal. * Remove terminal node from waterbody upstreams The internal argument was never being passed, but until a recent change, the default value happened to work. This commit makes sure we are not depending on the default value and also adds the explicit removal. * Do not expect header for csv. Also reorganize debug print statements * Fix crash in opportunistic parallelization The Opportunistic parallelization example was crashing when hitting the bottom of and interior network, since these are not defined by a outlet to the terminal_code. The code was changed to simply look for the order=0 reach, which was mostly sufficient, but as noted in a new #TODO, still does not produce exactly the result expected for all cases * Update readme etc. to v0.5.0 * Black parallel SingleSeg files * add argparse to SingleSeg tests * Change names to clearly identify demo files `python src/fortran_routing/mc_pylink_v00/MC_singleSeg_singleTS/mc_srch_stime_demo.py` and `src/fortran_routing/mc_pylink_v00/MC_singleSeg_singleTS/mc_sseg_stime_NOLOOP_demo.py` demonstrate the reach-looped version (original) and the refactored single-segment versions of the muskingum-cunge algorithm, respectively. * Fix Pocono_TEST2 Converted to a mask to fix the broken dependency (was pointing to a local file) * Fix short arg strings and blacken Short and long args were improperly quoted -- fixed that Also blackened all source files in this commit. * Verbosify Step 1 * Quote choices for debuglevel Former-commit-id: e78c526
jameshalgren · May 26, 2020 · 49f1d31 · 49f1d31
1 parent dd53582
commit 49f1d31
Show file tree

Hide file tree

Showing 13 changed files with 2,256 additions and 722 deletions.
diff --git a/changelog.md b/changelog.md
@@ -1,6 +1,11 @@
 All notable changes to this project will be documented in this file.
 We follow the [Semantic Versioning 2.0.0](http://semver.org/) format.
 
+## 0.5.0 - 2020-05-21 
+
+###Added
+
+Added capability to distiguish reservoirs within the network (with an embedded capability to split at any arbitrary point in the network, if desired)
 
 ## 0.4.0 - 2020-04-08 
 

diff --git a/readme.md b/readme.md
@@ -80,7 +80,7 @@ If you have any questions, please contact [email protected] or dongha.kim@n
 
 ## Getting involved
 
-Our current focus is improving the speed of the parallel tree traversal. We welcome your thoughts, recommendations, comments, and of course, PRs. 
+Among other things, we are working on preparing more robust I/O capability for testing and on improving the speed of the parallel tree traversal. We welcome your thoughts, recommendations, comments, and of course, PRs. 
 
 Please feel free to fork the repository and let us know if you will be issuing a pull request. 
 More instructions will eventually be documented in [CONTRIBUTING](contributing.md).

diff --git a/security.md b/security.md
@@ -8,11 +8,12 @@ The public repositories MUST NOT leak any valid private credential (e.g., a work
 
 ## Supported Versions
 
-This product is being made available as a 0.4 version and is in active development at that version.
+This product is being made available as a 0.5 version and is in active development at that version.
 
 | Version | Supported          |
 | ------- | ------------------ |
-| 0.4   | :white_check_mark: |
+| 0.5   | :white_check_mark: |
+| 0.4   | __X__ |
 
 ## Reporting a Vulnerability
 

diff --git a/...00/MC_singleSeg_singleTS/mc_sseg_stime.py → ..._singleSeg_singleTS/mc_srch_stime_demo.py b/...00/MC_singleSeg_singleTS/mc_sseg_stime.py → ..._singleSeg_singleTS/mc_srch_stime_demo.py
diff --git a/...ingleSeg_singleTS/mc_sseg_stime_NOLOOP.py → ...Seg_singleTS/mc_sseg_stime_NOLOOP_demo.py b/...ingleSeg_singleTS/mc_sseg_stime_NOLOOP.py → ...Seg_singleTS/mc_sseg_stime_NOLOOP_demo.py
diff --git a/src/python_framework/networkbuilder.py b/src/python_framework/networkbuilder.py
@@ -34,9 +34,9 @@ def get_down_connections(
 
 def get_waterbody_segments(
     connections = None
-    , terminal_code = 0
-    , waterbody_col = 9
-    , waterbody_null_code = -9999
+    , terminal_code = -999
+    , waterbody_col = 3
+    , waterbody_null_code = 0
     , data_key = r'data'
     , downstream_key = r'downstream'
     , upstreams_key = r'upstreams'
@@ -54,7 +54,6 @@ def get_waterbody_segments(
     if verbose: print('waterbody segments ...')
     waterbody_segments = {key:con[data_key][waterbody_col] for key, con in connections.items() 
         if not (con[data_key][waterbody_col] == waterbody_null_code)}
-    #waterbody_dict = {
     if debuglevel <= -1: print(f'found {len(waterbody_segments)} segments that are part of a waterbody')
     if debuglevel <= -3: print(waterbody_segments)
     if verbose: print('waterbody_segments complete')
@@ -74,6 +73,7 @@ def get_waterbody_segments(
         for upstream in connections[waterbody_segment][upstreams_key]:
             if not upstream == terminal_code and not upstream in waterbody_segments:
                 waterbody_upstreams_set.add(upstream)
+    waterbody_upstreams_set.discard(terminal_code) #TODO: Is this the best place for this filter -- check if ever used.
     if debuglevel <= -1: print(f'found {len(waterbody_upstreams_set)} segments that are upstream of a waterbody')
     if debuglevel <= -3: print(waterbody_upstreams_set)
     if verbose: print('waterbody_upstreams_set complete')
@@ -198,8 +198,10 @@ def get_up_connections(connections
                     # spanned the gap and the headwater is not actually not a terminating node.
                     # In that case, reset the node to be a blank list (or set, if using 
                     # that method), then proceed downstream.
-                    # TODO: THIS IS A DANGEROUS STEP AND DESERVES ADDITIONAL REVIEW
+                    # TODO: THIS IS A DANGEROUS/FRAGILE STEP AND DESERVES ADDITIONAL REVIEW
                     # TODO: TO MAKE SURE IT IS DOING WHAT WE INTEND AS DESCRIBED ABOVE
+                    # TODO: RESERVOIRS: For instance, this will probably break for subnetworks containing reservoirs
+
                     connections[dkey].update({upstreams_key : set()})
 
                 connections[dkey][upstreams_key].add(ukey)
@@ -273,12 +275,12 @@ def main():
         [13,240,12,0],
         [14,548,13,0],
         [15,920,14,0],
-        [16,920,15,0],
-        [17,514,16,0],
+        [16,920,15,401],
+        [17,514,16,401],
         [18,458,17,0],
         [19,832,18,0],
         [20,543,19,0],
-        [21,240,16,0],
+        [21,240,16,401],
         [22,548,21,0],
         [23,920,22,0],
         [24,240,23,0],
@@ -292,6 +294,8 @@ def main():
     test_downstream_col = 2
     test_length_col = 1
     test_terminal_code = -999
+    test_waterbody_col = 3
+    test_waterbody_null_code = 0
 
     (test_connections) = get_down_connections(
                 rows = test_rows
@@ -308,14 +312,17 @@ def main():
      , test_terminal_ref_keys
      , test_circular_keys) = determine_keys(
                 connections = test_connections
-                , key_col = test_key_col
-                , downstream_col = test_downstream_col
                 , terminal_code = test_terminal_code
                 , verbose = True
                 , debuglevel = -2
                 )
 
-    test_junction_keys = get_up_connections(
+    (test_junction_keys 
+        , test_confluence_segment_set
+        , test_visited_keys
+        , test_visited_terminal_keys
+        , test_junction_count
+        )= get_up_connections(
                 connections = test_connections
                 , terminal_code = test_terminal_code
                 , headwater_keys = test_headwater_keys
@@ -324,6 +331,20 @@ def main():
                 , debuglevel = -2
                 )
 
+    # TODO: Set/pass/identify a proper flag value
+    if test_waterbody_col is not None:
+        (test_waterbody_set
+        , test_waterbody_segments
+        , test_waterbody_outlet_set
+        , test_waterbody_upstreams_set) = get_waterbody_segments(
+            connections = test_connections
+            , terminal_code = test_terminal_code
+            , waterbody_col = test_waterbody_col
+            , waterbody_null_code = test_waterbody_null_code
+            , verbose = True
+            , debuglevel = -2
+            )
+
     recursive_print.print_connections(
                 headwater_keys = test_headwater_keys
                 , down_connections = test_connections

diff --git a/src/python_framework/nhd_network_traversal.py b/src/python_framework/nhd_network_traversal.py
@@ -9,27 +9,54 @@
 import os
 import argparse
 
+
 def _handle_args():
-    parser = argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultsHelpFormatter)
-    parser.add_argument('--debuglevel',
-                        help='Set the debuglevel',
-                        dest='debuglevel',
-                        default=0)
-    parser.add_argument('-v, --verbose',
-                        help='Verbose output (leave blank for quiet output)',
-                        dest='verbose',
-                        action='store_true')
-    parser.add_argument('-t, --showtiming',
-                        help='Set the showtiming - leave blank for False',
-                        dest='showtiming',
-                        action='store_true')
-    parser.add_argument('-n, --supernetwork',
-                        help='List of supernetworks (Pocono_TEST1, Pocono_TEST2, LowerColorado_Conchos_FULL_RES, Brazos_LowerColorado_ge5, Brazos_LowerColorado_FULL_RES, Brazos_LowerColorado_Named_Streams, CONUS_ge5, Mainstems_CONUS, CONUS_Named_Streams, CONUS_FULL_RES_v20', 
-                        # TODO: accept multiple or a Path (argparse Action perhaps)
-                        # action='append',
-                        # nargs=1,
-                        dest='supernetwork',
-                        default='Pocono_TEST1')
+    parser = argparse.ArgumentParser(
+        formatter_class=argparse.ArgumentDefaultsHelpFormatter
+    )
+    parser.add_argument(
+        "--debuglevel",
+        help="Set the debuglevel",
+        dest="debuglevel",
+        choices=["0", "1", "2", "3"],
+        default=0,
+    )
+    parser.add_argument(
+        "-v",
+        "--verbose",
+        help="Verbose output (leave blank for quiet output)",
+        dest="verbose",
+        action="store_true",
+    )
+    parser.add_argument(
+        "-t",
+        "--showtiming",
+        help="Set the showtiming (leave blank for no timing information)",
+        dest="showtiming",
+        action="store_true",
+    )
+    parser.add_argument(
+        "-n",
+        "--supernetwork",
+        help="Choose from among the pre-programmed supernetworks (Pocono_TEST1, Pocono_TEST2, LowerColorado_Conchos_FULL_RES, Brazos_LowerColorado_ge5, Brazos_LowerColorado_FULL_RES, Brazos_LowerColorado_Named_Streams, CONUS_ge5, Mainstems_CONUS, CONUS_Named_Streams, CONUS_FULL_RES_v20",
+        choices=[
+            "Pocono_TEST1",
+            "Pocono_TEST2",
+            "LowerColorado_Conchos_FULL_RES",
+            "Brazos_LowerColorado_ge5",
+            "Brazos_LowerColorado_FULL_RES",
+            "Brazos_LowerColorado_Named_Streams",
+            "CONUS_ge5",
+            "Mainstems_CONUS",
+            "CONUS_Named_Streams",
+            "CONUS_FULL_RES_v20",
+        ],
+        # TODO: accept multiple or a Path (argparse Action perhaps)
+        # action='append',
+        # nargs=1,
+        dest="supernetwork",
+        default="Pocono_TEST1",
+    )
 
     return parser.parse_args()
 
@@ -40,70 +67,75 @@ def main():
 
     # find the path of the test scripts, several levels above the script path
     root = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
-    test_folder = os.path.join(root, r'test')
-    
+    test_folder = os.path.join(root, r"test")
+
     debuglevel = -1 * int(args.debuglevel)
     verbose = args.verbose
     showtiming = args.showtiming
-    supernetworks = {str(args.supernetwork):{}}
+    supernetworks = {str(args.supernetwork): {}}
 
     # supernetworks = {}
     # supernetworks.update({'Pocono_TEST1':{}})
-    # supernetworks.update({'LowerColorado_Conchos_FULL_RES':{}}) 
+    # supernetworks.update({'LowerColorado_Conchos_FULL_RES':{}})
     # supernetworks.update({'Brazos_LowerColorado_ge5':{}}) ##NHD Subset (Brazos/Lower Colorado)"""
-    # supernetworks.update({'Brazos_LowerColorado_FULL_RES':{}}) 
-    # supernetworks.update({'Brazos_LowerColorado_Named_Streams':{}}) 
+    # supernetworks.update({'Brazos_LowerColorado_FULL_RES':{}})
+    # supernetworks.update({'Brazos_LowerColorado_Named_Streams':{}})
     # supernetworks.update({'CONUS_ge5':{}}) ##NHD CONUS order 5 and greater"""
     # supernetworks.update({'Mainstems_CONUS':{}})
     # supernetworks.update({'CONUS_Named_Streams':{}})
     # supernetworks.update({'CONUS_FULL_RES_v20':{}}) # = False
 
     for supernetwork in supernetworks:
         supernetworks[supernetwork] = nnu.set_supernetwork_data(
-          supernetwork = supernetwork
-            , geo_input_folder = os.path.join(test_folder, r'input', r'geo')
-            , debuglevel = debuglevel
-            , verbose = verbose
+            supernetwork=supernetwork,
+            geo_input_folder=os.path.join(test_folder, r"input", r"geo"),
+            debuglevel=debuglevel,
+            verbose=verbose,
         )
-        if debuglevel <= -1: 
-            if verbose: print(f'\n\nSupernetwork:')
-            print(f'{supernetwork}')
-        if debuglevel <= -2: 
-            if verbose: print(r'All items in the above supernetwork:')
-            for k,v in supernetworks[supernetwork].items():
-                 print(f"{{'{k}' : {v}}}")
-        if showtiming: start_time = time.time()
-
-        network_out_values = \
-          nnu.get_nhd_connections(
-            supernetworks[supernetwork]
-            , debuglevel = debuglevel
-            , verbose = verbose
+        if debuglevel <= -1:
+            if verbose:
+                print(f"\n\nSupernetwork:")
+            print(f"{supernetwork}")
+        if debuglevel <= -2:
+            if verbose:
+                print(r"All items in the above supernetwork:")
+            for k, v in supernetworks[supernetwork].items():
+                print(f"{{'{k}' : {v}}}")
+        if showtiming:
+            start_time = time.time()
+
+        network_out_values = nnu.get_nhd_connections(
+            supernetworks[supernetwork], debuglevel=debuglevel, verbose=verbose
         )
 
         recursive_print.print_basic_network_info(
-          connections = network_out_values[0]
-            , headwater_keys = network_out_values[3]
-            , junction_keys = network_out_values[7]
-            , terminal_keys = network_out_values[4]
-            , terminal_code = supernetworks[supernetwork]['terminal_code']
-            , verbose = verbose
+            connections=network_out_values[0],
+            headwater_keys=network_out_values[3],
+            junction_keys=network_out_values[7],
+            terminal_keys=network_out_values[4],
+            terminal_code=supernetworks[supernetwork]["terminal_code"],
+            verbose=verbose,
         )
 
-        if debuglevel <= -3: 
-        # THE RECURSIVE PRINT IS NOT A GOOD IDEA WITH LARGE NETWORKS!!!
-        # The `Pocono_TEST1` supernetwork is a good test case to run with 
-        # the debuglevel set at -3. 
+        if debuglevel <= -3:
+            # THE RECURSIVE PRINT IS NOT A GOOD IDEA WITH LARGE NETWORKS!!!
+            # The `Pocono_TEST1` supernetwork is a good test case to run with
+            # the debuglevel set at -3.
             recursive_print.print_connections(
-                        headwater_keys = network_out_values[3]
-                        , down_connections = network_out_values[0]
-                        , up_connections = network_out_values[0]
-                        , terminal_code = supernetworks[supernetwork]['terminal_code']
-                        , terminal_keys = network_out_values[4]
-                        , terminal_ref_keys = network_out_values[5]
-                        , debuglevel = debuglevel
-                        )
-        if showtiming: print(f"Supernetwork `{supernetwork}` read and traversed\n... in %s seconds.\n\n" % (time.time() - start_time))
-
-if __name__ == '__main__':
+                headwater_keys=network_out_values[3],
+                down_connections=network_out_values[0],
+                up_connections=network_out_values[0],
+                terminal_code=supernetworks[supernetwork]["terminal_code"],
+                terminal_keys=network_out_values[4],
+                terminal_ref_keys=network_out_values[5],
+                debuglevel=debuglevel,
+            )
+        if showtiming:
+            print(
+                f"Supernetwork `{supernetwork}` read and traversed\n... in %s seconds.\n\n"
+                % (time.time() - start_time)
+            )
+
+
+if __name__ == "__main__":
     main()