Basic waterway checks (#332)

* Standard waterway checks Signed-off-by: Taylor Smock <[email protected]> * WaterWayCheck: Add additional test for layer=-1 + no layer waterway connections Signed-off-by: Taylor Smock <[email protected]> * ElevationUtilities: Check elevations with third-party data * Currently supports HGT files (either uncompressed or in a zip archive) * Can be used to give better instructions (e.g., waterway goes up a hill) Signed-off-by: Taylor Smock <[email protected]> * WaterWayCheck: Use elevations for better instructions Signed-off-by: Taylor Smock <[email protected]> * WaterWayCheck: Check if the final node of a way is on a boundary. This does not work with my sample files, since the waterways do not have the synthetic tags, and sometimes a waterway that another waterway connects to isn't in the atlas. Signed-off-by: Taylor Smock <[email protected]> * ElevationUtilities: Support more compression formats. The additional compression formats should be useful for integrating third party data (e.g., object detections). This is from the new CompressionUtilities class. Signed-off-by: Taylor Smock <[email protected]> * FIXUP: ElevationUtilities: Sonar Signed-off-by: Taylor Smock <[email protected]> * ElevationUtilities: FIXUP sonar issues Signed-off-by: Taylor Smock <[email protected]> * CompressionUtilities: Add tests Signed-off-by: Taylor Smock <[email protected]> * ElevationUtilities: Expand tests slightly Signed-off-by: Taylor Smock <[email protected]> * WaterWayChecks: Add tests for differing elevations Signed-off-by: Taylor Smock <[email protected]> * WaterWayChecks: Add documentation Signed-off-by: Taylor Smock <[email protected]> * WaterWayChecks: Update docs to include elevation check information Signed-off-by: Taylor Smock <[email protected]> * WaterWayCheck: FIXUP: Typo (intersectinWaterways -> intersectingWaterways) Signed-off-by: Taylor Smock <[email protected]> * WaterWayCheck: Append instructions to flag, if one exists * For example, 1. The waterway <xxx> does not end in a sink (ocean/sinkhole/waterway/drain) 2. The waterway <xxx> crosses the waterway <yyy>. Signed-off-by: Taylor Smock <[email protected]> * WaterWayCheck: Add sample configuration Signed-off-by: Taylor Smock <[email protected]> * WaterWayCheck: FIXUP: Update docs to no longer indicate that a single issue ends the check Signed-off-by: Taylor Smock <[email protected]> * WaterWayCheck: FIXUP: Update docs to indicate that crossing waterways are found Signed-off-by: Taylor Smock <[email protected]> * ElevationUtilities: Expand EXT/Ext to extension (appropriately capitalized) Signed-off-by: Taylor Smock <[email protected]> * ElevationUtilities: Add constructor for when a test wants to specify parameters manually Signed-off-by: Taylor Smock <[email protected]> * WaterWayCheck: Improve documentation for a confusing code section * This mostly helps explain the coastline check. Signed-off-by: Taylor Smock <[email protected]> * WaterWayCheck: Use Atlas.lineItemsContaining instead of Atlas.lineItemsIntersecting Signed-off-by: Taylor Smock <[email protected]> * WaterWayCheck Configuration: Add challenge section Signed-off-by: Taylor Smock <[email protected]> * ElevationUtilities: Add test for constructors * Also, modify so that the settings use the same casing as other configuration settings Signed-off-by: Taylor Smock <[email protected]> * WaterWayCheck: Add additional information when a waterway crosses a coast but does NOT end in the ocean Signed-off-by: Taylor Smock <[email protected]> * WaterWayCheck: Make configuration prettier Signed-off-by: Taylor Smock <[email protected]> * WaterWayCheck, ElevationUtilities: Avoid single character variables Signed-off-by: Taylor Smock <[email protected]> * WaterWayCheck: Spacing Signed-off-by: Taylor Smock <[email protected]> * WaterWayCheck: Get atlas in main check instead of submethod Signed-off-by: Taylor Smock <[email protected]> * WaterWayCheck: Add more documentation Signed-off-by: Taylor Smock <[email protected]> * CommonTagFilters: Move some filters OceanBleedingCheck and WaterWayCheck into a separate file for reuse Signed-off-by: Taylor Smock <[email protected]> * WaterWayCheck: Formatting Signed-off-by: Taylor Smock <[email protected]> * WaterWayCheck: FIXUP: Nitpicks and some simplifications Signed-off-by: Taylor Smock <[email protected]> * WaterWayCheck: Ensure that intersecting waterways actually intersect * Originally, was just getting waterways based off of the bbox of the original waterway Signed-off-by: Taylor Smock <[email protected]> * WaterWayCheck: Move flagging logic into separate methods Signed-off-by: Taylor Smock <[email protected]> * WaterWayCheck: Add comment expounding upon incline check and resolution of data Signed-off-by: Taylor Smock <[email protected]>
osmlab · Sep 10, 2020 · 3a5c6f2 · 3a5c6f2
1 parent fe52307
commit 3a5c6f2
Show file tree

Hide file tree

Showing 15 changed files with 2,260 additions and 6 deletions.
diff --git a/config/configuration.json b/config/configuration.json
@@ -1089,5 +1089,26 @@
       "instruction": "Open your favorite editor and edit the features overlapping the ocean so they either validly overlap or do not overlap.",
       "difficulty": "MEDIUM"
     }
+  },
+  "WaterWayCheck": {
+    "ocean": {
+      "boundary": "natural->coastline",
+      "valid": "natural->strait,channel,fjord,sound,bay|harbour->*&harbour->!no|estuary->*&estuary->!no|bay->*&bay->!no|place->sea|seamark:type->harbour,harbour_basin,sea_area|water->bay,cove,harbour|waterway->artificial,dock"
+    },
+    "waterway": {
+      "elevation": {
+        "distance.min.start.end": 450.0,
+        "resolution.min.uphill": 1.0
+      },
+      "sink.tags.filters": "natural->sinkhole|waterway->tidal_channel,drain|manhole->drain",
+      "tags.filters": "waterway->river,stream,tidal_channel,canal,drain,ditch,pressurised"
+    },
+    "challenge": {
+      "description": "Waterways that have circular flows, are crossing, do not have a sink, or are going uphill (if elevation data was provided)",
+      "blurb": "Fix waterways such that they make physical sense",
+      "instruction": "Open your favorite editor and edit the waterways",
+      "difficulty": "NORMAL",
+      "defaultPriority": "LOW"
+    }
   }
 }
diff --git a/docs/available_checks.md b/docs/available_checks.md
@@ -99,3 +99,4 @@ This document is a list of tables with a description and link to documentation f
 | LineCrossingWaterBodyCheck | The purpose of this check is to identify line items (edges or lines) and optionally buildings, that cross water bodies invalidly. |
 | MalformedPolyLineCheck | This check identifies lines that have only one point, or none, and the ones that are too long. |
 | [SelfIntersectingPolylineCheck](checks/selfIntersectingPolylineCheck.md) | The purpose of this check is to identify all edges/lines/areas in Atlas that have self-intersecting polylines, or geometries that intersects itself in some form. |
+| [WaterWayCheck](checks/waterWayCheck.md) | This check finds closed waterways (circular water motion), waterways without a place for water to go (a "sink"), crossing waterways, and waterways that go uphill (requires elevation data). |
diff --git a/docs/checks/waterWayChecks.md b/docs/checks/waterWayChecks.md
@@ -0,0 +1,54 @@
+# WaterWay Checks
+
+#### Description
+This check identifies waterways that are closed (i.e., would have a circular water flow), waterways that do not have a place for the water to go (a "sink"), and waterways crossing waterways. It also looks for ways that may be going uphill (requires elevation data, see [ElevationUtilities](../elevationUtilities.md))
+
+#### Live Example
+1) This canal ([osm id: 46115760 version 5](https://www.openstreetmap.org/way/46115760)) is a closed waterway, which makes no semantic sense without a pump somewhere in the waterway. In this case, it should be tagged `natural=water` + `water=canal`.
+
+2) The waterways at ([osm id: 500672157](https://www.openstreetmap.org/node/500672157) on 2020-08-20) cross each other, when they should be connected.
+
+#### Useful configuration variables:
+```json
+{
+    "WaterWayCheck": {
+        "waterway.sink.tags.filters": "natural->sinkhole|waterway->tidal_channel,drain|manhole->drain",
+        "waterway.tags.filters": "waterway->river,stream,tidal_channel,canal,drain,ditch,pressurised",
+        "ocean.valid": "natural->strait,channel,fjord,sound,bay|harbour->*&harbour->!no|estuary->*&estuary->!no|bay->*&bay->!no|place->sea|seamark:type->harbour,harbour_basin,sea_area|water->bay,cove,harbour|waterway->artificial,dock",
+        "ocean.boundary": "natural->coastline",
+        "waterway.elevation.resolution.min.uphill": "1" (meter),
+        "waterway.elevation.resolution.min.start.end": "457.2" (meters)
+    }
+}
+```
+You may also want to look at [ElevationUtilities](../utilities/elevationUtilities.md).
+
+#### Code Review
+In [Atlas](https://github.com/osmlab/atlas), OSM elements are represented as Edges, Points, Lines,
+Nodes & Relations; in our case, we’re are looking at [LineItems](https://github.com/osmlab/atlas/blob/dev/src/main/java/org/openstreetmap/atlas/geography/atlas/items/LineItem.java), which Lines and Edges are specific subtypes of. This is due to the fact that a waterway may either be a _navigable_ way or a _non-navigable_ way (rivers are generally considered _navigable_, while streams may or may not be _navigable_).
+
+Our first goal is to validate the incoming Atlas object. Valid features for this check will satisfy
+the following conditions:
+* Must be an LineItem with one of the following tags:
+    * `waterway=RIVER`
+    * `waterway=STREAM`
+    * `waterway=TIDAL_CHANNEL`
+    * `waterway=CANAL`
+    * `waterway=DRAIN`
+    * `waterway=DITCH`
+    * `waterway=PRESSURISED`
+
+```java
+    @Override
+    public boolean validCheckForObject(final AtlasObject object)
+    {
+        return !isFlagged(object.getOsmIdentifier()) && object instanceof LineItem
+                && this.waterwayTagFilter.test(object);
+    }
+```
+
+After the preliminary filtering, each object goes through a series of `if` statements. The first checks if the line is closed. The second checks if the waterway is going uphill (requires elevation data), and if the resolution of the elevation data is good enough to determine that the waterway goes uphill, the object is flagged. At this point, we check to see if the waterway ends in a sink, for this check, we attempt ensure that the waterway ends inside the boundaries, and not in a neighboring area. Furthermore, we reuse the check for uphill ways to help improve the error message. Once all of those checks have finished, we check for waterway intersections. If more than one check flags the object, instructions and offending objects are appended to the CheckFlag.
+
+
+To learn more about the code, please look at the comments in the source code for the check.
+[WaterWayCheck.java](../../src/main/java/org/openstreetmap/atlas/checks/validation/linear/lines/WaterWayCheck.java)
diff --git a/docs/utilities/elevationUtilities.md b/docs/utilities/elevationUtilities.md
@@ -0,0 +1,16 @@
+# ElevationUtilities
+
+#### Description
+This is a general utilities class that allows checks to get elevation data. The class currently only understands HGT files, which by specification are 1 degree by 1 degree tiles. There is a [script](../../../scripts/elevationData) which can be used to get NASA SRTM elevation data (~90m accuracy throughout the world, some locations have no data).
+
+#### Configuration
+
+```json
+{
+    "ElevationUtilities": {
+        "elevation.srtm_extent": 1.0 (degree),
+        "elevation.srtm_ext": "hgt" (file extension),
+        "elevation.path": "elevation"
+    }
+}
+```
diff --git a/scripts/elevationData/requirements.txt b/scripts/elevationData/requirements.txt
@@ -0,0 +1,9 @@
+certifi>=2020.6.20
+chardet>=3.0.4
+defusedxml>=0.6.0
+idna>=2.10
+python-dateutil>=2.8.1
+requests>=2.24.0
+six>=1.15.0
+tqdm>=4.48.2
+urllib3>=1.25.10
diff --git a/scripts/elevationData/srtm-hgt.py b/scripts/elevationData/srtm-hgt.py
@@ -0,0 +1,69 @@
+#!/usr/bin/env python3
+"""
+Get SRTM (HGT) files from the Nasa SRTM mission via USGS in a server-friendly
+way.
+
+There is a pause of 10 seconds between downloads, and if a file is already
+downloaded, it is not redownloaded. This also looks at timestamps, although
+it is unlikely that the SRTM files will be significantly updated.
+"""
+from defusedxml import ElementTree as ET
+from tqdm import tqdm
+import argparse
+import dateutil.parser
+import os
+import requests
+import time
+import typing
+
+url = "https://dds.cr.usgs.gov/srtm/version2_1/SRTM3/"
+def download_file(url: str, output_dir: str="elevation"):
+    name = url.split("/")[-1]
+    with requests.get(url, stream=True) as response:
+        response.raise_for_status()
+        with open(os.path.join(output_dir, name), "wb") as fh:
+            for content in tqdm(response.iter_content(chunk_size=4096)):
+                fh.write(content)
+
+def file_needs_download(url: str, output_dir: str="elevation") -> bool:
+    name = url.split("/")[-1]
+    filepath = os.path.join(output_dir, name)
+    if os.path.isfile(filepath):
+        head = requests.head(url)
+        head.raise_for_status()
+        last_modified = dateutil.parser.parse(head.headers.get('Last-Modified')).timestamp()
+        content_length = int(head.headers.get('Content-Length'))
+
+        creation = os.path.getctime(filepath)
+        modification = os.path.getmtime(filepath)
+
+        return modification < last_modified or creation < last_modified or content_length != os.path.getsize(filepath)
+    return True
+
+def get_sub_url(url: str) -> typing.List[str]:
+    response = requests.get(url)
+    sub_url = []
+    for line in response.iter_lines():
+        try:
+            tree = ET.fromstring(line)
+            for child in tree.findall("a"):
+                sub_url.append(child.attrib.get("href"))
+        except ET.ParseError as e:
+            pass
+    return sub_url
+
+def download_url(url: str, output_dir: str="elevation") -> None:
+    for sub_url in tqdm(get_sub_url(url)):
+        for sub_sub_url in tqdm(get_sub_url(url + sub_url)):
+            if file_needs_download(url + sub_url + sub_sub_url, output_dir = output_dir):
+                tqdm.write("Downloading " + url + sub_url + sub_sub_url)
+                download_file(url + sub_url + sub_sub_url, output_dir = output_dir)
+                time.sleep(10)
+            else:
+                tqdm.write(url + sub_url + sub_sub_url + " not downloaded")
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(description="Download SRTM files")
+    parser.add_argument("directory", nargs='?', default="elevation", help="The directory to save files to")
+    args = parser.parse_args()
+    download_url(url, output_dir = args.directory)
diff --git a/src/main/java/org/openstreetmap/atlas/checks/utility/CommonTagFilters.java b/src/main/java/org/openstreetmap/atlas/checks/utility/CommonTagFilters.java
@@ -0,0 +1,20 @@
+package org.openstreetmap.atlas.checks.utility;
+
+/**
+ * Hold common tag filters (should be used in more than one check)
+ * 
+ * @author Taylor Smock
+ */
+public final class CommonTagFilters
+{
+    private CommonTagFilters()
+    {
+        // Hide constructor
+    }
+
+    /** Boundary filter for ocean boundaries */
+    public static final String DEFAULT_OCEAN_BOUNDARY_TAGS = "natural->coastline";
+    /** Tag filter for oceans (without coastline) */
+    public static final String DEFAULT_VALID_OCEAN_TAGS = "natural->strait,channel,fjord,sound,bay|"
+            + "harbour->*&harbour->!no|estuary->*&estuary->!no|bay->*&bay->!no|place->sea|seamark:type->harbour,harbour_basin,sea_area|water->bay,cove,harbour|waterway->artificial,dock";
+}
diff --git a/src/main/java/org/openstreetmap/atlas/checks/utility/CompressionUtilities.java b/src/main/java/org/openstreetmap/atlas/checks/utility/CompressionUtilities.java
@@ -0,0 +1,116 @@
+package org.openstreetmap.atlas.checks.utility;
+
+import java.io.BufferedInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+
+import org.apache.commons.compress.archivers.ArchiveException;
+import org.apache.commons.compress.archivers.ArchiveInputStream;
+import org.apache.commons.compress.archivers.ArchiveStreamFactory;
+import org.apache.commons.compress.compressors.CompressorException;
+import org.apache.commons.compress.compressors.CompressorStreamFactory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Utilities that are useful for compressed or archived streams.
+ *
+ * @author Taylor Smock
+ */
+public final class CompressionUtilities
+{
+    private static final Logger logger = LoggerFactory.getLogger(CompressionUtilities.class);
+
+    /**
+     * Get an uncompressed and unarchived input stream
+     *
+     * @param inputStream
+     *            The inputstream to make unarchived/uncompressed
+     * @return A usuable inputstream (if unarchived, it will be at the first entry)
+     * @throws IOException
+     *             If the inputstream cannot be read
+     */
+    public static InputStream getUncompressedInputStream(final InputStream inputStream)
+            throws IOException
+    {
+        final BufferedInputStream bufferedInput = new BufferedInputStream(inputStream);
+        try
+        {
+            return decompressedInputStream(bufferedInput);
+        }
+        catch (final CompressorException | IOException e)
+        {
+            // OK. Not compressed.
+        }
+        try
+        {
+            return unarchivedInputStream(bufferedInput);
+        }
+        catch (final ArchiveException | IOException e)
+        {
+            // OK. Not archived or compressed. Just return it.
+            return bufferedInput;
+        }
+    }
+
+    /**
+     * Decompress an inputstream
+     *
+     * @param inputStream
+     *            The inputstream to decompress
+     * @return The decompressed (and potentially unarchived) inputstream. If unarchived, the
+     *         position will be at the first entry.
+     * @throws IOException
+     *             If there is a problem reading the stream
+     * @throws CompressorException
+     *             If there is a problem decompressing, or if it is not a compressed file
+     */
+    private static InputStream decompressedInputStream(final InputStream inputStream)
+            throws IOException, CompressorException
+    {
+        final InputStream uncompressed = new CompressorStreamFactory()
+                .createCompressorInputStream(inputStream);
+        final BufferedInputStream buffered = new BufferedInputStream(uncompressed);
+        try
+        {
+            return unarchivedInputStream(buffered);
+        }
+        catch (final ArchiveException | IOException e)
+        {
+            // OK. Probably not archived.
+        }
+        return buffered;
+    }
+
+    /**
+     * Try to unarchive an inputstream
+     *
+     * @param inputStream
+     *            The inputstream to unarchive
+     * @return The unarchived input stream
+     * @throws IOException
+     *             If there is an IOException
+     * @throws ArchiveException
+     *             If there is a problem with the archive (or it isn't one)
+     */
+    private static InputStream unarchivedInputStream(final InputStream inputStream)
+            throws IOException, ArchiveException
+    {
+        final ArchiveInputStream toRead = new ArchiveStreamFactory()
+                .createArchiveInputStream(inputStream);
+        try
+        {
+            toRead.getNextEntry();
+        }
+        catch (final IOException e)
+        {
+            logger.trace(e.getLocalizedMessage());
+        }
+        return toRead;
+    }
+
+    private CompressionUtilities()
+    {
+        // Hide constructor
+    }
+}