Spatial join area #1713

Jonathan24680 · 2025-01-17T09:57:40Z

This merge request adds the possibility to use arbitrary WKT Geometries (i.e. areas) in SpatialJoin queries. Right now, areas can get approximated as the center point or the true distance between them is estimated. In addition to that a future PR will add pre computed bounding boxes to the index generation to enable even faster query times (like it already exists for geopoints)

codecov · 2025-01-21T09:20:13Z

Codecov Report

Attention: Patch coverage is 94.32990% with 11 lines in your changes missing coverage. Please review.

Project coverage is 90.06%. Comparing base (d7a70c7) to head (e42d9d0).
Report is 12 commits behind head on master.

Files with missing lines	Patch %	Lines
src/engine/SpatialJoinAlgorithms.cpp	93.71%	5 Missing and 6 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1713      +/-   ##
==========================================
+ Coverage   90.03%   90.06%   +0.02%     
==========================================
  Files         395      396       +1     
  Lines       37904    38062     +158     
  Branches     4263     4277      +14     
==========================================
+ Hits        34128    34280     +152     
- Misses       2479     2486       +7     
+ Partials     1297     1296       -1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

…ints

joka921

An initial general review of everything but the tests.
When you have addressed the initial principle suggestions I will have a closer look

src/engine/SpatialJoinAlgorithms.h

src/engine/SpatialJoinAlgorithms.cpp

joka921 · 2025-01-26T12:22:09Z

src/engine/SpatialJoinAlgorithms.cpp

+    } catch (...) {
+      return std::nullopt;
+    }


does boost geometry have a mode or overload of readWkt that doesn't use exceptions? then use that.

No. At least I'm not aware of such a function or overload and could not find one.

src/engine/SpatialJoinAlgorithms.cpp

joka921

A pass on SpatialJoinAlgoirthms.cpp (where most of the code lies anyway) :)

src/engine/SpatialJoinAlgorithms.cpp

joka921 · 2025-02-03T19:41:31Z

src/engine/SpatialJoinAlgorithms.cpp

+// ____________________________________________________________________________
+double SpatialJoinAlgorithms::convertDegreesToMeters(
+    AnyGeometry geometry1, AnyGeometry geometry2) const {
+  return boost::apply_visitor(DistanceVisitor(), geometry1, geometry2) * 78.630;


What exactly is the multiplication doing here, the magic constant should be explained.
Is the function name actually computeDistanceInMeters (you first compute it, and then do some magic multiplication to convert)

a very verbose explanation of this is in the header file. But you're right, the function should be called different, as it first gets the distance in degrees and then converts it. I called it computeDist, since that's what the other compute distance functions are called. Then it's consistent and depending on the parameters available, one can call one or the other computeDist function

Thank you for the clarification, I am much less confused now:)

src/engine/SpatialJoinAlgorithms.cpp

joka921 · 2025-02-03T19:45:53Z

src/engine/SpatialJoinAlgorithms.cpp

+      AnyGeometry geom1 = geo1.geometry.value();
+      AnyGeometry geom2 = geo2.geometry.value();
+      if (geo1.geoPoint.has_value()) {
+        geom1 = convertGeoPointToPoint(geo1.geoPoint.value());
+      } else if (geo2.geoPoint.has_value()) {


What is stored e.g. in geo1.geometry.value() if geo1 points to a POINT already? This last part looks a bit confusing.

one line above, there is a comment, which states "one point and one area". So this is the case, where we have one GeoPoint and one area. Then geom1 and geom2 get initialized to be the geometries (which is wrong in one case. When the geometry is a GeoPoint, then geomX.geometry is std::nullopt). The if below checks, which of those two is not a geometry but a geopoint. Then it converts geopoint to boost point (as boost requires both types to be boost geometries). You're right, that they should not be initialized with .value, since that is std::nullopt for at least one of them. I still have to write a test, which checks that, but it should work now

joka921 · 2025-02-03T19:48:24Z

src/engine/SpatialJoinAlgorithms.cpp

-        boost::geometry::read_wkt(getAreaString(idtable, row, col), geometry);
-      } else {
-        geometry = Point(point.value().getLng(), point.value().getLat());
+


Easier control flow:
if (point.has_value) { return point; } else { return getAnyGeometry(...)} Everything else then is not nested in the if`.

'(with my suggestion for the AnyGeometry)

i don't understand what you mean

if (point.has_value()) { return std::optional{AnyGeometry{convertPoint...()}}; } return std::optional{getAnyGeometry(...);

You can drop the std::optional if you explicitly set the return type of the lambda to std::optional<AnyGeometry>

Of course only relevant, if we still have this code at all.

src/engine/SpatialJoinAlgorithms.cpp

joka921

Another round of reviews.

src/engine/SpatialJoinAlgorithms.cpp

joka921 · 2025-02-06T08:12:25Z

src/engine/SpatialJoinAlgorithms.cpp

+// ____________________________________________________________________________
+double SpatialJoinAlgorithms::convertDegreesToMeters(
+    AnyGeometry geometry1, AnyGeometry geometry2) const {
+  return boost::apply_visitor(DistanceVisitor(), geometry1, geometry2) * 78.630;


Thank you for the clarification, I am much less confused now:)

src/engine/SpatialJoinAlgorithms.cpp

joka921 · 2025-02-06T08:28:20Z

src/engine/SpatialJoinAlgorithms.cpp

-        boost::geometry::read_wkt(getAreaString(idtable, row, col), geometry);
-      } else {
-        geometry = Point(point.value().getLng(), point.value().getLat());
+


if (point.has_value()) { return std::optional{AnyGeometry{convertPoint...()}}; } return std::optional{getAnyGeometry(...);

You can drop the std::optional if you explicitly set the return type of the lambda to std::optional<AnyGeometry>

src/engine/SpatialJoinAlgorithms.cpp

src/engine/SpatialJoinAlgorithms.h

joka921

Another pass

src/engine/SpatialJoinAlgorithms.cpp

joka921 · 2025-02-10T10:15:03Z

src/engine/SpatialJoinAlgorithms.cpp

+    const auto& areaBox = bbox.value();
    Point p = calculateMidpointOfBox(areaBox);
    return GeoPoint(p.get<1>(), p.get<0>());


I think below the code gets much more readable if you do
if (!useMidpoint) { return computeDist(index1, index2);}; //.
Then it becomes much clearer that the whole complexity is just for the midpoint case (is this actually useful still?)
If you want to keep the midpoint algorithm then you can move the code into a separate function computeDistUsingMidpoints and then the actual computeDist becomes very short and readable.

i don't get what you mean. The if else code blocks in the current implementation are equally large and complex for both cases.

src/engine/SpatialJoinAlgorithms.cpp

src/engine/SpatialJoinAlgorithms.h

sparql-conformance · 2025-02-12T22:07:47Z

Conformance check passed ✅

No test result changes.

Details: https://qlever.cs.uni-freiburg.de/sparql-conformance-ui?cur=e42d9d040bda33a56968b7b2e7fe5067af4f5b2b&prev=72973057897738fb8e5943f534791e7cbe4ebf5c

sonarqubecloud · 2025-02-13T00:44:18Z

Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

Jonathan24680 added 16 commits December 18, 2024 00:01

untested changes

c7a1ea8

continued development

a266201

added all areas to the dataset and some tests

2cc8504

make file_buffer_size in the test index be a parameter

d784513

bugfix area dataset

f6e1647

backup commit

1a4ff8f

area test with points and areas working

e3a2644

moved some stuff from dev space into the project

f38cb1d

new tests

4d71c4e

first test works now for the areas as well

38c9585

all tests are now executed and passed

ddea6d3

clang format

1902d44

merge master

31e7054

backup with removed filebuffersize compiling but not testing

cba4324

make buffer size of test index variable

d5c934c

clean up code

c1dc963

Jonathan24680 requested a review from joka921 January 17, 2025 10:00

Jonathan24680 added 2 commits January 20, 2025 19:25

sonarqube

c5284ef

correct bug for function without inline in header file

fc1a1fe

arbitrary geometry types and areas dont need to be approximated as po…

ec75e89

…ints

joka921 requested changes Jan 26, 2025

View reviewed changes

Jonathan24680 added 5 commits January 31, 2025 16:27

PR feedback

4fd698c

PR feedback

173e646

backup

7dd8156

PR feedback

1c60bf9

spelling mistake

3b8ea1a

joka921 requested changes Feb 3, 2025

View reviewed changes

Jonathan24680 added 2 commits February 4, 2025 08:57

Sonarqube issues

a121ef2

PR changes

1168f32

Jonathan24680 added 5 commits February 4, 2025 10:39

PR changes

ca7f08e

PR changes

9d2956e

PR changes

14032bd

Sonarqube issues

29171b5

format check

457f8d4

joka921 requested changes Feb 6, 2025

View reviewed changes

Jonathan24680 added 9 commits February 7, 2025 01:03

backup

bf886c4

strange bug

10bdf7b

solved strange bug

5f12f22

PR feedback

978b512

Sonarqube issue

4132701

Merge branch 'master' into SpatialJoinArea

d711586

consistent naming of QueryBox and BoundingBox

f3c5b01

add test for distance between GeoPoints and Areas

a46fccd

improve true area distance calculation

2cc0fef

joka921 requested changes Feb 10, 2025

View reviewed changes

Jonathan24680 added 6 commits February 10, 2025 22:09

backup

96a903a

backup

47ba454

PR changes

61c6b98

clang format

c0bf282

Sonarqube issus and bugfix

988ba8d

format check

e42d9d0

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Spatial join area #1713

Spatial join area #1713

Jonathan24680 commented Jan 17, 2025 •

edited

Loading

codecov bot commented Jan 21, 2025 •

edited

Loading

joka921 left a comment

joka921 Jan 26, 2025

Jonathan24680 Feb 1, 2025

joka921 left a comment

joka921 Feb 3, 2025

Jonathan24680 Feb 4, 2025

joka921 Feb 6, 2025

joka921 Feb 3, 2025

Jonathan24680 Feb 4, 2025 •

edited

Loading

joka921 Feb 3, 2025

joka921 Feb 3, 2025

Jonathan24680 Feb 4, 2025

joka921 Feb 6, 2025

joka921 Feb 6, 2025

joka921 left a comment

joka921 Feb 6, 2025

joka921 Feb 6, 2025

joka921 left a comment

joka921 Feb 10, 2025

Jonathan24680 Feb 10, 2025

sparql-conformance bot commented Feb 12, 2025

sonarqubecloud bot commented Feb 13, 2025

Spatial join area #1713

Are you sure you want to change the base?

Spatial join area #1713

Conversation

Jonathan24680 commented Jan 17, 2025 • edited Loading

codecov bot commented Jan 21, 2025 • edited Loading

Codecov Report

joka921 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joka921 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Jonathan24680 Feb 4, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joka921 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joka921 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sparql-conformance bot commented Feb 12, 2025

Conformance check passed ✅

sonarqubecloud bot commented Feb 13, 2025

Quality Gate passed

Jonathan24680 commented Jan 17, 2025 •

edited

Loading

codecov bot commented Jan 21, 2025 •

edited

Loading

Jonathan24680 Feb 4, 2025 •

edited

Loading