Add osm-flex to CLIMADA core

[luseverin]

Changes proposed in this PR:

• Add osm-flex to the requirements
• Add section in exposure_polygons_lines tutorial to show how to obtain exposure data from OpenStreetMap using osm-flex

Comments:
Adding the osm-flex package to the dependencies does not seem to create any conflicts. I then modified the tutorial Exposures_polygons_lines (https://climada-python.readthedocs.io/en/stable/tutorial/climada_entity_Exposures_polygons_lines.html) to briefly show how to use osm-flex to retrieve data. This is the main part of the review.

PR Author Checklist

• Read the Contribution Guide
• Correct target branch selected (if unsure, select develop)
• Descriptive pull request title added
• Source branch up-to-date with target branch
• Documentation updated
• Tests updated
• Tests passing
• No new linter issues
• Changelog updated

PR Reviewer Checklist

• Read the Contribution Guide
• CLIMADA Reviewer Checklist passed
• Tests passing
• No new linter issues

[chahank]

Nice! I see that the Unit test are currently failing. The error message is however empty, so maybe this is related to this PR, or maybe not. It would be good to check to be sure.

[chahank]

Excellent addition!

I would however separate the OSM examples into a separate tutorial that would also be directly visible as sub-element of the Exposures tutorials. I think this would be cleaner. Otherwise, it is quite hard to find.

I would then also add a final example of an impact calculation with the obtained OSM features. The extraction of points, lines and polygons is already documented in the OSMflex documentation, so we do not have to duplicate it too much. But how to make an exposure out of the elements and then the impact is useful.

[luseverin]

Thanks for the feedback! I put the retrieval of exposure data from OSM + a simple impact calculation in a separate tutorial and added a small mention to this new tuto at the begininng of the polygons_lines tutorial.

Regarding the failing tests, something weird was happening (tests were getting cancelled for no apparent reason), but it looks all good now.

[chahank]

At the moment the tutorial has 53,393 changes (!). Please reduce this to only the relevant information. At the moment it is not even possible to make comments on the tutorial due to the large number of changes ;)

[chahank]

I would have some comments on the phrasing. I think the beginning is a bit unclear for new users.

Also, I would make sure that the impact computation at the shows some impact. At the moment it looks like there is 0 impact.

[chahank]

I also do not really understand the example the impact example. The goal is not to copy the lines polygon tutorial, but to have an example of using OSM data. Ideally, this is the first thing on top of the tutorial.

[luseverin]

Basically, what I have put in the tutorial is:

1. Examples of loading OSM data using osm-flex: roads as lines and forests as polygons for the netherlands
2. Setting up of the road and forest OSM exposure data as CLIMADA exposure object
3. Example of calculation of the impact on the roads and forests using winter storms as hazard

Would you suggest I put more emphasis on 3.? But in the end we still need to load the data from OSM at some point (but I could try to make this part shorter e.g. remove the plots). Let me know!

[chahank]

Yes, I would just make a very quick example to show that it can be used for impact computations (exposure comes from OSM, one needs to assign values), and then the details for the the roads and the forests.

Maybe also include a link to the OSM tags (all the things that can be downloaded). Maybe also add a note that if things don't work with OSMflex one should first look in the documentation there.

[chahank]

Do you need help to sanitize the files? (there should not be 53000 new lines for this PR)

[luseverin]

No I do not need help for that, I was just thinking of how to reorganize the tuto. Now I added a quick example at the start, is that good enough? Should I still add more detailed examples in the next sections e.g. using other osm layers or assigning values using LitPop?

PS: I now also need to fix this weird error in the u_lp.plot_eai_exp_geom(impact); : ValueError: Length of values (861875) does not match length of index (861881)

[chahank]

Super nice! I would suggest the following fine-tuning:

1- correct some of the typos (e.g., Honduras -> Netherlands)
2- Make in the first example clear what the implicit value is. Now you decided to put the value 1e5 per 500m of lines.
3- Avoid using capitalized variables unless they are meant to be global constants (which e.g., IMPF_LINE are not. There are more in the tutorial.)
4- At the end, explain well how to go from OSM inputs to an exposure with values
5- Add an example of OSM with points
6- Speak of osm-flex as a python package, not a module
7- Remove unecessary code (e.g., in the all of import osm_flex.download import osm_flex.extract import osm_flex.clip import osm_flex.simplify are not needed in the first cell. There are more in the tutorial I think.)
8- I think the whole section How to compute impacts using polygons or lines as exposure in CLIMADA can be removed. Instead, clearly link to the tutorial on that. I think the goal of this tutorial is to show how to make an exposures from OSM-flex. So, it is not about reproducing the OSM-flex tutorials, not to reproduce the lines-polygons tutorial for impacts, but how to make exposures from OSM with OSM-flex in CLIMADA.
9- Maybe show an example with clip and simplify?
10- Do you need the import logging from climada.util.config import LOGGER LOGGER.setLevel(logging.ERROR)?

[luseverin]

Hey!
@chahank thanks for the feedback. A bit late, but I have incorporated your suggested changes. Here are more details:

1 - correct some of the typos (e.g., Honduras -> Netherlands)

I use Honduras now because I wanted to have an example showcasing higher damages, using TC Mitch. But in the end, the damages are not quite visible on the map. Any suggestions of a better example are welcomed.

2- Make in the first example clear what the implicit value is. Now you decided to put the value 1e5 per 500m of lines.
3- Avoid using capitalized variables unless they are meant to be global constants (which e.g., IMPF_LINE are not. There are more in the tutorial.)
4- At the end, explain well how to go from OSM inputs to an exposure with values

I added a small section on that.

5- Add an example of OSM with points
6- Speak of osm-flex as a python package, not a module
7- Remove unecessary code (e.g., in the all of import osm_flex.download import osm_flex.extract import osm_flex.clip import osm_flex.simplify are not needed in the first cell. There are more in the tutorial I think.)

Actually, I at least need to do import osm_flex.download if I want to use e.g. osm_flex.download.get_country_geofabrik as apparently those are not automatically imported with the import osm_flex.

8- I think the whole section How to compute impacts using polygons or lines as exposure in CLIMADA can be removed. Instead, clearly link to the tutorial on that. I think the goal of this tutorial is to show how to make an exposures from OSM-flex. So, it is not about reproducing the OSM-flex tutorials, not to reproduce the lines-polygons tutorial for impacts, but how to make exposures from OSM with OSM-flex in CLIMADA.

I fully agree! I removed all the redundant impact calculation part. Sorry if I had misunderstood your intentions.

9- Maybe show an example with clip and simplify?

I added a short example of simplify, but clip requires some additional dependencies to be installed (see https://osm-flex.readthedocs.io/en/latest/1_clipping_shapes.html), as those are apparently not already available within the CLIMADA environment. How do you want to proceed with this?

10- Do you need the import logging from climada.util.config import LOGGER LOGGER.setLevel(logging.ERROR)?

Yes because some of the garbage outputs I was obtaining in the notebook was actually coming from the LOGGER and not the pyhon warnings.

Let me know if there is something else missing!

[luseverin]

PS: I now also need to fix this weird error in the u_lp.plot_eai_exp_geom(impact); : ValueError: Length of values (861875) does not match length of index (861881)

By the way, I have raised an issue about this, see #992

Now I also see that the build of the ReadTheDocs fails, reading (see https://readthedocs.org/projects/climada-python/builds/26833144/):

Error
The sphinx.configuration key is missing. This key is now required, see our blog post for more information.

@chahank do you know anything about this? Should I also raise an issue regarding this or is there something wrong on my branch?

PS: Nevermind, apparently the ReadTheDocs issue has been taken care of by Emanuel already and it works now :)

[chahank]

Suggested change

	"We use in this example main roads in Honduras as exposures, and winter storms\n",
	"We use in this example main roads in Honduras as exposures, and tropical cyclone Mitch\n",

[chahank]

Make links to the module tutorial lines polygons?

[chahank]

I think you do not want your users path configuration in the docs.

[chahank]

Add a comment in the code to explain that this is not needed and just used here for the nice visualization of the tutorials. People should in general not silence warnings unless they mean to.

[chahank]

Not needed.

[chahank]

Clean up

[chahank]

Why plot the centroids?

[chahank]

I am generally a bit confused here. The idea is to make a quick example. But now you import tracks, compute cyclone winds, plot centroids etc... This is now very long. Why not just use and example with the API? The goal is to have ideally in one page an example of the use of osm in climada.

[chahank]

I think the default example should not be the pre-defined infrastructures, but the free osm keys. The infrastructure case is a special case, and the osm keys is the normal case.

[chahank]

Hey! @chahank thanks for the feedback. A bit late, but I have incorporated your suggested changes. Here are more details:

Awesome!

1 - correct some of the typos (e.g., Honduras -> Netherlands)

There are a few left. I noted those I saw as comments.

I use Honduras now because I wanted to have an example showcasing higher damages, using TC Mitch. But in the end, the damages are not quite visible on the map. Any suggestions of a better example are welcomed.

I am not sure this is ideal as you say. You can look into the paper https://iopscience.iop.org/article/10.1088/2515-7620/ad15ab and take any of the examples. Eg. railways and storm lothar in switzerland.

2- Make in the first example clear what the implicit value is. Now you decided to put the value 1e5 per 500m of lines.
3- Avoid using capitalized variables unless they are meant to be global constants (which e.g., IMPF_LINE are not. There are more in the tutorial.)
4- At the end, explain well how to go from OSM inputs to an exposure with values

I added a small section on that.

5- Add an example of OSM with points
6- Speak of osm-flex as a python package, not a module
7- Remove unecessary code (e.g., in the all of import osm_flex.download import osm_flex.extract import osm_flex.clip import osm_flex.simplify are not needed in the first cell. There are more in the tutorial I think.)

Actually, I at least need to do import osm_flex.download if I want to use e.g. osm_flex.download.get_country_geofabrik as apparently those are not automatically imported with the import osm_flex.

Ok, the important is to not import things you don't need in the tutorial. Otherwise it might be confusing to the reader why this is there.

8- I think the whole section How to compute impacts using polygons or lines as exposure in CLIMADA can be removed. Instead, clearly link to the tutorial on that. I think the goal of this tutorial is to show how to make an exposures from OSM-flex. So, it is not about reproducing the OSM-flex tutorials, not to reproduce the lines-polygons tutorial for impacts, but how to make exposures from OSM with OSM-flex in CLIMADA.

I fully agree! I removed all the redundant impact calculation part. Sorry if I had misunderstood your intentions.

9- Maybe show an example with clip and simplify?

I added a short example of simplify, but clip requires some additional dependencies to be installed (see https://osm-flex.readthedocs.io/en/latest/1_clipping_shapes.html), as those are apparently not already available within the CLIMADA environment. How do you want to proceed with this?

Makes sense! One can just mention clip as an option.

10- Do you need the import logging from climada.util.config import LOGGER LOGGER.setLevel(logging.ERROR)?

Yes because some of the garbage outputs I was obtaining in the notebook was actually coming from the LOGGER and not the pyhon warnings.

Ok, makes sense too. Just add a comment that this is not what users should copy. One way to do that is to have all the logging in a separate cell which you can recommend to not copy.

Let me know if there is something else missing!

[luseverin]

Ok so I have included your comments in the last revision. Everything should be in order now. Let me know if you think this is the case.

[chahank]

Great, thanks a lot. I think this looks good now 👍