incompatible with existing WMTS layers (including those used in non-Python examples)

[prusswan]

Describe the bug

Could not get WMTSLayer to work with these services:

https://sampleserver6.arcgisonline.com/arcgis/rest/services/WorldTimeZones/MapServer/WMTS (from https://developers.arcgis.com/rest/services-reference/enterprise/wmts-map-service/)

https://gibs.earthdata.nasa.gov/wmts/epsg4326/best (from https://nasa-gibs.github.io/gibs-api-docs/access-basics/)

I inspected the Python code that is part of installed arcgis package, and realized that WMTSLayer is assuming that the WMTS service only has one layer and one tilematrixset. This is probably the main reason why those WMTS services are not working.

To Reproduce
Steps to reproduce the behavior:

timezone_url = "https://sampleserver6.arcgisonline.com/arcgis/rest/services/WorldTimeZones/MapServer/WMTS"
timezone_wmts = WMTSLayer(url=timezone_url)
#timezone_wmts = get_local_wmts(timezone_url)

m4 = gis.map()
m4.add_layer(timezone_wmts)
m4

gibs_url = "https://gibs.earthdata.nasa.gov/wmts/epsg4326/best"
#gibs_url = "https://gibs.earthdata.nasa.gov/wmts/epsg4326/best/wmts.cgi?"
#gibs_url = "https://gibs.earthdata.nasa.gov/wmts/epsg4326/best/wmts.cgi?SERVICE=WMTS&request=GetCapabilities"

gibs_wmts = WMTSLayer(url=gibs_url)

m3 = gis.map()
m3.add_layer(gibs_wmts)
m3

error:

<copy the full error message here>

Screenshots
N/A

Expected behavior
Python SDK should work with existing WMTS services, and display informative errors if there are truly issues with the WMTS services

Platform (please complete the following information):

• Python API Version: 2.2 (on ArcGIS Pro 3.2), 2.3.1 (on Ubuntu/Anaconda)

Additional context
Add any other context about the problem here, attachments etc.

[nanaeaubry]

@prusswan

We made some updates with WMTSLayer at version 2.4.0. Can you update your version and see if this has been resolved?

[nanaeaubry]

@prusswan When using version 2.4.0 this is how the WMTS Timezone layer looks on the map:

Also for the other WMTS when attempting to add to the Map Viewer it makes you choose one layer:

[prusswan]

@nanaeaubry thanks for the fast turnaround, mainly we are trying to fix/troubleshoot this for ArcGIS Pro and 2.4 will not be readily available (and will also require Python 3.10 and above). Also, as kernel errors are not well propagated to the user interface (when accessed from Pro), hopefully the API can display a simple error/informative message in case the "Add Layer" pop up fails.

It would be best to follow the JS api (https://developers.arcgis.com/javascript/latest/api-reference/esri-layers-WMTSLayer.html) as much as possible since developers would be more familiar with that.

[nanaeaubry]

@prusswan
Sure we are now following the JS API a lot more at 2.4.0 since we introduced a new mapping package: arcgis-mapping
This uses the webmap spec to create maps and to deal with layers, popups, etc. We are more than welcome to take user feedback for this new package as well.

Unfortunately we do not backport fixes so the only way to benefit from this is to upgrade. I understand this can take some time to get there so I will leave the issue open and if you are able to try this at a later version and have requests we can continue with them here.

As far as a popup for the map, we can look into it but we are focused on automation and not re-creating the map viewer within a notebook. So we can look at other ways to accomplish this or having better documentation for users to understand what they are creating with the WMTSLayer class

[prusswan]

Using the new syntax for 2.4, gibs WMTS is still not working:

gibs_url = "https://gibs.earthdata.nasa.gov/wmts/epsg4326/best" #/1.0.0/WMTSCapabilities.xml"

from arcgis.layers import WMTSLayer
gibs_wmts = WMTSLayer(url=gibs_url)

#m3 = gis.map()
from arcgis.map import Map
m3 = Map()
#m3.add_layer(gibs_wmts)
m3.content.add(gibs_wmts)
m3

Error trace:

--> 113 text = self._get_capabilities_xml(capabilities_urls)
    115 self._properties = self._get_dict_from_xml(text)
    116 return self._properties

File ...\Lib\site-packages\arcgis\layers\_ogc\wmts.py:88, in WMTSLayer._get_capabilities_xml(self, urls)
     86         except (requests.exceptions.RequestException, ValueError):
     87             pass
---> 88 raise Exception("Could not retrieve valid XML from any of the provided URLs")

Exception: Could not retrieve valid XML from any of the provided URLs

[nanaeaubry]

@prusswan

The expected xml is not returned correctly from the service. We can patch on our end to anticipate this but it does not remove the fact that there are over 1000 layers in the service.

We need to think about the best way for a user to choose what layer from the service they want to add to the map and enhance the WMTS class to show them all as well

[prusswan]

It works with the JS API and many other libraries so I don't think the issue is with the service. Yes, WMTS may have any number of services so normally the user needs to query the service first to determine the capabilities and choose the tile matrix and layers.

…

On Wed, Nov 20, 2024, 8:56 PM Nanae Aubry ***@***.***> wrote: @prusswan <https://github.com/prusswan> The expected xml is not returned correctly from the service. We can patch on our end to anticipate this but it does not remove the fact that there are over 1000 layers in the service. We need to think about the best way for a user to choose what layer from the service they want to add to the map and enhance the WMTS class to show them all as well — Reply to this email directly, view it on GitHub <#2098 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFEBYQRZ7QY6WPNI3DMWE32BSBGFAVCNFSM6AAAAABPTST4P2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOBYGUYTIMJXGY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[nanaeaubry]

@prusswan Yes I agree the service isn't the issue but the way we handle it is the issue. We assume there was only one layer per service. I appreciate your input because we do not often work with these kinds of services.

I've put a fix to handle reading in multiple layers however we need to add a new method for users to query the layer they want to add to the map.

I am open to suggestions but the idea we had was:
In the map.content.add method there is an options parameter. We could allow a user to pass in the layer identifier there just like you have to choose the layer before adding it in the map viewer.

[prusswan]

I am open to suggestions but the idea we had was: In the map.content.add method there is an options parameter. We could allow a user to pass in the layer identifier there just like you have to choose the layer before adding it in the map viewer.

This will help, but the error needs to be more descriptive too (what is considered valid xml? I tried using a direct link to the capabilities xml file - it did not work). Some libraries also allow users to specify/pass in the WMTS definition in a way that is expected/supported by the library to avoid surprises. Users can't patch an external WMTS service or the Python library, but they would be able to patch the XML string.

For GIBS (https://nasa-gibs.github.io/gibs-api-docs/access-basics/) it is probably failing due to time dimension, but one other WMTS is also failing due to the parsing logic which assumes the layer and tile matrix set to be both single items, or both lists (when in fact, it can be a single layer used with multiple matrix sets, or multiple layers using an identical matrix set).