Support fetching external URLs

[pdawyndt]

This code should output a random haiku (scraped from a web page), but access to external URLS does not seem to be working. Strange limitation for a Python runtime that is already running in a browser (and has BeautifulSoup on board).

from urllib.request import urlopen

# download web page with random haiku
url = 'http://haikuguy.com/issa/random.php'
reader = urlopen(url)

# parse page until start of haiku is found
marker = '<p class="english">'
line = reader.readline().decode('utf-8')
while line and not line.startswith(marker):
    line = reader.readline().decode('utf-8')

# read three haiku lines and display them
if line.startswith(marker):
    print(line[len(marker):].strip()[:-6])
    line = reader.readline().decode('utf-8')
    print(line.strip()[:-6])
    line = reader.readline().decode('utf-8')
    print(line.strip()[:-4])

[alexmojaki]

See pyodide/pyodide#662 and pyodide/pyodide#375. You can read a URL with pyodide.open_url or js.fetch. I don't know why urllib isn't patched or something in pyodide.

In any case, the browser has many security restrictions which cause trouble for most URLs. Your example isn't https so it gets blocked. Other URLs are typically blocked by CORS.

[winniederidder]

@alexmojaki Is providing an override of urllib something you are doing in your own packages? Or should I make a Papyros-specific part? It would be similar to input/matplotlib by making urllib functions point to e.g. pyodide.open_url calls. The most common calls can easily be covered by this.

[alexmojaki]

I have no plans for this, go ahead.

[bmesuere]

@winniederidder you may add this to papyros, but note that external connections will be blocked by the CSP on Dodona.

[winniederidder]

After some studying, this issue would affect atleast the following libraries: urllib, urllib3, http.client, requests, SSL, websocket-client, websockets, and more. Doing this in a maintainable way from our side is thus not really feasible. This is something that has to be fixed upstream and is thus a limitation of Pyodide. As @bmesuere mentioned, projects integrating Papyros will have protection mechanisms limiting the accessible URLs.
I will open a new issue to discuss limitations of Pyodide for things like this.