pyppeteer is having problems launching or otherwise processing the hitlist

[gingeleski]

Hitlist processing hangs forever if Burp isn't up on the expected proxy port

[gingeleski]

I believe part of this bug was that the check for Burp Suite's proxy being up/running was not very robust. The proxy (i.e. 127.0.0.1:8080) was inadvertently provided for http only, not https. This caused a false positive because the test target is Github which forces TLS/https.

Anyway... there's still something going on here. Further debug is needed before issue can be closed.

[gingeleski]

[gingeleski]

pyppeteer was working before for sure. It would print stuff to the console and whatever.

Now there is some issue. A pull request out there for it suggests maybe an issue on Windows.

Need to dig in, look at pyppeteer's logging, etc.

[gingeleski]

Haven't done the above but seems that pyppeteer may be dead, at its official repo.

There are some forks farther ahead and this one looks promising - https://github.com/CircleOnCircles/pyppeteer/tree/dev-wackazong

[gingeleski]

From doing Ctrl+C:

(venv) PS C:\Users\gingeleski\workspace\conspiracy> python conspiracy.py --hitlist=./test/assets/hitlist1.txt play.google.com
[Conspiracy] Checking if Burp Suite proxy 127.0.0.1:8080 is running...
[Conspiracy] Starting asynchronous processing of hitlist now...
DEBUG about to get event loop
DEBUG about to run until complete
DEBUG in processing of hitlist
[I:pyppeteer.launcher] terminate chrome process...
Traceback (most recent call last):
  File "c:\users\gingeleski\anaconda3\Lib\socket.py", line 713, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [WinError 10061] No connection could be made because the target machine actively refused it

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "conspiracy.py", line 261, in <module>
    main()
  File "conspiracy.py", line 239, in main
    result = loop.run_until_complete(run_processing_on_hitlist(burp_proxy_is_up))
  File "c:\users\gingeleski\anaconda3\Lib\asyncio\base_events.py", line 466, in run_until_complete
    return future.result()
  File "conspiracy.py", line 142, in run_processing_on_hitlist
    browser = await get_browser(use_burp_suite_proxy)
  File "conspiracy.py", line 131, in get_browser
    return await pyppeteer.launch(headless=True)
  File "C:\Users\gingeleski\workspace\conspiracy\venv\lib\site-packages\pyppeteer\launcher.py", line 311, in launch
    return await Launcher(options, **kwargs).launch()
  File "C:\Users\gingeleski\workspace\conspiracy\venv\lib\site-packages\pyppeteer\launcher.py", line 189, in launch
    self.browserWSEndpoint = self._get_ws_endpoint()
  File "C:\Users\gingeleski\workspace\conspiracy\venv\lib\site-packages\pyppeteer\launcher.py", line 225, in _get_ws_endpoint
    with urlopen(url) as f:
  File "c:\users\gingeleski\anaconda3\Lib\urllib\request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "c:\users\gingeleski\anaconda3\Lib\urllib\request.py", line 526, in open
    response = self._open(req, data)
  File "c:\users\gingeleski\anaconda3\Lib\urllib\request.py", line 544, in _open
    '_open', req)
  File "c:\users\gingeleski\anaconda3\Lib\urllib\request.py", line 504, in _call_chain
    result = func(*args)
  File "c:\users\gingeleski\anaconda3\Lib\urllib\request.py", line 1346, in http_open
    return self.do_open(http.client.HTTPConnection, req)
  File "c:\users\gingeleski\anaconda3\Lib\urllib\request.py", line 1318, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "c:\users\gingeleski\anaconda3\Lib\http\client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "c:\users\gingeleski\anaconda3\Lib\http\client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "c:\users\gingeleski\anaconda3\Lib\http\client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "c:\users\gingeleski\anaconda3\Lib\http\client.py", line 1026, in _send_output
    self.send(msg)
  File "c:\users\gingeleski\anaconda3\Lib\http\client.py", line 964, in send
    self.connect()
  File "c:\users\gingeleski\anaconda3\Lib\http\client.py", line 936, in connect
    (self.host,self.port), self.timeout, self.source_address)
  File "c:\users\gingeleski\anaconda3\Lib\socket.py", line 713, in create_connection
    sock.connect(sa)
  File "C:\Users\gingeleski\workspace\conspiracy\venv\lib\site-packages\pyppeteer\launcher.py", line 174, in _close_process
    self._loop.run_until_complete(self.killChrome())
  File "c:\users\gingeleski\anaconda3\Lib\asyncio\base_events.py", line 454, in run_until_complete
    self.run_forever()
  File "c:\users\gingeleski\anaconda3\Lib\asyncio\base_events.py", line 408, in run_forever
    raise RuntimeError('This event loop is already running')
RuntimeError: This event loop is already running

[gingeleski]

I cleared the local Chromium that pyppeteer had downloaded, didn't make a difference.

However, I was able to successfully run the test script below. So somewhere along the way of me writing this tool the calls to pyppeteer went sideways. Probably easiest thing to do is try to get main script closer to this test script and consider the discrepancy.

import asyncio
from pyppeteer import launch

async def main():
    browser = await launch()
    page = await browser.newPage()
    await page.goto('http://example.com')
    await page.screenshot({'path': 'example.png'})
    await browser.close()

asyncio.get_event_loop().run_until_complete(main())

[gingeleski]

Figured out what the problem is and opened #47 to deal with it.

For now the check of whether Burp Suite is up has been hard-coded False.