HERE error handling leading to a bug

[radumas]

example log

[2024-12-27, 16:39:47 EST] {base.py:84} INFO - Retrieving connection 'here_api_key'
[2024-12-27, 16:40:48 EST] {here_api.py:125} INFO - Polling status of query request: 598638
[2024-12-27, 16:40:48 EST] {here_api.py:50} INFO - Getting Access Token
[2024-12-27, 16:41:48 EST] {here_api.py:125} INFO - Polling status of query request: 598638
[2024-12-27, 16:41:48 EST] {here_api.py:50} INFO - Getting Access Token
[2024-12-27, 16:42:48 EST] {here_api.py:125} INFO - Polling status of query request: 598638
[2024-12-27, 16:42:48 EST] {here_api.py:50} INFO - Getting Access Token
[2024-12-27, 16:43:48 EST] {here_api.py:125} INFO - Polling status of query request: 598638
[2024-12-27, 16:43:48 EST] {here_api.py:50} INFO - Getting Access Token
[2024-12-27, 16:44:48 EST] {here_api.py:125} INFO - Polling status of query request: 598638
[2024-12-27, 16:44:54 EST] {here_api.py:50} INFO - Getting Access Token
[2024-12-27, 16:45:54 EST] {here_api.py:125} INFO - Polling status of query request: 598638
[2024-12-27, 16:45:54 EST] {here_api.py:50} INFO - Getting Access Token
[2024-12-27, 16:46:54 EST] {here_api.py:125} INFO - Polling status of query request: 598638
[2024-12-27, 16:46:54 EST] {here_api.py:50} INFO - Getting Access Token
[2024-12-27, 16:47:54 EST] {here_api.py:125} INFO - Polling status of query request: 598638
[2024-12-27, 16:47:54 EST] {here_api.py:50} INFO - Getting Access Token
[2024-12-27, 16:47:54 EST] {here_api.py:152} INFO - Requested query completed
[2024-12-27, 16:47:54 EST] {taskinstance.py:3310} ERROR - Task failed with exception

Notice the alternating between "Polling status" and "Getting Access Token". There's an HTTPError being thrown that is not because the token expired. This happens 8 times, and then the counter is exhausted.

bdit_data-sources/here/traffic/here_api.py

Lines 128 to 134 in 79009ab

	try:
	query_status.raise_for_status()
	status = str(query_status.json()['status'])
	except requests.exceptions.HTTPError:
	#access token expires after 1 hr, try to generate up to 3 times.
	access_token = get_access_token(api_conn)
	token_counter+=1

At this point, the loop is exited and we get

bdit_data-sources/here/traffic/here_api.py

Lines 152 to 154 in 79009ab

	LOGGER.info('Requested query completed')
	return query_status.json()['outputUrl']

Traceback (most recent call last):
  File "/data/airflow/airflow_venv/lib/python3.10/site-packages/airflow/models/taskinstance.py", line 767, in _execute_task
    result = _execute_callable(context=context, **execute_callable_kwargs)
  File "/data/airflow/airflow_venv/lib/python3.10/site-packages/airflow/models/taskinstance.py", line 733, in _execute_callable
    return ExecutionCallableRunner(
  File "/data/airflow/airflow_venv/lib/python3.10/site-packages/airflow/utils/operator_helpers.py", line 252, in run
    return self.func(*args, **kwargs)
  File "/data/airflow/airflow_venv/lib/python3.10/site-packages/airflow/models/baseoperator.py", line 406, in wrapper
    return func(self, *args, **kwargs)
  File "/data/airflow/airflow_venv/lib/python3.10/site-packages/airflow/decorators/base.py", line 266, in execute
    return_value = super().execute(context)
  File "/data/airflow/airflow_venv/lib/python3.10/site-packages/airflow/models/baseoperator.py", line 406, in wrapper
    return func(self, *args, **kwargs)
  File "/data/airflow/airflow_venv/lib/python3.10/site-packages/airflow/operators/python.py", line 238, in execute
    return_value = self.execute_callable()
  File "/data/airflow/airflow_venv/lib/python3.10/site-packages/airflow/operators/python.py", line 256, in execute_callable
    return runner.run(*self.op_args, **self.op_kwargs)
  File "/data/airflow/airflow_venv/lib/python3.10/site-packages/airflow/utils/operator_helpers.py", line 252, in run
    return self.func(*args, **kwargs)
  File "/data/airflow/dags/pull_here_path.py", line 72, in get_download_link
    download_url = get_download_url(request_id, api_conn.extra_dejson['status_base_url'], access_token, api_conn.login, api_conn)
  File "/data/airflow/data_scripts/here/traffic/here_api.py", line 154, in get_download_url
    return query_status.json()['outputUrl']
KeyError: 'outputUrl'