Integration tests do not pass in local environment

[i-chvets]

Bug Description

When running integration test command using environments degined in tox.ini (tox -e integration) tests fail to pass.

It should be noted that in recent PR these tests successfully passed in CI (GH runners). In particular test_ingress_url test.

The following is a PASSED test in GH runner, but never finishes in local testing (see To Reproduce section for more details):

tests/integration/test_charm.py::TestCharm::test_ingress_url 
-------------------------------- live log call ---------------------------------
INFO     httpx:_client.py:1013 HTTP Request: GET https://10.1.0.42:16443/api/v1/namespaces/testing/services/istio-ingressgateway-workload "HTTP/1.1 200 OK"
PASSED

To Reproduce

Execute tox -e integration and observe errors and not passed tests. It looks like running this particular command mlflow-server application is deleted in one set of tests and another set of tests fails. See relevant log section for more details.

When model for excuting integration tests is created and option to keep models is given (to prevent destruction of mlflow-server) Running the following command tox -e integration -- --model test --keep-models produces different error. This time test_ingress_url test never completes. See relevant log section for more details.

Environment

Local development environment, in this repository, on main branch.

Relevant log output

# results of tox -e integration
____________________ TestCharm.test_add_relational_db_with_relation_expect_active _____________________
Traceback (most recent call last):
  File "/home/ichvets/cw/dev/mlflow-operator/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 341, in from_call
    result: Optional[TResult] = func()
  File "/home/ichvets/cw/dev/mlflow-operator/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 262, in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
  File "/home/ichvets/cw/dev/mlflow-operator/.tox/integration/lib/python3.8/site-packages/pluggy/_hooks.py", line 433, in __call__
    return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
  File "/home/ichvets/cw/dev/mlflow-operator/.tox/integration/lib/python3.8/site-packages/pluggy/_manager.py", line 112, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/ichvets/cw/dev/mlflow-operator/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 155, in _multicall
    return outcome.get_result()
  File "/home/ichvets/cw/dev/mlflow-operator/.tox/integration/lib/python3.8/site-packages/pluggy/_result.py", line 108, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/home/ichvets/cw/dev/mlflow-operator/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 80, in _multicall
    res = hook_impl.function(*args)
  File "/home/ichvets/cw/dev/mlflow-operator/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 177, in pytest_runtest_call
    raise e
  File "/home/ichvets/cw/dev/mlflow-operator/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 169, in pytest_runtest_call
    item.runtest()
  File "/home/ichvets/cw/dev/mlflow-operator/.tox/integration/lib/python3.8/site-packages/_pytest/python.py", line 1788, in runtest
    self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  File "/home/ichvets/cw/dev/mlflow-operator/.tox/integration/lib/python3.8/site-packages/pluggy/_hooks.py", line 433, in __call__
    return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
  File "/home/ichvets/cw/dev/mlflow-operator/.tox/integration/lib/python3.8/site-packages/pluggy/_manager.py", line 112, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/ichvets/cw/dev/mlflow-operator/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 155, in _multicall
    return outcome.get_result()
  File "/home/ichvets/cw/dev/mlflow-operator/.tox/integration/lib/python3.8/site-packages/pluggy/_result.py", line 108, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/home/ichvets/cw/dev/mlflow-operator/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 80, in _multicall
    res = hook_impl.function(*args)
  File "/home/ichvets/cw/dev/mlflow-operator/.tox/integration/lib/python3.8/site-packages/_pytest/python.py", line 194, in pytest_pyfunc_call
    result = testfunction(**testargs)
  File "/home/ichvets/cw/dev/mlflow-operator/.tox/integration/lib/python3.8/site-packages/pytest_asyncio/plugin.py", line 532, in inner
    _loop.run_until_complete(task)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/home/ichvets/cw/dev/mlflow-operator/tests/integration/test_charm.py", line 200, in test_add_relational_db_with_relation_expect_active
    await ops_test.model.relate(OBJECT_STORAGE_CHARM_NAME, CHARM_NAME)
  File "/home/ichvets/cw/dev/mlflow-operator/.tox/integration/lib/python3.8/site-packages/juju/model.py", line 1448, in relate
    result = await app_facade.AddRelation(endpoints=endpoints, via_cidrs=None)
  File "/home/ichvets/cw/dev/mlflow-operator/.tox/integration/lib/python3.8/site-packages/juju/client/facade.py", line 483, in wrapper
    reply = await f(*args, **kwargs)
  File "/home/ichvets/cw/dev/mlflow-operator/.tox/integration/lib/python3.8/site-packages/juju/client/_client13.py", line 946, in AddRelation
    reply = await self.rpc(msg)
  File "/home/ichvets/cw/dev/mlflow-operator/.tox/integration/lib/python3.8/site-packages/juju/client/facade.py", line 656, in rpc
    result = await self.connection.rpc(msg, encoder=TypeEncoder)
  File "/home/ichvets/cw/dev/mlflow-operator/.tox/integration/lib/python3.8/site-packages/juju/client/connection.py", line 629, in rpc
    raise errors.JujuAPIError(result)
juju.errors.JujuAPIError: application "mlflow-server" not found


# result of tox -e integration -- --model test --keep-models
tests/integration/test_charm.py::TestCharm::test_ingress_url 
-------------------------------------------- live log call ---------------------------------------------
INFO     httpx:_client.py:1013 HTTP Request: GET https://10.1.1.124:16443/api/v1/namespaces/test/services/istio-ingressgateway-workload "HTTP/1.1 200 OK"
. . . 
INFO     httpx:_client.py:1013 HTTP Request: GET https://10.1.1.124:16443/api/v1/namespaces/test/services/istio-ingressgateway-workload "HTTP/1.1 200 OK"
FAILED
_____________________________________ TestCharm.test_ingress_url ______________________________________
Traceback (most recent call last):
  File "/home/ichvets/cw/dev/mlflow-operator/.tox/integration/lib/python3.8/site-packages/aiohttp/connector.py", line 1155, in _create_direct_connection
    hosts = await asyncio.shield(host_resolved)
  File "/home/ichvets/cw/dev/mlflow-operator/.tox/integration/lib/python3.8/site-packages/aiohttp/connector.py", line 874, in _resolve_host
    addrs = await self._resolver.resolve(host, port, family=self._family)
  File "/home/ichvets/cw/dev/mlflow-operator/.tox/integration/lib/python3.8/site-packages/aiohttp/resolver.py", line 33, in resolve
    infos = await self._loop.getaddrinfo(
  File "/usr/lib/python3.8/asyncio/base_events.py", line 825, in getaddrinfo
    return await self.run_in_executor(
  File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib/python3.8/socket.py", line 918, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -5] No address associated with hostname

Additional context

Need to verify in some other development environment to exclude possible broken development environment on the reporter's development machine.

[NohaIhab]

I was able to reproduce the first part of the issue when running tox -e integration, the reason this is failing because we are not specifying a model name with -- model <model_name>, and due to how the tox file is written if we don't specify the model name then each test file will be executed in a different model.
See in the .tox:

mlflow-operator/tox.ini

Lines 76 to 82 in f18c4e6

	[testenv:integration]
	commands =
	pytest -v --tb native --asyncio-mode=auto {[vars]tst_path}integration/test_deploy_runners.py --keep-models --log-cli-level=INFO -s {posargs}
	pytest -v --tb native --asyncio-mode=auto {[vars]tst_path}integration/test_charm.py --log-cli-level=INFO -s {posargs}
	deps =
	-r requirements-integration.txt
	description = Run integration tests

causing the error we see in the first part of the logs above:

juju.errors.JujuAPIError: application "mlflow-server" not found

the application is not found in this model, because it was deployed in a different model.

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5283.

This message was autogenerated

[orfeas-k]

Thank you @NohaIhab for nailing down the root cause of this. We should thus make a sufficient improvement in our code (e.g. add a relevant comment in tox environment or configure it to use the same model name by default) in order to avoid future misunderstanding about when/how the tests work.