dbt-materialize: v1.8.0 migration

[bobbyiliev]

Motivation

Fixes MaterializeInc/database-issues#7800.

Initial PR to update the required dependencies for the dbt v1.8.0 migration.

[morsapaes]

@bobbyiliev, to fix linting, you can run: ../../bin/pyfmt; which will ✨ automatically✨ re-format any files. You then just need to push the changes in!

[morsapaes]

@nrainer-materialize, the tests are failing on what seems to be a Docker-related issue; do you have a hunch about what's going on?

$ docker compose run -eDBT_HOST=materialized -eKAFKA_ADDR=redpanda:9092 -eSCHEMA_REGISTRY_URL=http://schema-registry:8081 dbt pytest dbt-materialize/tests
time="2024-05-14T20:33:01Z" level=error msg="error waiting for container: context canceled"
Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "pytest": executable file not found in $PATH: unknown

[nrainer-materialize]

@nrainer-materialize, the tests are failing on what seems to be a Docker-related issue; do you have a hunch about what's going on?

$ docker compose run -eDBT_HOST=materialized -eKAFKA_ADDR=redpanda:9092 -eSCHEMA_REGISTRY_URL=http://schema-registry:8081 dbt pytest dbt-materialize/tests
time="2024-05-14T20:33:01Z" level=error msg="error waiting for container: context canceled"
Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "pytest": executable file not found in $PATH: unknown

Well, it says that it cannot find pytest in the Docker image. While I first could not find any suspicious changes in the pull request, I am wondering now if the issue could be related to the removal of the following code (though, just guessing):

    extras_require={
        "dev": ["dbt-tests-adapter~=1.7.0"],
    },

Does this dependency maybe ship with pytest?

[morsapaes]

That was my first hunch, but dbt-tests-adapter is now bundled in dbt-adapters. A previous commit from Bobby included the dependency explicitly, but was failing with the same error. It makes sense that this is expecting pytest to be installed with the connector, though!

[nrainer-materialize]

That was my first hunch, but dbt-tests-adapter is now bundled in dbt-adapters. A previous commit from Bobby included the dependency explicitly, but was failing with the same error. It makes sense that this is expecting pytest to be installed with the connector, though!

Well, it is the Docker image, which does no longer include pytest.

This build uses materialize/dbt-materialize:mzbuild-XCEFLV6YVDL2A4UB7KZCNNFNQ2TJID2G. A build on main uses materialize/dbt-materialize:mzbuild-XXTN55FJRPU2CVG5GVWMQK2JI6EPGFBD.

This branch:

nrainer@mz-nrainer mz-repo % docker run -it materialize/dbt-materialize:mzbuild-XCEFLV6YVDL2A4UB7KZCNNFNQ2TJID2G bash
root@6e752dc26bef:/# ls
bin  boot  dbt-materialize  dev  etc  home  lib  media	mnt  opt  proc	root  run  sbin  srv  sys  tmp	usr  var
root@6e752dc26bef:/# find . -name "pytest"

main:

nrainer@mz-nrainer ~ % docker run -it materialize/dbt-materialize:mzbuild-XXTN55FJRPU2CVG5GVWMQK2JI6EPGFBD bash
root@fead2752336c:/# ls
bin  boot  dbt-materialize  dev  etc  home  lib  media	mnt  opt  proc	root  run  sbin  srv  sys  tmp	usr  var
root@fead2752336c:/# find . -name "pytest"
./usr/local/bin/pytest
./usr/local/lib/python3.8/site-packages/pytest
root@fead2752336c:/# 

That is, if the Docker image is based on the changed setup file (which I assume), it has to do with the dependencies.

[nrainer-materialize]

Try adding

RUN pip install pitest

to misc/dbt-materialize/Dockerfile.

[morsapaes]

____________ ERROR collecting tests/adapter/test_relation_types.py _____________
ImportError while importing test module '/dbt-materialize/tests/adapter/test_relation_types.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
usr/local/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
dbt-materialize/tests/adapter/test_relation_types.py:19: in <module>
    from fixtures import (
dbt-materialize/tests/adapter/fixtures.py:16: in <module>
    from dbt.tests.adapter.hooks import test_model_hooks as core_base
E   ModuleNotFoundError: No module named 'dbt.tests.adapter'

This confirms that we're just not picking up the test dependencies from dbt-adapters (or, /dbt-test-adapters) correctly. @colin-rogers-dbt, it feels like we're missing some basic nuance that is probably straightforward for you to single out; mind giving this a look? Adding pytest to the Dockerfile and adding the dependency to the dbt-tests-adapter subdirectory seems to have done it!

[morsapaes]

Great job on the upgrade, @bobbyiliev! There's a couple of minor things we can follow-up on, but I'm confident about rolling out the release.