Dataset only emit for specific models

[DanMawdsleyBA]

Description

We're looking into leveraging datasets but one problem we're having is the ability to control which models produce datasets. Would it be possible to implement something to be able to control which models produce datasets. Perhaps using a list or regex of models to emit datasets?

Use case/motivation

As part of dbt we use a lot of working tables were we don't want to emit datasets and want to cut down on the number of datasets as the airflow graph can get hard to use.

Related issues

No response

Are you willing to submit a PR?

• Yes, I am willing to submit a PR!

[tatiana]

@DanMawdsleyBA I haven't tried this, but it may be possible with Cosmos 1.8., thanks to @wornjs' #1339 contribution:
https://astronomer.github.io/astronomer-cosmos/getting_started/custom-airflow-properties.html

You could try to set RenderConfig(emit_datasets=False) at a DbtDag or DbtTaskGroup level, and in the models you wanted to emit datasets, you could try to configure:

version: 2

models:
  - name: name
    description: description
    meta:
      owner: '[email protected]'
      cosmos:
        operator_kwargs:
          emit_datasets: True

Since this parameter is set at an operator level:

astronomer-cosmos/cosmos/operators/base.py

Line 34 in 184e45f

:param emit_datasets: Enable emitting inlets and outlets during task execution

Could you please try it and let us know how it goes?

[DanMawdsleyBA]

I have had a try of this which doesn't seem to work. I also tried setting the emit datasets to false from the operator args:

        operator_args={'dbt_cmd_flags': ['--exclude', 'test_type:unit']
                       , 'full_refresh': True
                       , "emit_datasets": False}

This doesn't work and still emits the datasests. The only way I have managed to turn them off is within the render_config:

        render_config=RenderConfig(
            emit_datasets=False)

But this will still turn them off for the entire dag.

Wonder if this is because of the behavior in converter.py

        task_args = {
            **operator_args,
            "project_dir": execution_config.project_path,
            "partial_parse": project_config.partial_parse,
            "profile_config": profile_config,
            "emit_datasets": render_config.emit_datasets,
            "env": env_vars,
            "vars": dbt_vars,
        }

For the emit_datasets its only using the value defined from the render config