Skip to content

Commit

Permalink
Improve OpenLineage documentation
Browse files Browse the repository at this point in the history
Support users who may have questions similar to:
* #1423
* #1430
  • Loading branch information
tatiana authored Dec 27, 2024
1 parent 44f8655 commit 0d29e0b
Showing 1 changed file with 22 additions and 2 deletions.
24 changes: 22 additions & 2 deletions docs/configuration/lineage.rst
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,7 @@ Configuring Lineage
===================

Since Cosmos 1.1, it uses internally the `openlineage-integration-common <https://github.com/OpenLineage/OpenLineage/tree/main/integration/common>`_
to parse artifacts generated with dbt commands and create lineage events. This feature is only available for the local
and virtualenv execution methods (read `execution modes <../getting_started/execution-modes.html>`_ for more information).
to parse artefacts generated with dbt commands and create lineage events.

To emit lineage events, Cosmos can use one of the following:

Expand All @@ -15,6 +14,27 @@ To emit lineage events, Cosmos can use one of the following:
No change to the user DAG files is required to use OpenLineage.


Known limitations
-----------------

This feature is only available for the local
and virtualenv execution methods (read `execution modes <../getting_started/execution-modes.html>`_ for more information).

Additionally, since Cosmos uses the open-source `openlineage-integration-common <https://github.com/OpenLineage/OpenLineage/tree/main/integration/common>`_, it relies on this library to support specific dbt adapters. As of 27 December 2024, the version 1.26.0 of this package supports:

* Athena
* BigQuery
* Databricks
* DuckDB
* Dremio
* Postgres
* Redshift
* Snowflake
* Spark
* SQLServer

Contributions are also welcome in the `OpenLineage project <https://github.com/OpenLineage/OpenLineage/blob/main/integration/common/openlineage/common/provider/dbt/processor.py#L36C1-L47C22>`_ to support more adaptors.

Installation
------------

Expand Down

0 comments on commit 0d29e0b

Please sign in to comment.