Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug in df.write #21

Open
LeoGMagalha opened this issue Aug 21, 2024 · 0 comments
Open

Bug in df.write #21

LeoGMagalha opened this issue Aug 21, 2024 · 0 comments

Comments

@LeoGMagalha
Copy link

When we use df.write operations to external tables, the agent identifies the output table name as a location path:

code:
(
df.write.partitionBy("pk_datref_ox")
mode("overwrite")
format("parquet")
saveAs Table("workspace_db.tableB")
)
log spark:

24/08/21 09:18:24 INFO @penMetadata Transport: lineage was sent successfully to OpenMetadata for from Table: workspace_db:tableA, toTable: /C:/Users/leomag/spark- warehouse/workspace_db.db/tableB

log OM server:

127.0.0.6--[21/Aug/2024:12:17:22 +0000] "GET /api/v1/search/query?q-fullyQualifiedName% 3Aathena-sirius-teste.*%2FC%3A%2Fusers%2Fleomag%2Fspark-warehouse% 2Fworkspace_db.db%2Ftableb&size=10 HTTP/1.1" 200 301 "." "Apache- HttpClient/4.5.14 (Java/15.0.1) 55

So, the output table it´s not found in the Open Metadata and the lineage do not work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant